FIELD OF INVENTIONThe invention relates generally to vehicles havingguidance, navigation, and control processing in a wirelessnetwork and particularly relates to autonomous and semi-autonomousvehicles being in mutual wireless communicationover an IP data network.
BACKGROUNDTraditionally, remote controlled (RC) vehicles use radiofrequencies for navigation and guidance. The controlsignals are usually directly modulated onto an analogcarrier then transmitted through radio bands by a controlmodule. The vehicle has a receiver that demodulates thecarrier and responds to the control signals. For recreational RC applications, there is no sensory feedbackprovided by the vehicle itself except active visualmonitoring by the controller. For certain military andaeronautical applications, the vehicle may be equipped withsensors that can be monitored by the controller. Suchapplications may consume a large bandwidth in the feedbackpath and require higher power and frequency carrier thantheir recreational counterpart.
Radio frequency is highly susceptible to jamming andinterference from other transmitters. In essence, eachvehicle will require its own frequency for carrier in orderto avoid interference. In addition, the transmitter needsto generate a moderately high power carrier while stillremain portable. The control signal, as well as feedbackinformation provided by the vehicle, is limited by therelatively low bandwidth of the carrier. In fact, thefeedback information will needs its own carrier frequencyin order to avoid conflict with the control signal.
Remote control range is limited by the power of thetransmitter. Since these devices must observe FCCguidelines, transmission power is very limited, resultingin very limited range of control. In addition, since thecontrol module must provide power to the transmitter whilebeing portable, the size of the battery becomes a limitingfactor to the carrier strength as well. Each control andfeedback path will require its own frequency in the radioband, unless the carrier is further coded or modulated,resulting in additional complexity in the receiver.
Presently robots for entertainment purposes may receivewireless commands as data packet from a computer userinterface connected over local area network to a wireless access point using Internet Protocol. Wireless accesspoints and repeaters may be mounted to and powered by landvehicles. There is a need for mobile wireless access pointsand wireless nodes to be able to receive guidanceinstructions and operate in autonomous and semi-autonomousfashion.
SUMMARYThe present invention is its several embodiments is adistributed supervisory vehicle control system having avehicle that includes: one or more supervisory processinginterfaces and a sensor suite comprising sensors formeasuring one or more vehicle states; a wireless networkinterface for receiving one or more externally generatedcommands via data packets; and a guidance processinginterface operably connected to the wireless networkinterface wherein the guidance processing receives sensorsuite signals and wherein the guidance processing interfaceprocesses the externally generated commands and providesvehicle system commands. Preferably, the wireless networkinterface receives externally generated commands as datapackets according to Transmission Control Protocol/InternetProtocol (TCP/IP). In some embodiments, the one or moreexternally generated commands comprise one or morenavigation goals from one or more supervisory processinginterfaces. In some embodiments, the externally generatedcommands comprise one or more steering commands one or moresupervisory processing interfaces. Preferably, the wirelessnetwork interface transmits the measured one or more groundvehicle states to the one or more supervisory processors.For some of the vehicle embodiments of the system, thewireless network interface further comprises processing fordata packet forwarding particularly having a wirelessnetwork access point. In addition to vehicle body sensors,the sensor suite further comprises a visual band sensing element preferably having recording and temporary andpermanent local storage and local signal processingparticularly filtering and stabilization of signals fromthe visual band sensing element. In addition, the visualimage processing preferably includes pattern recognitionprocessing for the autonomous vehicles. For other vehicleembodiments, the sensor suite may include an infrared bandsensing element with recording and temporary and permanentstorage and signal processing particularly filtering of thesignals from the infrared band sensing element. In theautonomous vehicles, it is preferred that the infraredprocessing means of signals comprises pattern recognitionprocessing. The sensor suite may also an audio sensingelement such as a microphone and one or more temperaturesensing elements, preferably distributed about the vehicle.In some embodiments, the ground vehicle further includes anarm having tactile feedback sensors and force feedbacksensors. The directing processors of the supervisoryinterfaces preferably have a graphic user interfaces. Forthose vehicles operating in outside environments, thesensor suite preferably includes a global positioningsystem.
DESCRIPTION OF THE DRAWINGS- FIG. 1 illustrates a general system architecture of thepresent invention;
- FIG. 2 illustrates the span and handoff properties of aautonomous and semi-autonomous vehicles in communicationvia a mobile IP network;
- FIG. 3 illustrates the general guidance, navigation andcontrol of the autonomous and semi-autonomous vehicles ofthe present invention;
- FIG. 4 illustrates the mobile group roaming and station-keepingof the semi-autonomous vehicles being mobile IP nodes about the autonomous or semi-autonomous vehiclesbeing a mobile IP wireless access point; and
- FIG. 5 illustrates the mobile group constellation andstation-keeping of the semi-autonomous air vehicles beingmobile IP nodes about the autonomous or semi-autonomous airvehicles being a mobile IP wireless access point and thecooperation with a cellular base station and an enhancedautonomous air vehicles being both a cellular node and amobile IP wireless access point.
DETAILED DESCRIPTIONIllustrated in FIG. 1 is an example of embodiment of thepresent invention whereby the mobile IP-based wirelessaccess point (MIPWAP) vehicles receive and send theirsupervisory commands and provide state feedbackrespectively and one or more mobile IP-based wireless node(MIPWB) vehicle receive supervisory commands and send statefeed. In this example, one or more human supervisor mayinteract with one or more MIPWAP vehicles via asupervisorinterface 102. Thesupervisor interface 102 is preferablyconnected to anetwork 104 and then to the Internet 106 viaafirst gateway router 108 or thesupervisor interface 103may be directly connected 111 to a secondIP gatewayrouter 109. A stationarywireless access point 114supporting TCP/IP and operating according to IEEE 802.11bor 802.11g for example is connected to the Internet 106 viaa thirdIP gateway router 110. From the stationarywireless access point 114, data packets are sent to thefirst MIPWAP 120 preferably according to Mobile IP. Thus,the router connected to the first MIPWAP's home agent,i.e., the firstIP gateway router 108 for example, is atunnel entry point. The exit point is determined viaMobile IP's Registration procedure were foreign AgentDiscover in employed to determine whether the MIPWAP 120 isconnected to a foreign link, the thirdIP gateway router 110 for example and to acquire the care-of address of thethirdIP gateway router 110 to then register this care-ofaddress with the home agent. The care-of address can thenbe used to encapsulate IP payloads for tunneling betweenthe firstIP gateway router 108 and thethird gatewayrouter 110. The MIPWAP120 then functions as a hub orswitch to forward packets to and from its associatedMIPWBs122. In addition, where distances may exceed the range ofthe stationary IPwireless access point 114 or acombination of the the stationary IPwireless access point114 and two or more MIPWAPs 120 in an ad hoc packet-forwardingnetwork, the third IP gateway router may connectto the Public Switched Telephone Network (PSTN) 150 via aPSTN gateway 152. Acellular base station 154 connects to amobile switching center 156. A second MIPWAPembodiment130 has a cellular telephone system interface 131where thecellular signals are converted 132 via a computer telephoneinterface (CTI) into data packets that are sent via the IPwireless portion 133 of the second MIPWAP 130. The secondMIPWAP, or enhanced MIPWAP (EMIPWAP) 130, then functions asa hub or switch to forward packets to and from itsassociatedMIPWBs 142.
In the first example embodiment of theoperator interface102, the user (not shown) conveys information to andreceives information from the Internet 106 by means of anInternet Service Provider (ISP), for example, using a dial-upmodem, digital subscriber line (DSL) or cable modem, orT1/T3 line. The ISP then provides the link to the systemof core routers and other network devices capable ofcommunicating guidance and control information to and fromtheoperator interface 102 located any where in the world.Theoperator interface 102 may be directly coupled to theISP, or be embedded within a subnet including an edgerouter in a LAN, WAN, or MAN.
An alternative operator interface also includes a computerand/or a joystick consistent with that described above. Inthis example embodiment, however, the interface includes awireless modem. The wireless modem may be used to directaccess the digitalcellular system 154, 156 through thefirst base station 154. Although the user interface inthis example may lie within the base station subsystem asthe one or more base stations through which one or more ofthe remotely guidedground vehicles 120, 122, 130 andairvehicles 520, 522, and 530 interact, theoperator interface102 may be geographically remote from the vehicle. In thisexample embodiment then, theoperator interface 102 as wellas the remotely guidedground vehicles 120, 122, 130 andair vehicles 520, 522, and 530 are capable of roamingthroughout areas in which there is cellular coverage.
Instead of using a dedicated carrier for control signal,this invention uses a wireless LAN (WLAN) as a carrier forthe IP WAP embodiment. The RC vehicle may then beidentified using a standard mobile LAN addressing scheme.The control of the remotely guidedground vehicles 120,122, 130 andair vehicles 520, 522, and 530 becomes anexercise in networking domain instead of signal domain.Interference from other radio band signals becomes a non-issue.WLAN is typically established as an infrastructureinside a building or community, and its wide bandwidth andabundance of points of presence (POP) makes it ratherdifficult to jam. In addition, WLAN hubs have a range of300 meters, and fully repeatable, resulting in far greaterrange than traditional radio frequency carriers.
As an additional measure, WLAN inherently incorporatesencryption. Security and integrity of control and feedback signals are enhanced by encryption and the ability to doauthentication is also possible. Since WLAN is a fullynetworked infrastructure, a single controller may controlmultiple vehicles, and vice versa, resulting in fullflexibility in controlling locomotion of the vehicle. Inaddition to using basic human control, each controller canbe aided by artificial intelligence to provide steeringassistance. The artificial intelligence can reside purelyon the vehicle or the controller, or distributed betweenthe two, depending on response time requirements.
Present wireless LAN standards, especially IEEE 802.11a,provides substantially amount of bandwidth for bi-directionaldata between the vehicle and the controller.The control module need not be equipped with a dedicatedtransmitter/receiver for the vehicle. The control modulewill only require a network connection that is either partof the WLAN, encompassing the WLAN, or connecting to it.
In essence, the control module battery no longer becomes alimiting factor for the range of remote control. The highbandwidth also provides the ability for enhanced sensoryfeedback. The vehicles may also have one or more arms orextensions to allow the vehicle to interact with itsenvironment outside of the general confines of the vehicle.The arm or extension may have tactile, force feedback andthe like. Multiple visual, audio as well as tactilefeedbacks are all possible according to instrumentationalternatives. Advanced instrument panels may be employedto facilitate more embedded levels of control, i.e., deeperthan basic higher-level supervisory control. This enhancedlevel of feedback provides an immersive experience for thehuman controller or supervisor.
FIG. 2 further illustrates the wireless IP network providedby thefirst MIPWAPs 120 where a supervisor interacts with the mobile network viasupervisor interface 102, alocalarea network 104 and an IPwireless access point 213. Inthis example, the IPwireless access point 213 is in directcommunication with thefirst MIPWAP 120 and thefirstMIPWAP 120 is in communication with aMIPWB 122 that ismoving in a clockwise direction about theperimeter 210 ofthe effective range 211of theMIPWAP 120. When theMIPWB122 is beyond theeffective range 211 of theMIPWAP 120,theMIPWB 122 may be handed off to anext MIPWAP 220 of thesame local area network as thefirst MIPWAP 120 or theMIPWB 122 may be handed of to aMIPWAP 222 of a foreignagent and Mobile IP Registration and Discovery proceduresfollow.
FIG. 3 illustrates by way of example the general guidance,navigation and control (GNC) processing 300 of each of theMIPWAP 120,EMIPWAP 130, andMIPWB 122 vehicles. Thesevehicles may be ground vehicles, air vehicles or watercraftor combinations thereof. Thetransceiving interface 302places the incoming IP data into asignal paths 304 thatare in turn placed into astage 306 for steering commands,trajectory commands, or where additional autonomy issupported, goals. Thetransceiving interface 302 receives,from avehicle output stage 308, the vehicle outputs 310 tobe sent to the one or more supervisory levels via thewireless IP network. In thisexample GNC processing 300,the low frequency commands 312, for example steeringcommands, are sent from the trajectory commands/goals stage306 to the low frequency/gross motion controller 314. Thelow frequency/gross motion controller 314 preferably hasexecutable rules and steps executed within amicroprocessor. Also in thisexample GNC processing 300,thegoals 316 and low frequency commands 316 are sent fromthe trajectory commands/goals stage 306 to theimageprocessing 318 than may include artificial intelligence processing. The artificial intelligence processing mayinclude expert systems, fuzzy logic, adaptive neuralnetworks, genetic algorithm, goal seeking algorithms, andany combination thereof. The image processing may includescene matching, edge matching, centroiding, constellationmatching, projected three-dimensional image matching, andany combinations thereof.
Theimage processing 318 sends low frequency commands 318such as steering and velocity magnitude to the lowfrequency/gross motion controller 314. The lowfrequency/gross motion controller 314 harmonizes orotherwise performs arbitration between the image processinglow frequency commands 318 and the trajectorycommands/goals stage low frequency commands 312. The rulesof harmonization and arbitration may be preloaded in thelow frequency/gross motion controller 314, sent asoverrides via thetransceiving interface 302, and may bemodified based on the situation awareness provided by theimage processing 318. Where gross motion of the vehiclerequires correction, the low frequency/gross motioncontroller 314 sends low frequency commands 326 toeffectors 329. For example, in the case where steeringchanges are required, the low frequency commands 326 directtheeffectors 328 to produce a direction heading change inthe vehicle. The low frequency/gross motion controller 314also sendscommands 315 to the high frequency/fine motioncontrollers 322. High frequency control may includeangular rate control of an air vehicle, for example, or thesuspension control of a land vehicle. Fine motion controlmay include the control one or more extendable arms of thevehicle, for example, and may include the pan and zoom ofimage sensors of thesensor suite 338. The highfrequency/fine motion controllers 322send command signals330 to theeffectors 329 to produce rate, angle or position changes in the effectors.
The effectors are preferably a distributed suite of motorsor other actuators, including electromagnetic andpiezoelectric devices, capable of changing the forces onthe vehicle either directly or by drawing on theenvironment in which the vehicle operates. For landvehicles, this may be a change in the orientation of one ormore wheels in contact with a friction surface or adifferential torque change in a tracked vehicle. Forwatercraft, this may be a change in rudder deflection andfor air vehicles, this may be a change in rudder and mayinclude changes in elevator deflection for example. Theeffectors may instrumented providing for the sensingchanges in theeffectors 329, for example changes in rudderdeflection, that may sent 330 the low frequency/grossmotion controller 314, the high frequency/fine motioncontrollers 322, thevehicle output stage 308. Theresulting changes in the status of theeffectors 330 thenproduces one or more changes in the location andorientation of the vehicle and its relationship with itsimmediate environment 332. Theseimmediate environment 332of the vehicle is sensed via a sensorsuite having sensors340 andsensor processing 338. Thesensors 340 may measurevehicle angular rates, vehicle linear acceleration, and mayinclude image sensors and other sensors not already part ofthe instrumentedeffectors 329. The sensor suite mayinclude absolute position and velocity sensors such as aGlobal Positioning System (GPS). The high frequencyfeedback signals 344 from thesensor processing 342 of thesensor suite 338 is sent to the high frequency/fine motioncontrollers 322 and may be sent to the images processing318 and thevehicle output stage 308. Thelow frequencyfeedback 346 is sent to the low frequency/gross motioncontrollers 314 and may be sent to the images processing 318 and thevehicle output stage 308. The processed sensedsignals that comprise thevehicle state feedback 348, forexample vehicle location, direction, velocity andorientation, may be sent to atrajectory estimator 350 thatprovides a refinedvehicle state estimate 352, and mayprovide trajectory state projections, to thesignalprocessing 318 particularly where it includes or isreplaced by artificial intelligence processing, and may beprovided to thevehicle output stage 308. Other groundvehicle commands 315 may include forward, reverse and parkor hold position commands, decelerating and acceleratingcommands. Other vehicle status indicators that may besensed include battery level, weight of total load orestimated payload, actual acceleration, speed, vibrationlevels, outside, cabin payload and engine temperatures,ground inclination, i.e., local level, and audio levels viamicrophones.
FIG. 4 illustrates the mobile group roaming and station-keepingof the semi-autonomous vehicles being mobile IPnodes about the autonomous or semi-autonomous vehiclesbeing a mobile IP wireless access point. FIG. 4 furthershows the exploitation of the autonomous or semi-autonomouscapability of each of theMIPWAP 120,EMIPWAP 130, andMIPWB 122 vehicles made possible by theirrespective GNCprocessing 300. In this example, anMIPWAP 120 is incommunication with a the stationary IPwireless accesspoint 114 operably connected to theInternet 106 and itssupervisor via anIP gateway router 110 of a foreign agentvia Mobile IP. In this example, theMIPWAP 120 is incommunication with and providing packer forwarding for twoMIPWBs 122. While theMIPWAP 120 may have aneffectivewireless range 211 andperimeter 210, the supervisory goalmay be to maintain theMIPWB 122 vehicles within a highlyeffective range 411 andperimeter 410. Accordingly, goals and steering commands either separately or collectively maybe sent to each of theMIPWB 122 vehicles to maintain thehighlyeffective range 411 andperimeter 410 of theMIPWAP120. In addition, to bound the autonomous nature of theMIPWB 122 vehicles provided by their GNC, theMIPWAP 120generate or pass on relative station-keeping goals orparameters in order to maintain a constellation, forexample, or to prevent oneMIPWB 122 vehicle from collidingwith another, as another example.
FIG. 5 illustrates the mobile group constellation andstation-keeping of the semi-autonomous air vehicles beingmobile IP nodes about the autonomous or semi-autonomous airvehicles being a mobile IP wireless access point and thecooperation with a cellular base station and an enhancedautonomous air vehicles being both a cellular node and amobile IP wireless access point. FIG. 5 further illustratesby example an air vehicle constellation abovelocal groundlevel 502 where acellular base station 554 transmitscellular signals to anair vehicle EMIPWAP 530 within theeffective cell perimeter 506 of thecellular base station554. The effectivecellular range 506 is typically longerthan theIP WAP range 511 of theEMIPWAP 530. TheEMIPWAP530 converts the received cellular signals to data packetsand forwards the data packets to anair vehicle MIPWAP 520having twoair vehicle MIPWBs 522 within theperimeter 560of theeffective range 561 of theair vehicle MIPWAP 520.My exploiting the autonomous or semi-autonomouscapabilities afforded theMIPWBs 522, theair vehicleMIPWAP 520, and theair vehicle EMIPWAP 530 by their on-boardGNC processing, theMIPWBs 522 may be given goals toremain within a veryeffective range 571 andperimeter 570of theair vehicle MIPWAP 520. In addition, theMIPWBs 522may be given goals or parameter to maintainstations 550,560 relative to theair vehicle MIPWAP 520 and the stations may be used to enhance the reconnaissance andforce effectiveness of the constellation and may be used tominimize in air collisions. Thecellular base station 554may be replaced by a radio tower and transmitter or otherRF transmitting assembly providing encoded signals to anMIPWAP 530 enabled to receive the encoded signals andtranslated the signals to data packets for subsequent IPWAP transmissions to theair vehicle MIPWB 522
For the cellular base station embodiments, the system maybe described as a distributed remote control system withwhich an operator may remotely navigate a vehicle accordingto one embodiment of the present invention. The remotecontrol system of the cellular embodiment comprises one ormore user interfaces represented by the plurality ofoperator interfaces 102, a communications network includinga digitalcellular system 156, 154, and the remotely guidedground vehicles 120, 122, 130 andair vehicles 520, 522,and 530 subject to the guidance information transmittedfrom the operator interface through the digital cellularsystem. The preferred communications network in thisembodiment comprises at least two underlying networkmodalities including a distributed, packet-switching (DPS)network and a wireless, circuit-switched communication(WCSC) network.
In the embodiment having the preferred communicationnetwork, the DPS comprises one or more nodes utilizingpacket protocols that permit the communication of controland sensory signals over an extended geographic regionseparating theoperator interface 102 from the remotelyguidedground vehicles 120, 122, 130 andair vehicles 520,522, and 530. The network in the preferred embodiments istheInternet 106, and generally relies on a protocol stacksuch as TCP/IP to transmit data packets between various nodes, although other forms of networks such as token ringand ATM would also be suitable. Use of the term Internetherein generally encompasses a system including one or moremetropolitan area networks (MAN), wide area networks (WAN),or local area networks (LAN) necessary to communicatesignals between theoperator interface 102 and remotelyguidedground vehicles 120, 122, 130 andair vehicles 520,522, and 530.
In the preferred cellular embodiments, the WCSC networkcomprises one or more radio frequency accesses pointsthough with a user may exchange navigational and sensorysignals using existing communications infrastructures.Preferably, the WCSC network includes the well digitalcellular system 154, 156 operating in concert with thePSTN150. The digital cellular system comprises a series ofradio frequency transceivers, namelybase stations 154,through which various mobile stations may connect withinthe regions of coverage, i.e., cells. The base stationsare connected to a mobile switch center (MSC) 156 forregistering caller mobile stations and tracking theirlocation.
Theuser interface 102 may manifest itself in any number ofdifferent forms. The first representative user interfacemay consist of a personal computer, workstation, mainframe,internet applicant, personal digital assistant, or tabletcomputer, for example. A representative computer (notshown) generally comprises a central processing unit (notshown) with random access memory (not shown), data inputdevice such as a keyboard (not shown), graphical displaysuch as a monitor (not shown), and an input/output device(not shown) operatively coupled to theInternet 106.Operating on the computer (not shown) is an operatingsystem and one or more software programs including a graphical user interface (GUI) for interacting with a userand conveying visual or symbolic information related theposition, orientation, and operation status of the remotelyguidedground vehicles 120, 122, 130 andair vehicles 520,522, and 530, generally RC vehicles.
Theinterface 102 in some embodiments further includes ajoystick or other manual position-sensitive controller forinputting navigation or guidance signals. In someembodiments, the joystick includes force feedback and/ortactile response.
In practice, the remote controlled vehicle requires awireless LAN transceiver, network protocol stack, controllogic and sensors. Each MIPWB vehicle operates like amobile LAN endpoint on the WLAN. The mobility can be builtinto either layer 2 or layer 3 of the network protocolstack. In these example embodiments, the invention usesmobile IP as the carrier protocol. The controller has awell-known network address for all of the RC vehicles underits control. Upon powering up, each of the RC vehiclesauthenticates itself with this controller. Effectively,each of the vehicles "signs in" with the main controller.
A simple "hello" and timeout protocol keeps the connectionalive between vehicle and controller. Once the connectionis established, the RC vehicle identifies itself with thecontroller and supplies its model number, capabilities, andcurrent configuration. The controller may download newconfiguration to the vehicle if necessary. Once the newconfiguration is downloaded, the controller may enablelocomotion and start steering or giving goals to the RCvehicle. The vehicle will start monitoring its sensors andsend the information back to the controller. For example,the vehicle may be equipped with a digital camera and thevideo feed is constantly sent back to the controller via WLAN. When the vehicle crosses to a different WLAN domain,it will re-establish itself with the controller through theauthentication process again, preferably using Mobile IP.
The following features and/or combinations of features alsoconstitute advantageous embodiments of the described and/orclaimed invention:
- The claimed and/or described ground vehicle whereinthe wireless network interface receives the one or moreexternally generated commands as data packets according toTransmission Control Protocol/Internet Protocol;
- The claimed and/or described ground vehicle whereinthe one or more externally generated commands comprise oneor more navigation goals from one or more supervisoryprocessing interfaces;
- The claimed and/or described ground vehicle whereinthe one or more externally generated commands comprise oneor more steering commands one or more supervisoryprocessing interfaces;
- The claimed and/or described ground vehicle whereinthe wireless network interface transmits the measured oneor more ground vehicle states to the one or moresupervisory processors;
- The claimed and/or described ground vehicle whereinthe wireless network interface further comprises processingfor data packet forwarding;
- The claimed and/or described ground vehicle whereinthe wireless network interface further comprises a wirelessnetwork access point;
- The claimed and/or described ground vehicle whereinthe sensor suite further comprises a visual band sensingelement;
- The claimed and/or described ground vehicle whereinthe infrared processing means of signals comprises patternrecognition processing.