US20120157263A1 - Multi-user smartglove for virtual environment-based rehabilitation - Google Patents

Abstract

A low-cost, virtual environment, rehabilitation system and a glove input device for patients suffering from stroke or other neurological impairments for independent, in-home use, to improve upper extremity motor function, including hand and finger control. The system includes a low-cost input device for tracking arm, hand, and finger movement; an open source gaming engine; and a processing device. The system is controllable to provide four types of multiple patient/user interactions: competition, cooperation, counter-operative, and mixed.

Description

CROSS REFERENCE TO RELATED APPLICATIONS
• Priority to U.S. provisional patent application No. 61/145,825 entitled “Multiple User Virtual Environment for Rehabilitation (MUVER)”, which was filed on. Jan. 20, 2009, and U.S. provisional patent application No. 61/266,543 entitled “Low Cost Smart Glove for Virtual Reality Based Rehabilitation”, which was filed on Dec. 4, 2009 is claimed.
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
• N/A
BACKGROUND OF THE INVENTION
• 1. Field of the Invention
• A device and system for rehabilitating hand and finger movements of stroke patients with neurological or orthopedic problems is disclosed, and, more specifically, a device and system that are structured and arranged to capture hand and wrist motion for the purpose of guiding a patient/user through rehabilitation exercises.
• 2. Summary of the Related Art
• Every year, between 700,000 and 800,000 Americans suffer a new or a recurring stroke in which a sudden variance in blood supply to the brain causes loss of brain function. About 150,000 Americans die from the event, leaving approximately 650,000 who must overcome or learn to live with the effects of any permanent or long-term disability. These effects may include an inability to return to work, which can lead to a loss of income and benefits such as health care, and/or the need for daily living assistance. Thus, stroke remains the leading cause of disability among adults in the United States.
• Eighty-five percent (%) of stroke patients, at least initially, suffer from physical disabilities that prevent them from performing normal functions and/or from returning to work. Moreover, six months after a stroke, 55-75% of survivors still experience limited upper extremity (UE) function. Indeed, in instances with initial UE paralysis, complete motor recovery has been reported in less than 15% of the cases.
• Despite these disappointing statistics, studies have shown that patients with chronic stroke have the potential to improve their physical ability and independence through rehabilitation. Indeed, stroke patients generally appear to be highly motivated to make further gains once conventional, therapist-assisted or therapist-supervised rehabilitation (“rehab”) has been completed. Rising health care costs, however, are causing stroke patients to be discharge from hospital sooner and causing the length of physical therapy sessions to be shorter.
• As a result, a rehabilitation program that can be performed in a stroke patient's home and performed without a visiting therapist being physically present in the patient's home could further increase rehab participation. Such a program saves time in transportation and cost for clinical fees.
• A key component of poor or incomplete functional recovery remains the impaired use of the stroke patient's hand and fingers. Critical motions of the hand and wrist have been determined to be gross finger flexion and extension, opposition of the thumb, radial and ulnar deviation, supination and pronation of the wrist, wrist flexion and extension, and the hand's position and orientation in space. Accordingly, there is a compelling need to improve available methods for UE rehabilitation in stroke patients, and in particular, methods that improve hand and finger function. Hand rehabilitation devices that are designed to be low-cost for in-home use remain a viable option for continuing the rehabilitation of the increasing number of stroke patients in the United States. However, hand rehabilitation devices by others tend to be complex, expensive, and/or not readily available to clinicians.
• There are many products currently available on the market that can achieve these or similar goals and that form the prior art. These include the Cyberglove, P5 Glove™, 5DT Dataglove, Acceleglove, Hand Mentor, and HandTutor. Other products of interest that are not necessarily expressly for rehabilitation but that contain several of the critical factors that can be incorporated into this design include the Nintendo® Wii™. Some of these products, such as the Cyberglove, are very precise, but too costly for in-home purposes ($10,000 per Cyberglove). Others are more affordable, e.g., the P5 Glove™ (approximately $100 per glove), but, without modification, do not deliver data accurately enough for the intended application. Hence there is tension and a necessary trade-off between cost and performance.
• FIG. 1 shows the P5 Glove™. The P5 Glove™10 was released for the home personal computer (PC) video game market in 2002 by Essential Reality, LLC of New York, N.Y. Although it is not wireless, the P5 Glove™10 is lightweight and allows for six degrees of tracking freedom including the three translational axes (the x-, y-, z-direction) and the three rotational axes (yaw, pitch, and roll).
• The P5 Glove™device10 uses infrared (IR) technology, e.g., light-emitting diodes (LEDs), and bend sensors, to track movement of the patient/user's hand and/or fingers.Bend sensors9, which are discussed in greater detail below, are adapted to measure the bend or flexion of a digit, e.g., a finger or toe. Thebend sensors9 are disposed along the back of each finger and thumb, to provide independent finger and thumb measurements. Thebend sensors9 are mechanically coupled to the patient/user's hand by aring8 that fits around each fingertip.
• With the P5 Glove™p010, IR sensors are structured and arranged to determine three-dimensional (3D) positioning. However, conventionally, the P5 Glove™ device10 is electrically coupled to an infrared tower (or receptor) (not shown), e.g., via a PS/2 cable (not shown). The infrared tower, in turn, is electrically coupled to a PC (not shown), e.g., via a USB cable (not shown). Although this arrangement makes it easy to use at home, because of the infrared control receptor, the work space is limited, which is to say that the range within which motion can be detected is limited to only about three or four feet between the glove and the receptor.
• Typically, there are three ways in which IR sensors can be used to determine position. The first way involves a single IR LED that is disposed at a pre-set, known position. A sensor is disposed on the tracking object. Based on the angle and intensity of the sensed light, the position of the sensors relative to the LED can be determined. A second way in which IR light can be used to determine position is by disposing a single, IR light detecting sensor at a known position and by moving objects that emit IR light relative to the sensors. Finally, in a third technique, the LED and the IR detecting sensors are disposed proximate each other. IR light from the LED reflects off of objects within the illumination area. The sensor picks up the reflected light, from which the position of the reflecting object can be determined. Many multi-touch tables such as the Microsoft® Surface utilize reflective infrared technologies.
• Although infrared positioning is accurate and relatively inexpensive it is not the most useful or most accurate method of determining the position of a P5 Glove™. Indeed, because IR light detection is predicated on beams of light traveling between an emitter and a sensor, obstructions to the beam path limit this capability. Consequently, because a patient/user's hands move in many directions and at many angles there is no guarantee that emitted IR beams will reach the sensor without being obstructed or reflected.
• FIG. 2 shows a Hand Mentor Rehabilitation Device11 (“Hand Mentor”). The Hand Mentor11 is manufactured by Columbia Scientific, LLC of Tucson, Ariz. and has been cleared by the Food and Drug Administration (FDA) for use in rehabilitation clinics.
• The Hand Mentor11 encourages patients to restore the range of motion of their wrist and hand using the principles of Repetitive Task Practice (RTP) and Constraint Induced Therapy (CI). As shown inFIG. 2, the hand of the patient fits into asleeve12 that is adapted to sense and to generate a signal commensurate with the level of resistance caused by flexor spasticity.
• Thedevice11 offers three different program types that are adapted to reduce spasticity, to recruit specific muscle groups, and/or to improve motor control. The resistance signals are transmitted to aprocessing device13 that includes software (or, alternatively, is hard wired) that is designed for unsupervised patient use of thedevice11. Advantageously, thedevice11 can also offer a therapist option, to establish rehabilitation regimens and generate data for documenting and reporting the patient/user's progress.
• Referring toFIG. 3, SensAble Technologies (ST) of Woburn, Mass. manufactures a line ofhaptic input devices14, which are designed to gather motion input and to provide feedback to the patient/user's fingers, hand, and arm. TheST device14 most suited for use in a rehabilitation virtual environment is a six degree-of-freedom PHANTOM® SensAble model, in which patients/users grasp a pen- or pencil-like portion15 of thedevice14 in either hand and control the x-direction, y-direction, z-direction, roll, pitch, and yaw.
• The PHANTOM® six degree-of-freedom device14 interfaces with a PC (not shown) via a parallel port (not shown). Thedevice14 typically comes bundled with several software demos and with a software development kit specific to the inputs and limitations of the device.
• The Falcon from Novint Technologies, Inc. of Albuquerque, N. Mex. is shown inFIG. 4. TheFalcon device16 was originally designed as an input device for playing games on a PC. Thedevice16 has three degrees-of-freedom; however, different grips with a plurality of buttons or dials can be added to provide more degrees of freedom.
• TheFalcon device16 includes a 4″×4″×4″ workspace and has a two-pound (force) capability. TheFalcon device16 interfaces with a PC, e.g., using a universal serial bus (USB), e.g., aUSB20. TheFalcon device16 is sold with several games already available for it as well as driver software to play PC games.
• The Wii™ manufactured by Nintendo Company, Limited of Kyoto, Japan was released in the Fall of 2006 as a personal video gaming console. Referring toFIG. 5, theWii™ controller17 has two input device components: a WiiRemote™ controller18 and aNunchuk™ controller19. The Wii™Remote™ controller18, when used solely, is normally held in a player's dominant hand. However, players choosing to use both theNunchuk™ controller19 and the Wii™Remote™ controller18, usually hold the Wii™Remote™ controller18 in the right hand and theNunchuk™ controller19, which is electrically coupled to the Wii™Remote™ controller18, in the player's left hand.
• The Wii™ gaming console (not shown) connects directly to a power source and to a television or other display device. Each Wii™Remote™ controller18 and each Wii™Nunchuk™ controller19 communicates with the gaming console wirelessly via a sensor bar (not shown), e.g., using Bluetooth wireless technology. Although a large library of publicly-available Wii™ games exists, presently, there is no licensed software development kit available to the public.
• Referring toFIG. 6, a Rutgers Master II-NDForce Feedback Glove20 is shown. Theglove device20 was developed in 2002 at Rutgers University and is structured and arranged to use a plurality of, e.g., four,pneumatic actuators21 and a plurality of sensors. The direct-drive configuration of theactuators21 provides force to the tips of thefingers23 via finger rings24 that are that mechanically-coupled to theactuators21. Sensors are disposed on the patient/user'spalm22, to avoid the presence of wires at thefingertips23. The Rutgers Master II-ND device20 is a research only device and there is no indication of software or a software development kit.
• Referring toFIG. 7, aCyberGlove II device25 manufactured by Immersion Technologies of San Jose, Calif. is shown. TheCyberGlove device25 is one of the leading products for sensing and capturing motion in the current market. Thedevice25 is made from lightweight elastic and each sensor is extremely thin and flexible, making the sensors virtually undetectable. The fabric on top of thedevice25 is a stretch material that is provided for comfort. The fabric on the bottom of thedevice25 is made of an open or mesh material for better ventilation.
• Thedevice25 is wireless and has a capacity of making eighteen or twenty-two high-accuracy, joint-angle measurements. Theglove25 uses a proprietary resistive bend-sensing technology to capture real-time digital joint-angle data. The 18-sensor model includes two bend sensors that are disposed on each finger, four abduction sensor, and sensors for monitoring thumb crossover, palm arch, wrist flexion, and wrist abduction. The 22-sensor model includes a third bend sensor for each finger.
• TheCyberGlove II device25 is electrically coupled to a PC, e.g., using a wireless USB receiver. The software that come bundled with theglove25 is for evaluation purposes only and is not for virtual reality. The is no publicly available software development kit for theCyberGlove device25.
• The 5DTData Glove device26 shown inFIG. 8, is designed for use in motion capture and animation. Thedevice26 material is stretch Lycra® and the fingertips are exposed to facilitate the grasping function.
• Thedevice26 is adapted to sense multiple bends, e.g., finger flexion, but is unable to measure the attitude or orientation of the hand in space or with respect to the patient/user's body. Thedevice26 features automatic calibration and has an on-board processor (not shown).
• Two 5DT versions are commercially available: the5DT Data Glove 5 Ultra device, which includes five bend sensors to measure discrete finger and thumb flexure, and the5DT Data Glove14 Ultra device (depicted inFIG. 8), which uses two bend sensors on each finger and thumb and one bend sensor per abduction between adjacent digits.
• The5DT Data Glove5Ultra device26 is adapted to include Bluetooth technology to make it wireless and, also, can include a cross-platform SDK. The bundled software that comes with thedevice26 has no rehabilitation applications.
• Combinations of the commercially-available prior art technologies shown inFIGS. 1-8 have also been investigated by others for hand rehabilitation purposes. For example, the Rutgers Hand Master I and Hand Master II (FIG. 6) have been used in combination with a Cyberglove™ (FIG. 7) to improve hand function in stroke patients. The system uses the palm-mountedpneumatic pistons21 and virtual reality to improve resisted finger flexion and non-resisted finger extension.
• Robotic devices that train the entire arm, such as the MIT-Manus, have also been shown as benefit for stroke patients. More recently, the Bi-Manu-Track robotic arm trainer has been found to be equally as effective as electrical stimulation training. A study utilizing the Howard Hand Robot found greater mobility gains for stroke subjects who exercised with robotic assistance in virtual reality during a relatively longer, e.g., three-week, intervention in comparison with subjects who had robotic assistance only during the last week-and-a-half of training.
• A reported study by Fischer et al. concluded that there was no difference between three groups of stroke subjects who trained on a reach-to-grasp task in virtual reality with and without two different types of robotic assistance to finger extension during the training. Finally, a pilot study performed with a new Finger Trainer robotic device found some improvements in active movement and less development of spasticity in comparison with a control group that received bimanual therapy. However, this Finger Trainer was designed to perform passive finger movement only.
• None of these devices, however, meets the need for a low-cost, simple, UE motor training device that patients could use easily in their homes, and, potentially, use with other patients over a network, the Internet, and the like. Presently, few virtual environment-based (VE-based) or robotic systems for hand rehabilitation exist. Those that do exist are prohibitively expensive, and most are not commercially available. Moreover, none is suitable for independent home use by patients and, furthermore, none provides for multiple patient/user interaction over the Internet.
• Hence, it would be desirable to provide a low-cost device that stroke or other patients could independently use in the home, to improve UE function and especially improved UE function of the hand. Such a device would also be useful as an adjunct to ongoing rehabilitation therapy, providing patients with an interesting and motivating way to perform a home exercise program. If designed appropriately, such a system could be used by a therapist to establish exercise programs that were adjustable in level of difficulty, and tailored to the patient's specific interests. These features would likely increase patient motivation and compliance.
• It would also be desirable to facilitate interactions with other patient/users over a local or a wide area network, the World Wide Web, the Internet, and so forth to make practice more fun and to enhance motivation. Such virtual interactions may also alleviate feelings of social isolation in patients who remain housebound due to mobility problems.
SUMMARY OF THE INVENTION
• To meet the apparent need, a Multiple-User Virtual Environment for Rehabilitation (MOVER) is disclosed. The MUVER is structured and arranged to enable multiple patients and system users at remote locations to interact with each other in virtual space with activities designed to enhance UE and skilled-hand function. The intended application is for use as a supplemental, in-home rehabilitation tool for people with hand function and coordination disabilities, specifically the type of disability that would result from a stroke. Advantageously, MUVER will be the first inexpensive, VE-based system that patients could purchase, e.g., for home use, that is specifically designed to enhance finger and thumb movement in addition to arm movement.
• The MUVER system is flexible enough to include a variety of different rehabilitation devices to control the MUVER software. For example, the MUVER system is structured and arranged to monitor force and torque produced by the hand and fingers during grasping and manipulation tasks and can be extended to control ankle movements.
• The system includes a virtual reality game-type interface that will have “scenes” developed specifically for patients with stroke who need to practice finger, hand, and arm movements. The activities will be functional movements that involve the whole arm as well as hand, but with specific emphasis on hand and finger motions. Feedback features and training routines, based on principles of motor learning, facilitate motor recovery in patients at different levels of motor ability.
• The device and system uniquely combine an ability to track hand position and orientation in space with tracking of finger and thumb configuration using an input device. This feature combination is critical to using the device to display a wide variety of hand and upper extremity exercises in virtual reality displays.
• One such input device is fashioned like a glove for use with a multiple-user virtual environment system for rehabilitation exercise of a human hand and digits. The input device is structured and arranged to generate signals corresponding to at least one of a discrete movement and an attitude of said hand and said digits. The device includes a glove that can be readily donned and doffed on either hand by a user; a first plurality of sensors, each sensor being structured and arranged to provide data on movement and range of movement of at least one of the index finger, the middle finger, and the ring finger; a second plurality of sensors that is structured and arranged to provide data on movement of the thumb; and a positioning and tracking system that is structured and arranged to generate position coordinates in three rotational axes and three translational axes to determine at least one of the attitude and a velocity of said hand.
• Another unique feature will be feedback lights placed on the back of the hand which will allow the patient to know if they are performing the correct motion while looking at their hand, as opposed to the screen. This will allow patients with impaired perceptual abilities to concentrate on the task while not having to interact as much with a computer interface.
• It may also have leisure applications to, in particular gaming. The rehabilitation exercises will essentially be mini-games, so the device could easily be adapted for non-rehabilitation related virtual gaming.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
• The foregoing and other objects, features, and advantages of the invention will be apparent from the following more particular description of preferred embodiments of the invention, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the invention.
• FIG. 1 shows a view of a P5 Glove;
• FIG. 2 shows a view of a Hand Mentor Rehabilitation Device;
• FIG. 3 shows a view of SensAble Phantom® devices;
• FIG. 4 shows a view of a Novint Falcon device;
• FIG. 5 shows front and side views of a Wii™ remote control and a view of a Wii™ Nunchuck™ device;
• FIG. 6 shows a view of a Rutgers. Master II-ND Force feedback Glove device;
• FIG. 7 shows a view of a Cyberglove™ II device;
• FIG. 8 shows a view of a5DT Data Glove5 Ultra device;
• FIG. 9 shows a schematic of the multi-user virtual reality rehabilitation (MUVER) system in accordance with the present invention;
• FIG. 10 shows schematics of four common multi-user virtual interactions;
• FIG. 11 shows a back side of a glove input device having bend sensors disposed on finger portions of the glove in registration with the index, middle, and ring fingers;
• FIG. 12 shows bend sensors disposed on the glove ofFIG. 11 in registration with the thumb and the base of the wrist;
• FIG. 13 shows bends sensors for capturing wrist flexion/extension and for tracking radial and ulnar deviations and an IMU;
• FIG. 14 shows and arrangement of electroluminescent wire light-emitting devices for providing visual signals to a patient/user;
• FIG. 15 shows two views of a second glove input device embodiment;
• FIG. 16 shows an IMU and a processing unit for the glove inFIG. 15;
• FIG. 17 shows an embodiment of an input glove device using Hall effect sensing;
• FIG. 18 shows the palm portion of the input glove device shown inFIG. 17;
• FIG. 19 shows the back of the had portion of the input glove device shown inFIG. 17;
• FIG. 20 shows and arm sleeve embodiment of a glove input device;
• FIG. 21 shows a schematic of a hardware interface in accordance with the present invention;
• FIGS. 22A and 22B show embodiments of a banana grip base structure;
• FIG. 23 shows an embodiment of a globe base structure;
• FIGS. 24A and 24B show embodiments of teardrop-shaped base structures;
• FIGS. 25A and 25B show embodiments of pyramid base structures;
• FIG. 26 shows a programming schematic for scripting a virtual reality scene;
• FIGS. 27A-27D show graphics of four stages of an exemplary virtual environment;
• FIG. 28 summarizes the mean and standard deviations of various testing sub-phases shown inFIGS. 27A-27D;
• FIG. 29 shows another schematic of the multi-user virtual reality rehabilitation (MUVER) system in accordance with the present invention;
• FIG. 30 shows top and bottom portions to a ring prototype;
• FIG. 31 shows the top and bottom portions ofFIG. 30 assembled;
• FIG. 32 shows an IMU disposed in the assembled ring prototype;
• FIG. 33 shows the ring prototype with a plurality of bend sensors;
• FIG. 34 shows a knuckle plate for the prototype ofFIG. 33;
• FIG. 35 shows a ring prototype mechanically coupled to a knuckle plate;
• FIG. 36 shows an exemplary virtual environment scene for a single degree of freedom knob;
• FIG. 37 shows an exemplary virtual environment scene for a single degree of freedom hand device;
• FIG. 38 shows an exemplary virtual environment scene for an active hand device;
• FIG. 39 shows an embodiment of a SmartGlove input device;
• FIGS. 40A and 40B show MCP flexion/extension set-ups for 45 degrees and 90 degrees, respectively;
• FIGS. 40C and 40D show PIP flexion/extension set-ups for 45 degrees and 90 degrees, respectively;
• FIG. 41A shows an illustrative bar graph of MPC bend data for the input device shown inFIGS. 40A and 40B;
• FIG. 41B shows an illustrative bar graph of MPC bend data for the input device shown inFIGS. 40C and 40D; and
• FIG. 42 shows a schematic of a bi-manual SmartGlove system with arm splints for neutral wrist position and support.
DETAILED DESCRIPTION OF THE INVENTION
• U.S. provisional patent application No. 61/145,825 entitled “Multiple User Virtual Environment for Rehabilitation (MUVER)”, which was filed on Jan. 20, 2009, and U.S. provisional patent application No. 61/266,543 entitled “Low Cost Smart Glove for Virtual Reality Based Rehabilitation”, which was filed on Dec. 4, 2009 are incorporated herein in their entirety.
• As mentioned above, the increasing number of stroke patients necessarily recovering in their home is stoking the need for inexpensive, in-home, hand rehabilitation devices. The devices described above in the background section provide a basis for a variety of options for rehabilitation with the correct software and therapist-approved regimen. The NU-MUVER (Northeastern University Multiple-User Virtual Environment for Rehabilitation) system has been developed by Northeastern University of Boston, Mass. to meet the need for an inexpensive device that can be used to rehabilitate hand and finger movements of stroke survivors and other patients experiencing neurological or orthopedic problems. The NU-MUVER system is designed to be used at home and/or over a network, e.g., a LAN, a WAN, the World Wide Web, the Internet, and the like, alone or with others, e.g., a therapist, other patients, and so forth.
• The NU-MUVER consists of three basic components: an input device that generates data on position, attitude, and orientation of the patient/user's hand in space as well as of individual finger and thumb movements; commercially-available graphics software that provides object and animation routines that can be used to construct various movement re-training scenes; and a control unit that includes control software that enables a networking capability, movement parsing, performance scoring, recording, storage, manipulation, and display of data; and multiple training “scenes” that are designed to facilitate the practice of a particular movement(s) that is/are therapeutic for discrete patient populations.
• Referring toFIG. 9, aMUVER system90 in accordance with the present invention is shown. Thesystem90 is structured and arranged to provide a plurality ofvirtual environments100 designed for specific rehabilitation exercises and for multiple patients/users91 to interact with others, and withthird parties96, e.g., medical personnel, physical therapists, and the like.Virtual environments100, or worlds, that are designed for more than one patient/user91 are called Multi-User Virtual Environments (MUVE). Because the instant MUVE is for rehabilitation, thesystem90 is referred to as a “Multi-User Virtual Environment for Rehabilitation” orMUVER90. The elements of theMUVER90 are shown in the figure and are discussed in greater detail below.
• TheMUVER system90 is designed to be modular, which is to say that the number of patients/users91 and the size of the virtual environment en gross or of each discrete, individual or personalvirtual environment100 can vary and, moreover, can be easily changed. To facilitate modularity further, each patient/user91 is equipped with his/her own personal computer (PC)93 on whichMUVER software94 is installed. As shown inFIG. 9, aninput device92, thePC93, and thesoftware94 define each personalvirtual environment100. Theinput device92 in each individualvirtual environment100 is adapted to enable each patient/user91 to interact with other patients/users91, athird party96, and the like.
• Communication from and between personalvirtual environments100 takes place over and through avirtual environment network95, e.g., a LAN, a WAN, the World Wide Web, the Internet, and the like. This approach differs appreciably from other virtual environments in which a dedicated server operates the virtual environment for each of the patients/users. This feature facilitates recording and logging communications between thevirtual environment network95 and a third party'scomputer96 for later evaluation.
The Virtual Environments
• The design of a unique virtual environment has several stages. The first stage is to use the nature of the patient/user's disability to determine what rehabilitation exercises or movements would be appropriate and feasible to emulate in a virtual environment. Preferably, the rehabilitation exercises or movements are selected by a physical therapist, a physician, a medical specialist, and the like.
• Once appropriate rehabilitation exercises have been chosen, the next step is to choose the character of multi-user interaction. Once again, preferably, the character of interaction is determined by a physical therapist, a physician, a medical specialist, and the like. Common types of multiplayer or multi-user virtual interaction, e.g., competitive interaction, counter-operative (versus) interaction, cooperative interaction, mixed interaction, and any other combination of the first three interactions, are illustrated in theFIG. 10.
• “Competitive interaction” occurs where each patient/user91 of a plurality of patients/users91, who have no direct interaction between them, completes the same task having the same goals for which a comparative score can be assigned. “Counter-operative” or “versus interactions” occur where a first patient/user91 works against a second patient(s)/user(s)91 to achieve competing goals, which only one of the patients/users91 can obtain. “Cooperative interaction” occurs where two or more patients/users91 work jointly to complete a common goal or task. “Mixed interactions” occur where patients/users91 work together to complete a common goal but the performance of each patient/user91 is scored comparatively.
• Once a desired virtual interaction is selected, the setting of the individualvirtual environment100 can be established. Preferably, the setting is simple and appropriate for the discrete patient/user91 and, moreover, is designed to make the goal(s) clear for each patient/user. After thevirtual environment100 is completely designed, rigorous testing for both interaction and substance is necessary.
• TheMUVER system90 includes three components: aninput device92, a graphic display device, and a controller. For the purpose of illustration and not limitation, theMUVER system90 will be described in terms of a SmartGlove™ as the input device, a Panda3D graphics engine for the graphics display device, and specialty software and driver programs for controlling thesystem90. These components are discussed in the subsequent sections. However, brief descriptions of sensors and of positioning and tracking systems are provided.
Sensors
• Sensing devices (“sensors”) are provided to sense movement, e.g., bending, flexion, and so forth. The bend or flexion of a human finger can be measured using various methods, which are collectively referred to as bend sensors.
• Electronic bend sensors use physical geometries and material properties to alter an electrical signal in proportion with angle or pressure. Bend radius and bend angle affect sensor output voltage. Bend sensors have been used for finger position measurement for quite some time, with the first large-scale commercial application appearing in 1989 with the Nintendo® Power Glove. There remains a wide range of currently-available products that use bend sensors, from very simple to very expensive.
• Other types of bend sensors include optical fiber sensors and mechanical measurement devices. Electromechanical sensors provided in, for example, the Nintendo® Power Glove use the patented technology of Abrams Gentile Entertainment Inc. (“Abrams”). Abrams defines five different electromechanical methods for changing the resistance of an electrically-conductive construction based on a bend angle. A first economic application of these technologies involves a resistive sensors having a carbon-based, electrically-conductive ink as a stretched part, which changes electrical resistance in response to applied pressure. Using simple baseline calibration routines, reliable measurements of bend angle are attainable.
• Other similar electromechanical inventions have been patented, including some that modulate capacitance of the sensor in a similar manner. This class of sensors offers better reliability and accuracy than resistive sensors, and is also able to determine the direction of bending rather than just the magnitude of the bend.
• Optical bend sensors typically include a light source that is coupled to a light detector using, for example, an optical fiber. As the fiber bends, less light traverses the length of the fiber due to total internal reflection (TIR). For example, at higher bend angle, relatively few rays strike the detector and more rays exit the fiber at large angles.
• This particular optical technology was used to develop the Data Glove, one of the earliest hand data recording systems, made by VPL Research, Inc. The optical method provides a repeatable measurement of a bend angle; however, it is less cost effective than either of the previously described technologies. Other optical technologies improve on the concept by using multiple fibers in a bundle or by pre-bending the fiber in a certain direction, and are therefore able to measure direction of bend as well as magnitude.
• Finally, a class of angle measurement sensors exists that relies on mechanical means such as the tension of a cable disposed inside a rigid tube, or the relative position of members in an armature. These systems are many, but, in most cases, are better suited to a particular application. Hence, these systems are not inherently flexible. Conventionally, with these systems, changes in the geometry of a mechanical arm assembly are measured. Although such a system could potentially cost less than, for example, a resistive bend sensor, the time spent on design and troubleshooting would likely offset any cost increases. Accuracy also can be quite good, however, greater mechanical tolerances must be controlled for repeatable measurements.
• Hall effect sensors are switches that are activated in the presence of a magnetic field such as generated by a magnetic field-producing device, e.g., a magnet. The sensor contains a capacitor that generates an electrical current and a magnetic field perpendicular thereto. The magnetic charges generated follow a straight line except when in proximity of a magnetic field at which time the path of the charge becomes non-linear, i.e., curves, and accumulates on one face of the sensor. The distance at which the magnetic field causes the sensor to act like a switch is a function of the strength of the magnetic field and, therefore, the magnet, and the current density specified by the sensor.
Positioning and Tracking Systems
• The ability to generate position coordinates in six axes (three translational and three rotational) and the ability to continuously track the position coordinates are critical to the operability of the device and system. Several commercially-available positioning systems can produce position coordinates accurately and track multiple points at once. For example, magnetic tracking systems combine very high tracking resolution with high-speed sampling, which contribute to utilization in virtual reality simulations. Disadvantageously, magnetic tracking systems are very expensive and, furthermore, the likelihood of successfully integrating magnetic tracking in an inexpensive, home system is not very high. The magnetic fields associated with these systems also may experience high interference in home operation, affecting proper and satisfactory system operation.
• Radio frequency positioning and tracking, e.g., using a few identification tags (RFID) in combination with a plurality of receiving units, is a possible alternative. Typically, RFID systems determine the positioning of an object, e.g., a hand, by triangulating, e.g., measuring the time it takes for the RFID signal to travel to/from the object for each of the plurality of receiving units. However, conventional RFID systems operate at or near the wireless spectrum of most household, making interference an issue. RF systems also would not provide the accuracy necessary for the present invention.
• Infrared positioning, which is discussed above, can be both accurate and inexpensive. However, IR relies on line-of-site signals, making obstructions a huge problem.
• Inertial measurement units (IMUs) are adapted to determine the orientation of an object (in space), the velocity of the object, and 3D positions using dead reckoning. IMUs can be structured and arranged to gather data from all six degrees of freedom and avoids the shortcomings of the IR and electro-magnetic options. One reason why IMUs have not been used heretofore, has to do with dead reckoning.
• “Dead reckoning” refers to all object positions being measured relative to a pre-established and known initial starting (“home”) point. As soon as the IMU moves from the initial starting point, sensors, e.g., multi-axis accelerometers, gyroscopes, and the like, provide data to a processing unit that is adapted to calculate the speed of the IMU and the distance traveled from home. Each successive move relates back to all previous movements, hence, positioning is a compilation of a plurality of discrete, smaller movements.
• Finally, three-dimensional cameras provide real time transitional axis positional data but do not provide rotational axis positional data. Typically, an illumination source emits an IR light having a discrete, pre-established frequency, e.g., 44 MHz. A plurality of sensors—half of which operate at the pre-established frequency and half of which are out-of-phase with that frequency—measures the time it takes for the IR light to be reflected by an object and to return to the sensor, which is to say, the time-of-flight (TOF). TOF data provide accurate depth data of the object, which can be gathered as quickly as an acceptable 60 frames per second.
Input Device (SmartGlove™ )
• Anideal input device92 is a wearable glove that is sized to be universal, i.e., useable on either hand, or adjustable, and, optionally, has its fingertip portions removed, to accommodate different hand sizes. Preferably, theinput device92 is structured and arranged to enable a patient/user91 to don it and doff it using only one hand. A total weight not to exceed one pound and a dorsal weight not to exceed eight ounces is recommended to facilitate use by patients/users91.
• Ideally, six axes of movement of the hand are feasible and, more importantly, are recognizable for the purpose of generating and recording movement and orientation data. Additionally, motion of each finger and thumb is not hindered and individually isolatable.
• Preferably, theinput device92 is structured and arranged to measure at least one of the following accurately: finger flexion/extension measured to at least 90°; wrist flexion/extension, i.e., dorsal action, at ±90°; wrist-radial deviation up to 40°; and wrist-ulnar deviation up to 50°; and supination/pronation of the forearm up to 180°.
• Referring toFIG. 11, a firstinput device embodiment70 includesbend sensors71,72, and73, which are disposed on the back of theinput device92 on finger portions that are in registration with the patient/user's index, middle, and ring fingers. To reduce weight and cost, a sensor on the pinky finger, whose movement generally follows that of the adjacent ring finger very closely, is optional. As shown inFIG. 12, abend sensor74 can also be disposed on the back of theinput device92 in registration with the thumb and abend sensor75 can be disposed on theinput device92 at the base of the palm of the hand. Thelatter bend sensor75 is adapted to bend as the heel of the thumb crosses the palm to oppose one or more of the fingers, e.g., during a pinch motion.
• Referring toFIG. 13, a two-dimensional bend sensor77, which is disposed on the back of theinput device92 in registration with the wrist and oriented along the axis of the ulna, is provided to capture wrist flexion/extension. To track radial and ulnar deviations, abend sensor78 is disposed on the ulnar side inside of the hand, generally oriented along the axis of the thumb in a neutral position. At a neutral hand position, thebend sensor78 for radial and ulnar deviations is slightly bent. When the hand is rotated or turned in an outward direction from the neutral position, thesame sensor78 will appear to remain straight. However, when the hand is rotated or turned inwardly from the neutral position, thesensor78 will detect and measure the greatest movement.
• Preferably, theglove70 can be hardwired to a base station that includes the electronics required to communicate to thePC93, e.g., wirelessly or via aUSB99. This eliminates the need to attach an electronics board to the patient/user's forearm. The base station can be ergonomically shaped and can include a mechanical button for dead reckoning purposes. A plurality of, e.g. two, input buttons can also be provided to facilitate digital YES (or 1) and NO (or 0) input for navigating the software. To prevent false readings, software will only recognize button input at discrete, pre-established times, e.g., between exercise sets.
• A feedback system, e.g., a haptic system, an audible speaker, and/or light emitting devices, can also be disposed on or within the finger portions of the glove and/or on the back of theglove70, e.g., with or in theIMU housing76, to provide vibratory or auditory clues and/or visual signals during exercises. For example, referring toFIG. 14, electro-luminescent (el)wire79 can be exposed around each of the index, middle, and ring fingers so that when any of the fingers is moved into a correct position, theel wire79 disposed about the correctly-positioned finger can be illuminated, e.g., by a sequencer (not shown) electrically coupled to a power source and a power control device (not shown), e.g., an inverter.
• Referring toFIG. 15, asecond glove embodiment60 is shown. Theglove60 includesbend sensors61 and62 that are disposed, respectively, on the metacarpalphalangeal (MCP) joint and the proximal interphalangeal (PIP) joint of the thumb and of each finger, including the pinky finger. The MCP andPIP bend sensors61 and62 are adapted to record arcuate bend data associated with the motion or movement of each finger. Athird bend sensor63 is disposed on the back of the hand at the base of the thumb. For measuring wrist flexion/extension, a bi-directional bend sensor (not shown) is disposed to extend across the wrist on the palm side of theglove60.
• Optionally, aswitch pad64, e.g., a capacitive touch sensor, can be disposed on or within the tip of the thumb portion of theglove60 for providing and recording pinch data. In operation, when the tip of one or more of the glove fingers contacts, i.e., activates, theswitch pad64, the controller is adapted to use the touch data and bend sensor data generated to differentiate and identify the pinching finger from the non-pinching fingers.
• AnIMU65 is provided on theglove60, e.g., between the knuckles and the wrist on the portion of theglove60 corresponding to the back of the patient/user's hand. During prototype testing, the inventors used anIMU65 that includes three integrated circuits: two vibrating beam gyroscope assemblies and one micro-machined accelerometer. Each of the three chips generate data in all three acceleration axes and in all three rotational axes. Hence, the degree of freedom is six.
• TheIMU65 is electrically coupled to aprocessing unit66 that is removably attached to the patient/user's forearm as shown inFIG. 16. By attaching theprocessing unit66 on the patent/user's forearm, the weight of theunit66 is removed from the hand so as not to hinder or interfere with hand movement while keeping the sensors61-64 proximate to theunit66.
• A Halleffect glove embodiment50 is shown inFIGS. 17-19. Single bend sensors covering both the MCP andPIP joints51 are disposed on each of the finger portions of theglove50. A plurality, e.g., three, bend sensors52-54 are disposed on the thumb portion of theglove50. Abend sensor52 is disposed between the thumb portion and the index finger portion of theglove50 to track relative movement between the fingers and the thumb, anotherbend sensor53 is disposed along the axis of the thumb portion to track movement of the thumb, and athird bend sensor54 is disposed at the base of the thumb portion of theglove50 to measure roll of the wrist joint as the patient/user's thumb reaches across the palm.
• To measure finger pinch, on the palm side of theglove50,Hall effect sensors56 are disposed on the tips of each glove finger and a magneticfield generating device57, e.g., a magnet, is disposed at or near the tip of the glove thumb. When the magnetic field from the magneticfield generating device57 approaches and/or contacts any of thesensors56, thesensors56 switch, providing data to the controller. The controller is adapted to use the touch data and bend sensor data generated to differentiate between and to identify the pinching finger from the non-pinching fingers.
• To properly measure hand position and orientation, anIMU52 is disposed on the back of the hand portion on theglove50. AnotherHall effect sensor59 is also disposed in the palm of theglove50 to enable dead reckoning. More specifically, whenever the patient/user91 places his/her hand correctly on a base station (described in greater detail below) that is equipped with a magnetic-field generating device (not shown), theHall effect sensor59 will generate a signal from which the controller will call and execute an algorithm, software, driver programs, and the like to calibrate theIMU52.
• Advantageously, with Hall effect sensors in theglove50 to activate a signal, the base can be wireless. Although not required, a wireless base minimizes the proliferation of wires and cables, which can get tangled and/or hinder movement.
• As shown inFIG. 19, astrap40 can be removably attached to the patient/user's forearm. Thestrap40 can include a power source (not shown), e.g., one or more batteries, a controller (not shown), e.g., an Arduino USB board manufactured by Arduino Software of Italy, and anaccelerometer41. Comparison of accelerometer readings from theaccelerometer41 on the forearm and from an accelerometer disposed in theIMU52 can be used to determine the angle of wrist bending.
• Referring toFIG. 20, anarm sleeve embodiment45 is shown. The embodiment includes a glove portion42 and apulley portion43. To measure the flexion/extension of the patient/patient/user's digits,bend sensors44 and46 are dispose on the back of the hand portion of the glove portion42, to be in registration with the MCP and the PIP of each finger, while at least onebend sensors47 is disposed along the PIP of the thumb. To measure the roll of the thumb, at least one bend sensor (not shown) is disposed on the glove portion42 along the crease of the thumb along the palm of the hand. Flexion/extension of the wrist can be measured by a bend sensor (not shown) that is disposed on the glove portion42 in registration with the posterior side of the patient/user's wrist.
• Optionally, the glove portion42 can be fingerless, permitting better fit across a variety of hand sizes. When fingerless, a plurality of rubber caps are attached to the tips of each finger and thumb. Inside each rubber cap is a small push button that is proximate the finger or thump tips. The operation of the buttons in the fingerless version is the same as that previously described.
• Thepulley portion43 is provided to measure radial and ulnar deviations. To that end, thepulley portion43 includes astrap48 that is securely but releasably attachable to the, e.g., medial side of the, glove portion42, e.g., using a hook and pile material, at a first end; that runs along the lateral side of the patient/user's wrist; and that passes through asmall pulley49 that is disposed, e.g., in an arm sleeve, above the patient/user's elbow.
• Referring toFIG. 21, an exemplary hardware interface, i.e.,input device92, that each patient/user91 will use is the SmartGlove™ developed at Northeastern University of Boston, Mass. TheSmartGlove™92 was chosen as aninput device92 because of a low cost, off-the-shelf device that would be suitable for home use. Newer technologies could provide the same kind of interface for patients/users91 while maintaining the high usability and low cost for the practitioner.
• An input device mounting box prototype will be described referring toFIGS. 30-33.FIG. 30 shows top andbottom portions101,102 of amounting box110 that, when assembled as shown inFIG. 31, form a box-like structure that is structured and arranged to rest on the back of a patient/user's hand and to accommodate all of the necessary sensing devices. Adepressed area106 for receiving anIMU108 is provided in thetop portion101. A pair ofvertical standoffs109 are provided to orient theIMU108. Aslot104 for accommodating aHall effect sensor105 is also provided in thetop portion101.
• Theprototype110 is releasably attachable to the back of the patient/user's hand using, for example, a hook and pile combination that can be routed through a pair ofopenings103 provided for that purpose.
• Referring toFIG. 33,bend sensors107, which will be disposed within the material of theSmartGlove™92, are shown optionally coupled to exiting sleeve rings109 at a first end and, to mimic the structure of the hand, are mechanically attached to a single point at the rear of the mountingbox110 at a second end. As shown inFIG. 35, eachbend sensor107 enters the mountingbox110 via arespective slot112. Elastic string can be used for attaching thebend sensors107 to the single point. The string prevents thesensors107 from rotating about the single point while also allowing thesensors107 freedom to translate along the axis of each finger as flexion/extension occurs. This enables thesensors107 to retain the geometry of the patient/user's hand as knuckles are flexed and relaxed. It also enables thebend sensors107 to remain in a constant position relative to the fingers.
• Optionally, to fix and hold constant the radius over which an MCP bend sensor will flex and to prevent thebend sensors107 from sliding to the sides of the patient/user's knuckles, a knuckle plate111 (FIGS. 34 and 35) can be provided. Because signals from thesensors107 vary with the bend radius independent of the actual angle of the bend, if the radius can be held constant, the angle of the MCP joint's bend can be accurately modeled.
• TheSmartGlove™92 was also chosen because it is easily connected to aPC93, e.g., wirelessly or via a universal serial bus (USB)port99, and offers satisfactory control to the patient/user91. For use in connection with theMUVER system90, a USB 2.0 is preferred for greater data transfer at a faster rate.
• Technical specifications for the SmartGlove™ input device92 include:
Finger Sensor Specifications
• Five independent finger measurements
• 60 Hz refresh rate
• 0.5 degree resolution
Tracking System Specifications
• Optical tracking system
• 3-foot range from “receptor”
• 45 Hz refresh rate
• Six DOF
Yaw\Pitch\Roll Specifications
• 3 degree resolution
• 3 degree accuracy
X\Y\Z Specifications
• 0.125 inch resolution at 3 foot range from “receptor”
• 0.5 inch accuracy at 3 foot range from “receptor”
USB System Specs
• USB 1.1 compliant
• HID specification compliant
• At least two USB interfaces provided, i.e., a native P5 mode, and standard mouse mode
• Referring again toFIG. 21, during operation, which is to say, during manipulation of the hand rehabilitation (input)device92, the patient/user's hand and/or finger movements are transmitted to thePC93, e.g., wirelessly and/or via theUSB connection99. Adevice driver97 for theSmartGlove™92, which can be installed in the operating system of theSmartGlove™92 or, alternatively, as shown inFIG. 21, inoperating system98 of thePC93, interprets the input data and provides the data to thevirtual reality software94.
• As shown inFIG. 39, although theIMU115 is incorporated into theinput device92 and disposed on the back of the patient/user's hand, when aUSB connection99 is used, to reduce the total weight of the system on the patient/user's hand acable mount112 can be disposed on the patient/user's forearm.
• Thevirtual reality software94, in turn, uses theSmartGlove™92 input data to generate output signals designed to display appropriate images in a virtual reality on a display device (not shown), e.g., the display device of a PC. Preferably, the MUVER programming display software includes three different pieces that are illustrated inFIG. 26: agame engine82, 3D models andgraphics83, and ascripting code84.
Bi-Manual System
• Although the above description describes a single input device, the system can also include multiple input devices, e.g., a pair of SmartGloves™ for the left and right hands of the patient/user, that can be used simultaneously. Multiple input devices in general and, more specifically, a pair of SmartGloves™, permit more complex and realistic rehabilitation tasks such as, in virtual reality, simultaneously grasping a jar with a first hand and removing a lid from the jar with a second hand.
• An illustrativebi-manual system120 is shown inFIG. 42. Each of the patient/user's hands and forearms are supported in adjustable arm splints111 for neutral wrist position and support. Theend portions119 of each of the splints are ergonomically curled to make the natural position as comfortable for the patient/user as possible. Each hand is disposed within aninput device113 that can be a Spandex®/cotton blend glove.
• Bend sensors114 are positioned (within a pocket in or within the material of the glove113) across the MCP and PIP of the patient/user's index, middle, and ring fingers, to measure finger flexion and extension. Afourth bend sensor116 is disposed along the back of the hand proximate the patient/user's wrist to measure wrist flexion and extension. Afifth bend sensor117 is disposed along the back of the thumb to measure the rotation of the thumb with respect to the patient/user's palm and fingers.
• Hall effect sensors118 can be wired in the finger tip portions of theglove113 and are adapted to interact when they come in proximity of a small magnet (not shown) that is disposed in the palm of the glove. The small magnets in each of the gloves are used in combination with the globe base (see below) to calibrate the position of the gloves with respect to the base.
• AnIMU115 is disposed on the back portion of theglove113 for monitoring the three-dimensional hand position, i.e., attitude, of the patient/user's hand. To reduce the weight on the patient/user's hand and to facilitate connection of wiring to theIMU115, acable housing112 is wrist-mounted. More preferably, wireless communication of data can be effected.
Base Unit
• With the present invention, the patient/user must also be able to achieve the same dead reckoning position at the start and/or at the completion of each exercise due to, inter alia, the nature of IMU positioning. To best achieve this, the system includes a base structure that accounts for ergonomics and an activation sequence that enables the controller to receive data once the patient/user has placed his/her hands in the appropriate position.
• FIG. 22A show abanana grip base30 having a plurality ofRFID tag receivers31 andFIG. 22B shows thesame grip base30 with a patient/user's hands placed in the appropriate staring (“home”) position. The ergonomics of thebanana grip base30 allows patients/users to place their hands on thebase pad32 without having to strain to cause them to lie flat. TheRFID receivers31 do not require power, allowing thegrip base30 to remain completely passive, i.e., wireless.
• According to this approach, RFID transceivers59 (see,FIG. 18) are disposed in theinput device50 so that when the patient/user's hands are disposed at the “home” position, thereceivers31 andtransceivers59 are proximate, causing thetransceiver59 to emit a signal to that effect. A drawback of the RFID approach is its accuracy. Slight deviations from a true “home” position may skew the results of the exercise.
• FIG. 23 shows aglobe base35 that includes imprintedhand grooves33 that define the “home” position.Capacitive touch sensors34 can be disposed at each of the finger and thumb tips of thehand grooves33. Theglobe base35 is adapted so that the patient/user's digits must touch each of thetouch sensors34 in bothhand grooves33 for location data to zero itself. This design is particularly attractive due to its simplicity and, further, it causes little strain on the patent/user's hands. Thehand grooves33 andtouch sensors34 at the fingertips of thehand grooves33 make it intuitive to use. Alternatively, Hall effect sensors and magnets can be integrated into theinput device92 and theglobe base35.
• Theglobe base35 is a single piece, ergonomic design that requires a power source. The relatively large size make it possible to house electronics, e.g., a processing board, a Bluetooth® receiver, and other accessories (including the input devices when not in use), within thebase35 itself. Various feedback systems, e.g., speakers for audio feedback, vibration devices for haptic feedback, and LEDs for visual feedback, can also be disposed within or on the outer surface of theglobe base35.
• Prototype testing by the inventors highlighted the need for providing wrist and or forearm support for accommodating and supporting when the patient/user's hands when properly positioned in the “home” position. The incorporation of medical arm splints with the globe base35 (FIG. 42) allows the patient/user to maintain his/her hands in a neutral, “home” position, which is to say: full pronation, no extension or flexion, and no radial or ulnar deviation. By employing a splint that is not in wrist extension—as most are—patients having, for example, a contracture or spasticity in wrist or finger flexor muscles can more easily achieve the correct, neutral position.
• Disadvantageously,touch sensors34 in the fingertips alone do not ensure that the patient/user's palms are also touching thebase35. As a result, location data may be incorrect. Thetouch sensors34 also require the glove device to be fingerless as only contact with exposed skin at the fingertips will activate thesensors34. Finally, the size of theglobe base35 is relatively large, requiring additional exercise area.
• FIGS. 24A and 24B show ateardrop base39, one of which is provided for each hand. Theteardrop base39 design creates an ergonomic platform on which patients/users may rest his/her hands. Atactile switch button38 is disposed on the base39 so that when the patient/user's hand is properly positioned over thebase39, thebutton38 will activate the device. Theactive switch button38 can be electrically coupled to the controller, e.g., via a USB connection. The main problem with the teardrop design is that thebutton38 will not necessarily be repeatedly activated by the same part of the patient/user's hand and because twobases39 are needed—one for each hand—requiring two dead reckoning signals.
• FIGS. 25A and 25B show apyramid base37, one of which will also be provided for each hand. The design creates an ergonomic platform for hand placement, which is particularly effective for stroke patients who frequently have difficulty spreading their hands. Thepyramid base37 includes a magnetic field-generating source36, e.g., a magnet, that is adapted to activate a Hall effect sensor that is disposed in the palms of the input device gloves. Hall effect activation allows hand placement on the base37 to be repeatable within an acceptable tolerance proportional to the resolution of the sensor36, while allowing the base37 to be passive, i.e., wireless and not requiring power.
• The simplicity of design, ergonomic shape and Hall effect activation are advantages of thepyramid base37. Disadvantageously, twobases37 are needed—one for each hand—requiring two dead reckoning signals and, furthermore, the sensor disposed in the glove input device may cause discomfort.
Three-dimensional Graphics Engine
• A 3D graphics engine is a library of subroutines for 3D rendering and game development. The 3D models andgraphics83 portion populates the engine code and follows the rules of thegame engine82. Thescripting code84 in thegame engine82 controls many of the low level features, e.g., physics and display, and, preferably, is written in Python™ Script. More preferably, thescripting code84 can overwrite some or all low-level game engine code while also providing unique features to the3D models83.
• Software demands for theMUVER90 are both specific and advanced. Indeed, any development platform selected requires network capabilities and 3D graphics. Preferably, theMUVER90 includes a software package that is simple both to implement and to change and that, also, is capable of providing the features that theMUVER90 requires, e.g., network coding, 2D/3D rendering, and so forth. One possible software option for theMUVER90 is to use a 3D graphics engine for most of the code and using a scripting language to program in the various scenes and the use of theSmartGlove™92.
• There are many commercially-available game engines that have varying levels of programming sophistication. Five such engine solutions that could be integrated into theMUVER90 include: Panda3D, Blender, Source and Hammer, XNA®, and Flash®. Each solution has advantages and disadvantages with respect to the other solutions.
• The Panda3D graphics engine was originally created by Disney but is currently owned by Carnegie-Mellon University of Pittsburgh, Pa. The software integrated into the graphics engine is open source, which is to say that it is free to the public for download for commercial and non-commercial use and can be freely modified. The Panda3D graphics engine uses the Python™ scripting (programming) language and is written in object-oriented, C++ libraries and modules. Panda3D has comprehensive support for networking that allows for rich virtual interactions between patient/users. Data input methods for the Panda3D advantageously include direct input of Head Mounted Displays (HMD) and VR trackers.
• Panda3D remains a preferred platform because of its license agreement as well as its capabilities. However, Panda3D requires additional middleware which increases cost and complexity. Middleware is a term or art used to define a software “bridge” between hardware such as between aninput device92 and the software of the MUVER. As a result, the middleware must be compatible with the input device, i.e., theSmartGlove™92, but must also be able to generate a compatible output signal to the chosen development software.
• Potential middleware examples include software development tools, e.g. SWIG, that are adapted to connect programs written in a first programming language, e.g., C and C++, with a variety of high-level programming languages. Typically, SWIG is used to create high-level, interpreted or compiled programming environments, and user interfaces. SWIG is used with different types of languages—including common scripting languages such as Python™, which is the scripting language of Panda3D. Advantageously, SWIG is open source and, hence, may be freely used, distributed, and modified for both commercial and non-commercial use.
• SWIG, however, is a difficult program to work with and very unstable. Notwithstanding, SWIG enables programmers to write programming methods for theSmartGlove™92 in C/C++ and, subsequently, to “wrap” them so that they can be read in Python™ programming code. This features allows theSmartGlove™92 to be useable in the BlenderGE as a set of Python™ scripts.
• A second commercially-available middleware is GlovePIE (Glove Programmable Input Emulator), which was originally developed to emulate joystick and keyboard input. Succinctly, GlovePlE emulates movements made with theSmartGlove™92 using software macros and, further, binds the movements to an input device such as a keyboard or a joystick. As a result, use of theSmartGlove™92 as an input device is extended to any program that is traditionally control using a keyboard or a joystick. Disadvantageously, only certain joystick/keyboard movements are emulated by theSmartGlove™92, which limits the number of movements the would be available to a practitioner and/or a patient/user.
• Advantageously, however, use of GlovePIE is no longer confined to VR gloves; but, rather, now supports emulating a myriad of input, using a myriad of devices, e.g., Polhemus, Intersense, Ascension, WorldViz, 5DT, and eMagin products. GlovePIE may also control MIDI or OSC output.
• Hardware supported by the GlovePIE includes:
• Nintendo Wii Remote
• NaturalPoint (or eDimensional) TracklR, OptiTrack, SmartNav
• FakeSpace Pinch Gloves (9600 baud by default, but can be changed)
• Concept 2 PM3 rowing machines (ergo or erg)
• All joysticks or gamepads recognized by Windows
• Parallel port gamepads (with PPJoy)
• All keyboards
• Mice with up to 5 buttons and 2 scroll wheels
• Most microphones (don't have to be high quality)
• Most MIDI input or output devices
• Essential Reality P5 Glove™
• 5DT Data Glove (all versions)
• eMagin 2800 3D Visor HMD
• Polhemus trackers (must be set to 115200 baud): IsoTrak II, FasTrak, Liberty, Patriot, Liberty Latus
• Ascension trackers: Flock of Birds, MotionStar, etc.
• Intersense trackers: InterTrax, InertiaCube, IS-300, IS-600, IS-900, IS-1200, etc.
• WorldViz PPT trackers (all versions)
• GameTrak (only as a joystick, no direct support)
• Yet another commercially-available middleware product is OpenTracker, which is manufactured by Argent Data Systems of Santa Maria, Calif. OpenTracker is another open source product that is adapted to create a full-featured tracking software package that can be integrated into any software as a device library. The major advantages of OpenTracker are that it is the most full featured and robust middleware package. It also natively integrates into C/C++. Hardware supported by the OpenTracker includes:
• A.R.T. optical tracker (Windows, Linux)
• ARToolkit(Windows, Linux)
• ARToolkitPlus(Windows, Linux)
• Parallel Port (Windows, Linux)
• CyberMouse(Windows)
• Origin Instruments DynaSight (Windows, Linux)
• Ascension Flock of Birds (Windows, Linux)
• Polhemus FastTrak/IsoTrak (Windows, Linux)
• Garmin GPS devices like eTrex and similar (Windows)
• ICube X (Windows)
• Intersense InterTrax2 USB (Windows)
• Joysticks (Windows)
• evdev interface (mouse, keyboard, joysticks) (Linux)
• Barco MagicY (Windows)
• Midi (Windows)
• gTec gMobilab/gMobilab+ (Windows, Linux)
• P5Glove (Windows)
• Phantom Omni (Linux requires 3rd party driver, Windows?)
• 3DConnexion SpaceDevice (Windows; requires 3Dxware SDK)
• 3DConnexion SpaceMouse (Plus USB) (Windows; requires 3Dxware SDK)
• Ubisense Tracker (Windows, Linux)
• Polhemus UltraTrak (Windows, Linux)
• Wacom Graphire (Windows)
• Wii (Windows)
• XSense (Windows, requires MT9 SDK from XSens)
• Finally, the Virtual-Reality Peripheral Network (VRPN) is a set of classes within a library and a set of servers that are designed to implement a network-transparent interface between application programs and the set of physical devices, e.g., trackers and so forth, used in a virtual-reality system. Conceptually, a PC or other host controller is disposed at each VR station to control the peripherals, e.g., tracker, button device, haptic device, analog inputs, sound, and the like.
• VRPN provides middleware connections between the application(s) and the hardware devices using an appropriate class-of-service for each type of device sharing the link. The application remains unaware of the network topology. Advantageously, VRPN can be used with devices that are directly connected to the system that is executing (running) the application, using separate control programs or running the applications as a single program.
• VRPN also provides an abstraction layer that makes all devices of the same base class look the same. For example, all tracking devices are made to look like they are of the type vrpn_Tracker. As a result, all trackers will produce the same types of reports. At the same time, it is possible for an application that requires access to specialized features of a certain tracking device, e.g., telling a certain type of tracker how often to generate reports, to derive a class that communicates with this type of tracker. If this specialized class were used with a tracker that did not understand how to set its update rate, the specialized commands would be ignored by that tracker.
• Current VPRN system types include: Analog, Button, Dial, ForceDevice, Sound, Text, and Tracker. Each type abstracts a set of semantics for a specific device type. There are one or more servers for each type of device, and a client-side class to read values from the device and control its operation. VRPN is the preferred middleware solution because of its capabilities and robustness, which the other solutions lacked. Historically, VRPN has also been used successfully with Panda3D.
• Returning to the game engines, Blender is another open source software package manufactured by the Blender Foundation that focuses on digital modeling and animation. An integrated game engine called BlenderGE uses Python™ as a scripting (programming) language. The main advantages of Blender are that the software is included in the BlenderGE and, because, when Blender is used as a digital animation package, it has many features to use armatures, e.g., a human hand.
• The Source Game Engine and the Hammer Level Editor (“Source and Hammer”) are manufactured by the Valve Corporation, a video game company. Source and Hammer is a leading character animation graphics engine whose main advantage is an advanced physics engine and multiple patient/user capabilities. Source and Hammer can be used for non-commercial purposes.
• XNA™ is a product of Microsoft® Corporation of Seattle, Wash. XNA™ uses C# code and a shared library to facilitate creation of games and simulations. The biggest advantages of XNA™ are that the community is very knowledgeable and it is designed for making multi-user games. One caveat, however, is that XNA™ does not have a bundled game engine. Accordingly, using XNA™ software to create the MUVER would require much more programming than using one of the software packages associated with a game engine.
• Finally, Flash® manufactured by Adobe Systems Incorporated of San Jose, Calif. is primarily used for Web sites and for Internet applications. The biggest advantage of using Flash® is that most applications can be accessed from any Web browser, which means that installation is not mandatory. Like XNA™, however, Flash® does not have a bundled game engine and requires more programming than other software solutions. The multi-user options available for Flash®, however, are comparable to XNA™ and Panda3D but better than what is available for Blender.
Testing Results of Concept
• FIG. 27 shows an exemplary virtual environment (VE) training “scene” (or exercise) for competitive virtual interaction between two patient/users. The exemplary scene is designed to allow patients/ users to practice both an active grasp and a maintained grasp. An active grasp involves a pinching action using two to four fingers and the thumb and a release action. A maintained grasp involves active supination, which is to say, a hand further rotating toward a palm-up position. These movements are deemed by experts essential to improved hand function in stroke patients.
• In this VE training “scene”, a patient/user must move his/her hand and fingers, which are operationally coupled to the input device, first, to grasp or grip a virtual lid67 (FIG. 27A) and then to lift thevirtual lid67 from a virtual pot58 (FIG. 27B). Once the patient/user has successfully grasped and lifted thevirtual lid67, he/she supinates thevirtual lid67 to a palm-up position, all the while maintaining his/her grasp on thevirtual lid67. In the final stage of the scene, the patient/user pronates thevirtual lid67 to a palm-down position; returns thevirtual lid67 back to thevirtual pot72; and releases his/her grasp on the virtual lid67 (FIG. 27D).
• Preferably, the scene application is adapted so that a visual signal is generated upon successful completion of any or all of the movements or motor activity associated with stages. For example, once the patient/user satisfactorily is able to grip the virtual lid67 (FIG. 27A) thevirtual lid67 can change color, e.g., from grey to blue. Further, once thevirtual lid67 is lifted from the virtual pot68 (FIG. 27B) thevirtual lid67 can change color again, e.g., from blue to green. Similarly, upon successful grasp plus supination and transport (FIG. 27C), thevirtual lid67 can change color again, e.g., from green to red. Advantageously, the supination threshold for success can be pre-set, e.g., 45 degrees, and can be further adjusted to require a greater or lesser result. Finally, following pronation and transport (FIG. 27D), thevirtual lid67 can return to its original color, e.g., grey. Undergoing this cycle of stages, the trial is counted as a “success”.
• If, however, the patient/user loses his/her grasp while raising, supinating, and/or pronating thevirtual lid67, the application can be adapted to cause thevirtual lid67 to return automatically to the original position on thevirtual pot68 and to return to its original color, e.g., grey.
• The movement during each of the phases can be timed and counted separately. This provides more feedback to patients/users about their performance if the entire trial was not successful. The ability to count successful phases and trials and record time for each can be incorporated in the design for the scoring function described in greater detail below. Thus, each discrete data element can be displayed separately to provide feedback about performance either during the session or after. These discrete elements count the number of successes for each phase; count the number of successful trials, which is to say, that all phases are completed successfully; record the time for each phase; and record the time for each trial. Time can also be displayed as a mean for block of trials, with the number in block adjustable.
• FIG. 28 summarizes the results of testing of six (6) healthy subjects (four males, two females; 5 right-handed, one ambidextrous) who performed competitive interaction using the MUVER scene shown inFIG. 27. Each patient/user donned and calibrated the SmartGlove™ and was instructed in the hand rehabilitation movement depicted in the scene. Each subject was provided with up to five minutes to practice and become accustomed to working in the virtual environment.
• Subsequently, subjects were asked to complete ten movements as rapidly as possible. A short rest was given between each of the ten trial movements. The total time and the time for each phase were recorded for each trial and for each subject. Themean durations4 and associatedstandard deviations5 across blocks and subjects are summarized inFIG. 28 for each sub-phase of the movement task (grasp, turn, and return) and for the total task.
• Start to completion of grasping thelid67 averaged 1.6±0.7 sec. Grasp with turn and transport to right averaged 1.8±0.6 sec. Pronation, lid return, finger extension to release grasp averaged 2.1±0.9 sec. In summary, the mean time for the total task was 5.5±1.6 sec.
• Experimenters and subjects noted that the mapping of real world to virtual environment movements was not completely accurate, which probably accounts for times that are somewhat slower than expected in healthy subjects. This inaccuracy, however, could be due to a hardware problem in the glove position detection and/or due to the software that maps position and orientation data to the virtual scene elements.
• Referring toFIG. 29 a morecomplete MUVER System90 is shown. As previously mentioned, the practitioner'sinterface96 expands data collection of the basic system in two major ways: providing real-time feedback and modifying the virtual environment. Indeed, a practitioner can be a spectator and watch the actions and interactions of the patients/users91. They also have the ability to privately or openly provide feedback to a patient/user91 or multiple patients/users91 at once. As another form of feedback the practitioner can change or control theMUVER system90 based on the actions and interactions of the patients/users91. Modifying theMUVER90 has several advantages including changing the level of difficulty to better suit rehabilitation and also directing theMUVER90 to facilitate certain interactions between patients/users91.
• The morecomplete system90 includes aRapid Prototyping feature85. Rapid Prototyping (RP) is a manufacturing method that allows custom objects to be created from a .STL file quickly and for low cost. With this feature, theMUVER90 can be populated by models that can be exported as .STL files. Consequently, a practitioner can borrow an object from the real world; recreate it in the MUVER for virtual rehabilitation; and then create it using RP for use in actual rehabilitation exercises86.
Object Examples
• A first exemplary MUVER scene that can be presented virtually in accordance with the present invention is a simple, Single Degree of Freedom Knob (SK). In the real world, an SK is an Electro-Rheological Fluid (ERF) based device that can be manipulated by the patient/user in a single degree of freedom, e.g., clockwise or counterclockwise rotation about an axis, by rotating a variably-resistive knob. TheMUVER system90 allows a practitioner to place a patient/user in a virtual environment containing an SK device in the comfort and convenience of the patient/user's home. From a remote site, the practitioner is also able to customize the resistance level of the real world knobs for a particular patient/user.
• Avirtual knob87 that might be shown in a patient/user's virtual environment is shown inFIG. 36. The MUVER design for thevirtual knob87 is meant to be for one or two patient/users. A first patient/user manipulates his/her hand and fingers in the real world sufficiently to cause thecircle88 to rotate about an axis in either direction in the virtual environment. If there is a second patient/user, he/she likewise manipulates his/her hand in the real world sufficiently to cause the rounded square89 to rotate about the axis in the same direction in the virtual environment. If there is no second patient/user, the practitioner or, alternatively, a software program can control and manipulate the rate of rotation of therounded square89. Control of the square89 by a non-patient/user can be performed using another knob, a rehabilitation device, a keyboard, a mouse, and the like.
• The simplistic MUVER SK scene can be controlled to provide competitive, versus, or mixed virtual interactions. For the competitive interaction, the time it takes each patient/user to catch, i.e. to “tag”, the square89 during a pre-established amount of time can be recorded and compared. Another possible competitive interaction is, instead, to record the number of “tags” during a pre-established period of time for each patient/user.
• An exemplary versus interaction can include awarding points to the first, circle patient/user for every “tag” of the square89 and awarding points to the rounded square patient/user for avoiding being tagged over a set increment of time. An exemplary mixed interaction can include two patient/users comparing their scores and times with another pair of patient/users or with historical scores of the two.
• Scene themes, such as the SK example above, can play a very important role in the MUVER design. The customizable knob lends the device to many possible real scenes that can be made into virtual ones. The availability of real life scenes in a virtual environment can be beneficial because a particular patient/user may have performed the task regularly in the real world and, hence, already understands the movements needed to successfully complete it.
• Possible real life scenes to use in a MUVER for the SK can, for purposes of illustration and not limitation, include:
• Opening a door knob
• Unlocking a door
• Turning a car's ignition
• Opening a jar
• Using a rotary phone
• Manipulating a thermostat
• Reeling in a fish
• Screwing in a light blub
• Opening a bottle of wine
• Turning a stove on and off
• Juicing a lemon or orange
• Stirring coffee or soup
• Fictional scene themes that parallel events that the patient/user is not familiar with have a low learning curve because the patient/user may not know how to complete the movements successfully. For example, a non-fisherman may not know how to reel in a fish or a teetotaler may not know how to turn a corkscrew to open a bottle of wine. Notwithstanding, fictional scene themes have the advantage of being made expressly for exercising a certain movement or knob design. The prototype MUVER proposed here is an example of a fictional theme but the idea of tag is a common play mechanic that patient/user may be familiar with.
• Referring toFIG. 37 a diagram of virtual environment for a Single Degree of Freedom Hand Device (SHD) is shown. An SHD is a passive ERF device and, commonly, a research device meant for use in MRI machines.
• The single degree of freedom is linear and has a pre-established, e.g., three inch, stroke. As a result, the exercise routine the patient/user performs is simply grasp and release. Challenges include noise from the MRI machine, simplicity of the device, and using mirrors for the patient/user to see the computer monitor displaying the MUVER graphics. The real world SHD has many unique design aspects, which to the greatest extent possible are transferred over to the MUVER virtual environment design.
• The scene theme inFIG. 37 replicates the real world act of inflating aballoon85 using ahand pump86. This particular exemplary scene theme advantageously provides an objective that is simple and intuitive. Additionally, because the progressive inflation of the balloon presents the patient/user with feedback, no other feedback needs to be provided.
• An advantage of this MUVER scene theme is that it can facilitate all four kinds of multiple patient/user interactions: competitive, cooperative, versus, and mixed. The competitive interaction would be conducted by timing each patient/user for a pre-established number of pumps that will completely inflate theballoon85. When multiple pumps86 control the rate of inflation of thesame balloon85, a cooperative interaction can be accomplished. This type of interaction can be further explored by controlling the rhythm of pumps from each patient/user or having a set order that patients/users must pump in order to inflate theballoon85.
• By having the actions of one patient/user inflate theballoon85 while the actions of a second patient/user concurrently deflate it, one can provide counter-operative interaction. Finally, a mixed interaction using this MUVER design has the most depth and the greatest possibility, e.g., by having multiple patients/users inflate the same balloon while evaluating the relative percentages that each inflated. The patients/users could also be required to follow an inflation order or rhythm and could be judged on their respective accuracy at following it.
• As another example, an Active Hand Device (AHD) is a two degree of freedom, active ERF device. An exercise the patient/user does with the AHD includes the linear degree of freedom that the patient/user performs with the SHD as described above as well as a supination/pronation rotation similar to that of the SK. Advantageously, the AHD is also an active device, which is to say, that the AHD can provide variable resistance and push back against the patient/user, which can be used to enhance the experience. The major design considerations for such a device include having a customizable real world design so that the practitioner can choose to use both degrees or a single degree of freedom.
• Referring toFIG. 38, a diagram of the prototype MUVER for an AHD is shown. The patient/user is represented as thecircle69. The patient/user manipulates the AHD to move thecircle69 around atrack29 that can be designed by the practitioner. The patient/user receives feedback in the form of accuracy in movement and also in the speed with which thecircle69 circumnavigates thetrack29.
• Adjustable read lines28 represent the start and the end of thetrack29. The locations of the read lines28 can be modified depending on what the practitioner wants the patient/user to do for an exercise. Theellipse27 surrounding thecircle69 is a force gradient, inside of which the patient/user tries to keep thecircle69 as it moves around thetrack29.
• For example, exercises can be made more or less difficult based on the hands position relative to the shoulder, movement tests can be designed to test extremity coordination. Thus, the position/orientation feature is a valuable component to the rehabilitation package. The system is designed to work with two gloves simultaneously, to allow patients/users91 to use their capable hand to interact in exercises along with their disabled one.
• This is not to say that the mapping of real world movement must necessarily correspond to a virtual world movement, i.e., “direct mapping”. If the goal is rehabilitation, then whatever virtual “scene” best motivates a patient/user to expedite the rehabilitation process or rehabilitation milestones, the better. Accordingly, “abstract mapping” by which real world movement produced by the patient/user differs from the movement of the virtual world object(s) is also possible with the invention as claimed. For example, if the patient/user movement desired is wrist flexion and extension, a “direct mapping” scene may depict a hand waving while an “abstract mapping” scene may equate the flexion/extension to the movement of a cartoon animal.
Patient/User Scoring and Teacher Models
• Patient/user's performance and other data are stored locally and, furthermore, can be transmitted to a third party, e.g., a clinical provider, via the network. Performance data, e.g., hand and finger kinematics, can be used to assess progress over time, the level of impairment, and so forth. “Scoring” connotes reduction of performance data into a format that is easily usable to determine performance and progress. In particular, the scoring data is easily formatted into graphics, spreadsheets, summaries, and so forth.
• For example,FIGS. 40A-D show a patient/user's hand being constrained in 45 degree and 90 degree orientations, for measuring voltage as a function of joint bend of the MCP (FIGS. 40A and 40B) and the PIP (FIGS. 40C and 40D). Patient/user movement data forFIGS. 40A and 40B are displayed in a bar graph inFIG. 41A, which separates the data by finger, i.e., index finger, middle finger, and ring finger, and by the angle of constraint, i.e., 0, 45 degrees, and 90 degrees. Patient/user movement data forFIGS. 40C and 40D are displayed in a bar graph inFIG. 41B, which also separates the data by finger, i.e., index finger, middle finger, and ring finger, and by the angle of constraint, i.e., 0, 45 degrees, and 90 degrees. In both instances, at a glance, one can determine from the bar graphs that each of the joints of the patient/user can be moved.
• “Scoring” can be performed per “scene” or exercise, per phase within a “scene” or can be compiled over multiple scenes. “Scoring” can measure, for example, a number of repetitions, a magnitude of motion, speed of performance, speed of performance of a phase of a scene, accuracy of movement, and so forth.
• Optionally, the system can also include a “teacher model” capability, which provides the ability to record patient/user performance for analysis and later playback. During playback, patient/user errors can be highlighted and made known to the patient/user and correct performance can be demonstrated. U.S. Pat. No. 5,554,033 discloses a virtual teacher and is included herein in its entirety by reference.
Amplified Feedback and Adaptive Design
• Another advantage of the present system is a library of “scenes” or exercises, each scene being used for a discrete purpose or functional goal and being adjustable. More specifically, each of the “scenes” can be programmed to adjust the degree of difficulty of the scene. Indeed, defining the values of a set of parameters, e.g., speed of motion, magnitude of motion, smoothness of motion, hand orientation during motion, and the like, enables medical personnel, physical therapists, and the like to tailor scenes for a discrete patient/user. Hence, collectively, the “scene” database provides a variety of purposes and functional goals.
• Another feature of the present system is amplified feedback by which real world movement can be discretionally amplified prior to virtual mapping. For example, if a patient/user only bends his/her fingers ten degrees, the signal can be amplified by a factor of five so that, the virtual movement shows a 50 degree flexure. In this manner, amplified feedback can be used as a carrot to encourage patients/users.
• Those skilled in the art can appreciate that the simplistic and rudimentary scene themes described above are for illustrative purposes only. By mixing a real world scene theme with fictional scoring to facilitate multiple patient/user interactions, the MUVER design proves to be very robust.

Claims (41)

1. A multiple-user virtual environment system for rehabilitation exercise of mammalian trunk, extremities, and digits, the system comprising:

a communication network; and

a plurality of individual virtual environments, each of the individual virtual environments including:

at least one input device that is structured and arranged to generate at least one of movement, orientation, velocity, and position data corresponding to discrete movement of a portion of the mammalian trunk or of one or more mammalian extremities or digits,

a processing device that is adapted to receive the at least one movement, orientation, velocity and position data from the input device; to store said movement, orientation, velocity, and position data; and to generate image data therefrom for display on a display device, and

a virtual environment interface that is adapted to enable virtual environment communication and virtual environment data transfer between the processing device and the network.

2. The system as recited inclaim 1 further comprising an interface and processing device that enable a third party to observe and to record data from the processing device.

3. The system as recited inclaim 2, wherein the third-party processing device is structured and arranged to pre-establish or adapt at least one rehabilitation exercise for each of the plurality of virtual environments or to modify said exercise or virtual environment.

4. The system as recited inclaim 4, wherein the third-party processing device is structured and arranged to pre-establish a type of multi-user interaction for each of the at least one rehabilitation exercise and for each of the plurality of virtual environments.

5. The system as recited inclaim 4, wherein the type of multi-user interaction is selected from the group consisting of a competitive interaction, a counter-operative interaction, a cooperative interaction, and a mixed interaction.

6. The system as recited inclaim 4, wherein the third-party processing device is structured and arranged to adjust a degree of difficulty for each of the at least one rehabilitation exercises.

7. The system as recited inclaim 1, wherein the virtual environment interface includes at least one of:

game engine hardware having a programming code and a plurality of rules,

game engine software having a programming code and a plurality of rules,

a scripting programming code that is capable of overwriting low-level game engine programming code,

three-dimensional model software that is adapted to populate the engine programming code and to adhere to the plurality of rules, and

three-dimensional graphic software that is adapted to populate the engine programming code and to adhere to the plurality of rules.

8. The system as recited inclaim 1, wherein the network is selected from the group consisting of a local area network, a wide area network, the World Wide Web, and the Internet.

9. The system as recited inclaim 1, wherein the network is adapted to generate a virtual reality environment using the virtual environment data.

10. The system as recited inclaim 1, wherein the input device is adapted to monitor a position or an attitude of an extremity or of a digit in space.

11. The system as recited inclaim 1, wherein the discrete movement is selected from the group consisting of movement in an x-direction, movement in a y-direction, movement in a z-direction, pitch, roll, and yaw, wherein each of the x-direction, the y-direction, and the z-direction is mutually perpendicular.

12. The system as recited inclaim 1 further comprising a base unit that, in combination with the at least one input device, is structured and arranged to provide a dead reckoning starting and a dead reckoning ending point-of-reference, to enable the processing device to determine at least one of attitude and velocity of said at least one input device.

13. The system as recited inclaim 12, wherein the base unit is selected from the group comprising a banana grip base, a globe base, a pyramidal base or a tear-drop base.

14. The system as recited inclaim 13, wherein the globe base includes imprinted grooves that define the dead reckoning starting and the dead reckoning ending point-of-reference.

15. The system as recited inclaim 14, wherein the base unit includes adjustable arm splints for neutral wrist position and forearm support.

16. The system as recited inclaim 1, wherein the at least one input device comprises a pair of gloves.

17. The system as recited inclaim 1, wherein the processing device is adapted to map real world movement directly into similar or abstractly into different virtual world movement.

18. The system as recited inclaim 1 further comprising a teacher model capability to highlight shortcomings and errors of the user and to demonstrate how to correct said shortcomings and errors.

19. An input device for use with a multiple-user virtual environment system for rehabilitation exercise of a human hand and digits, the input device being structured and arranged to generate signals corresponding to at least one of a discrete movement and an attitude of said hand and said digits, the device comprising:

a glove that can be readily donned and doffed on either hand by a user, the glove having finger portions for at least a thumb, an index finger, a middle finger, and a ring finger;

a first plurality of sensors, each sensor being structured and arranged to provide data on movement and range of movement of at least one of the index finger, the middle finger, and the ring finger, each of the first plurality of sensors being disposed within the finger portions of said index finger, said middle finger, and said ring finger;

a second plurality of sensors that is structured and arranged to provide data on movement of the thumb; and

a positioning and tracking system that is structured and arranged to generate position coordinates in three rotational axes and three translational axes to determine at least one of the attitude and a velocity of said hand.

20. The input device as recited inclaim 19, wherein the first and the second pluralities of sensors are selected from the group comprising electronic bend sensors, resistive bend sensors, capacitive bend sensors, optical fiber sensors, mechanical measurement bend sensors, angle measurement sensors, Hall effect sensors or electromechanical sensors.

21. The input device as recited inclaim 19, wherein the positioning and tracking system is selected from the group comprising an inertial measurement unit (IMU), a radio frequency (RF) positioning and tracking system, an infrared positioning and tracking system, three-dimensional cameras or a magnetic tracking system.

22. The input device as recited inclaim 21, wherein the positioning and tracking system is an inertial measurement unit (IMU) that employs dead reckoning to determine the attitude of the hand and the velocity of the hand.

23. The input device as recited inclaim 22, the input device further including a Hall effect sensor for zeroing the IMU.

24. The input device as recited inclaim 19, wherein the device has a total weight that does not exceed sixteen ounces.

25. The input device as recited inclaim 24, wherein a dorsal weight on the input device does not exceed eight ounces.

26. The input device as recited inclaim 19, wherein the input device is structured and arranged to measure at least one of the following accurately:

finger flexion/extension measured to at least 90°;

wrist flexion/extension or dorsal action at ±90°;

wrist-radial deviation up to 40°;

wrist-ulnar deviation up to 50°; and

forearm supination/pronation up to 180°.

27. The input device as recited inclaim 19 further comprising a feedback system that is selected from the group comprising an audible speaker to provide auditory clues, at least one light-emitting device to provide a visual signal, and a haptic device to provide a vibratory signal.

28. The input device as recited inclaim 19 further comprising a touch sensor that is adapted to provide and record pinch data, the touch sensor being disposed in a tip portion of the glove thumb and being activated by contact with any of the index finger, the middle finger, the ring finger or a pinky finger.

29. The input device as recited inclaim 19 further comprising a communication means for providing hand and finger movement data and attitude data to a processing unit.

30. The input device as recited inclaim 19 further comprising a pulley portion to measure radial and ulnar deviations, the pulley portion being disposed above a user's elbow and being releasably attached to a medial side of the glove.

31. A method of providing a virtual environment system for rehabilitation exercise to a plurality of users over a communication network, the method comprising:

providing an individual virtual environments to each of the plurality of users, each of the individual virtual environments including an input device, a processing device, and a virtual environment interface;

generating at least one of movement, orientation, velocity, and position data signals corresponding to discrete movement of one or more mammalian trunk, extremities or digits disposed in the input device;

receiving the data signals from the input device;

generating image data for display on a display device and other data from said input data; and

enabling virtual environment communication and virtual environment data transfer between the processing device and the network.

32. The method as recited inclaim 31 further comprising enabling a third party to observe and to record image and other data from the processing device.

33. The method as recited inclaim 31 further comprising pre-establishing at least one rehabilitation exercise for each of the plurality of virtual environments.

34. The method as recited inclaim 31 further comprising pre-establishing a type of multi-user interaction for each of the at least one rehabilitation exercise and for each of the plurality of virtual environments.

35. The method as recited inclaim 34, wherein the type of multi-user interaction is selected from the group consisting of a competitive interaction, a counter-operative interaction, a cooperative interaction, and a mixed interaction.

36. The method as recited inclaim 31, wherein the discrete movement is selected from the group consisting of movement in an x-direction, movement in a y-direction, movement in a z-direction, pitch, roll, and yaw, wherein each of the x-direction, the y-direction, and the z-direction is mutually perpendicular.

37. The method as recited inclaim 31 further comprising generating a virtual reality environment using the virtual environment data.

38. The method as recited inclaim 31 further comprising monitoring a position of an extremity or of a digit in space.

39. The method as recited inclaim 31 further comprising:

scoring user performance using said input data; and

displaying scoring results after each exercise, daily, weekly, after each session, after each phase of an exercise and/or immediately.

40. The method as recited inclaim 31 further comprising amplifying the data signals corresponding to the at least one of movement, orientation, velocity, and position before displaying image data generated therefrom.

41. The method as recited inclaim 34 further comprising adjusting a degree of difficulty of said at least one rehabilitation exercise.

Images