US20120206419A1 - Collapsible input device - Google Patents

Abstract

In exemplary implementations of this invention, a handheld, collapsible input device (CID) may be employed by a user to input and manipulate 3D information. The CID telescopes in length. As a user presses the CID against a display screen, the physical length of the CID shortens, and the display screen displays a virtual end of the CID that appears to project through the screen into the virtual 3D space behind the screen. The total apparent length of the CID, comprised of a physical portion and a virtual portion, remains the same (after taking into account foreshortening). Thus, the user experience is that, as the user holds the physical CID and pushes it against the display screen, the end of the CID appears to be pushed through the display screen into the virtual 3D space beyond it. The CID housing may include a push button for user input.

Description

FIELD OF THE TECHNOLOGY
• The present invention relates generally to input and interaction technologies.
SUMMARY
• In exemplary implementations of this invention, a handheld, collapsible input device (CID) may be employed by a user to input and manipulate 3D information. The CID is long, like a pencil or pen. It telescopes in length. As a user presses the CID against a display screen, the physical length of the CID shortens: i.e., the CID physically collapses. As it does so, the display screen displays a virtual end of the CID that appears to project through the screen into the virtual 3D space behind the screen. The total apparent length of the CID, comprised of a physical portion and a virtual portion, remains the same (after taking into account foreshortening).
• Based on the length, position, and orientation of the device, the part of the device that is physically collapsed is rendered on the screen such that the user perceives that the CID continues into the virtual world beyond the screen.
• Thus, the user experience is that, as the user holds the physical CID and pushes it against a display screen, the end of the CID appears to be pushed through the display screen into the virtual 3D space beyond it.
• In exemplary implementations, the CID may be used to manipulate 3D information in a variety of ways. For example, a user may draw virtual 3D objects, such as a chair, using the virtual tip of the CID. Or a user may cut a virtual slice out of a virtual apple, using a virtual end of a CID which is, for that purpose, shaped like a saw or serrated cutting edge. From the user's perspective, it appears that the CID is directly manipulating 3D information beyond the screen.
• Also, a user may create virtual 3D objects by selecting from a menu of shapes. For example, a user may cause a virtual rectangle to be formed adjacent to the virtual tip of a CID, and then cause a virtual rectangular cuboid to be extruded from the rectangle. The shape may be selected from a menu by selecting an icon on a display screen. Or a user may select a shape from the menu by making a hand gesture with one hand, while holding the CID with the other hand.
• In exemplary embodiments, a grid function may be employed. It allows the virtual tip of the CID to snap to a point on a virtual 3D grid, making it easier for the user to create 3D objects with precise dimensions. Furthermore, a user may use the virtual tip of the CID to select three points to define a plane in virtual 3D space, then move the 3D plane to the 2D display surface and draw on that plane, and then reinsert the plane (with the drawing on it), into the virtual 3D space.
• In exemplary embodiments, haptic feedback is employed to heighten the illusion that the CID is directly interacting with virtual 3D objects in the virtual space beyond the display screen. This haptic feedback may vibratory. Alternately, a linear actuator in the CID may stop the physical CID from collapsing further, or even push the physical CID back into the user's hand, when the virtual end of the CID interacts with a virtual object.
• A number of working prototypes of this invention have been built.
• In two of these prototypes, reflective IR tags are used to determine the physical length of the CID as well as the 2D position and tilt angle of the CID relative to the display screen. In a third prototype, a linear potentiometer in the CID measures the physical length of the CID. In this third prototype, magnetic sensors in the display screen measure the 2D position and tilt angle of the CID relative to the display screen.
• In some prototypes, a user's head position is tracked using stereo infrared cameras and an ear-mounted infrared LED. The user's head position is used to determine the user's line of gaze into the virtual space beyond the display screen. This allows virtual objects, such as the virtual tip of the CID, to be displayed from the correct angle from the user's perspective. In another prototype, a user's head position is determined by face recognition. Alternately, gaze tracking may be used to determine a user's line of sight.
• The CID housing may include a push button for user input.
• The above description of the present invention is just a summary. It is intended only to give a general introduction to some illustrative implementations of this invention. It does not describe all of the details of this invention. This invention may be implemented in many other ways.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 shows a collapsible input device (CID) being held by a user, with the virtual end of the CID projecting beyond the display screen into the virtual 3D space behind the screen.
• FIG. 2A shows a CID, with full physical extension and no virtual end.
• FIG. 2B depicts the same thing asFIG. 1, from a different perspective.
• FIG. 3 shows a user holding a CID, and using it to carve a virtual 3D slice from a virtual 3D apple. The virtual cutting edge of the CID appears to project beyond the display screen into the virtual 3D space behind the screen.
• FIG. 4A shows various ways in which a CID may be moved.
• FIG. 4B shows the virtual end of a CID being used to draw a 3D virtual chair.
• FIG. 5A to 5C show a user holding a CID in one hand, and making gestures with the other hand. The various gestures cause different 3D virtual shapes to be formed. The virtual end of the CID may used to position these 3D shapes.
• FIG. 5A shows a virtual square being created in this way.
• FIG. 5B shows a virtual rectangular cuboid being extruded in this way.
• FIG. 5C shows a virtual ellipsoid being created in this way.
• FIG. 6 shows different hand gestures that may be used, to cause various 3D shapes to be created.
• FIG. 7 shows a screen displaying a menu of shapes, from which a user may select in order to create different virtual 3D shapes.
• FIG. 8A shows existing technology (prior art), in which a CAD drawing program has a 2D grid feature.
• FIG. 8B shows a 3D grid feature being used with a CID.
• FIG. 9 shows a CID being used, first, to specify three 3D points to define a virtual 3D plane, second, to draw on that plane after it has been brought up to the virtual 2D surface of the display screen, and third, to reinsert the plane, with the drawing on it, into virtual 3D space.
• FIG. 10 shows how a user may tilt a CID back and forth.
• FIG. 11 shows how a user may move a CID laterally along a display surface.
• FIG. 12A shows how the physical length of a CID may be shortened, while the virtual end of CID lengthens, or vice versa.
• FIG. 12B shows a system in which a user's head position is estimated by stereo infrared cameras. The user's head position is used to calculate the angle of the user's view into the virtual 3D space beyond the display screen.
• FIG. 12C shows a system in which a user's head position is tracked using a single camera and face recognition.
• FIG. 13 shows a prototype in which reflective IR tags on the CID are used to determine the length, position and orientation of the CID.
• FIG. 14 shows the same prototype, with the mounts for the IR tags exposed.
• FIG. 15 shows the shaft for an actuator and linear potentiometer, in a prototype.
• FIG. 16 shows how the shape of the virtual end of the CID may change.
• FIG. 17A is a diagram of a prototype in which the tip position and orientation of the CID are determined by magnetic sensors. InFIG. 17A, the prototype is fully extended to maximum physical length.
• FIG. 17B is a diagram of the same prototype, with the prototype collapsed to a shorter physical length.
• FIGS. 18A and 18B are diagrams showing sensor coils in a display screen. The sensor coils are used to detect changes in a magnetic field as a CID changes its tilt angle and 2D position relative to the display screen.
• FIG. 19 is a diagram of a cross-section of a prototype, in which the position, length and orientation of the CID are determined by tracking the position of reflective RF tags on the CID.
• FIGS. 20A and 20B show a linear potentiometer for measuring the length of a telescoping CID.
• InFIG. 20A, the CID is extended to full physical length. InFIG. 20B, the CID is collapsed to a shorter length.
• FIG. 21 is a diagram of a series elastic actuator for haptic feedback in a CID.
• FIG. 22 shows a cross-section of the end of a CID that is generally closest to the display screen. It includes a light sensor for optically sensing the orientation of the CID.
• FIG. 23 is a diagram that shows hardware for mechanically sensing the orientation of a CID.
• FIGS. 24A to 24D show how the display screen may be implemented in various form factors. The form factors shown inFIGS. 24A,24B,24C and24D, respectively, are a computer monitor, a table-mounted display, a wall-mounted display, and a handheld tablet computer.
• FIGS. 25A to 25D show alternate ways of implementing 3D display for use with a CID.
• FIG. 25A shows lenticular lenslets for 3D display.
• FIG. 25B shows a parallax barrier for 3D display.
• FIGS. 25C and 25D show integral imaging.
• FIG. 26 is a flow chart for face tracking, 3D rendering, and haptic feedback, in a prototype of this invention.
• FIGS. 27A to 27C show alternate embodiments, in which virtual images are displayed by projection. InFIG. 27A, an overhead projector is used. InFIG. 27B, the projection is from below. InFIG. 27C, a pico-projector is mounted on a CID.
• The above Figures illustrate some illustrative implementations of this invention, or provide information that relates to those implementations. However, this invention may be implemented in many other ways. The above Figures do not show all of the details of this invention.
DETAILED DESCRIPTION
• In exemplary implementations of this invention, a handheld, collapsible input device (CID) may be employed by a user to input and manipulate 3D information.
• FIG. 1 shows a user holding aCID1 and pressing thetip2 of theCID1 against adisplay screen3. Thedisplay screen3 displays a virtual end4 of the CID. From the user's perspective, it appears that the CID is being pushed through thedisplay screen3 into a virtual 3D space beyond the screen.
• FIG. 2A also shows a user holding aCID1 and touching itstip2 to thedisplay screen3. However, inFIG. 2A, thephysical CID1 is telescoped to its full length, and thedisplay screen3 does not display a virtual end of the CID. From a user's perspective, it appears that theCID1 is resting on the surface of thedisplay screen3 and is not pushed through thedisplay screen3 into the virtual 3D space beyond the screen.
• FIG. 2B depicts the same thing asFIG. 1, from a different perspective.
• A note regarding an illustration convention used herein: For ease of illustration, inFIG. 2B, the virtual end4 of the CID is displayed from the perspective of a viewer ofFIG. 2B, rather than from the perspective of the user who is holding the CID. This illustration convention will generally be followed for the remaining Figures. However, in practice, virtual 3D objects (including the virtual end of the CID) are usually rendered on the display screen from the perspective of the user who is holding the CID. (InFIG. 1, the line of gaze of the user holding the CID and of a viewer ofFIG. 1 coincide).
• FIG. 3 shows a user holding aCID301 and pressing thetip302 against thedisplay screen303. InFIG. 3, the user is using the virtual end304 of the CID to carve avirtual 3D slice305 from avirtual 3D apple306, leaving a virtual 3D hole307 where the slice used to be. The virtual end304 of the CID appears to be serrated or saw-like. More generally, the virtual end of a CID may be used to cut away portions of a virtual 3D object, or otherwise subtract from the volume of a virtual 3D object.
• FIG. 4A shows some ways in which a CID may be moved. InFIG. 4A, a user presses thetip402 of aCID400 against adisplay screen404. While doing so, the user may change the tilt angle of the CID relative to the display surface. For example, the user may move the top410 of thephysical CID400 in acircular orbit412 while keeping thetip402 in one place. In that case, theend406 of the virtual portion of the CID appears to orbit in virtual 3D space in the opposite direction.
• The pad at the end of thetip402 is sticky or otherwise has a high coefficient of friction, which tends to prevent thetip402 from slipping across the display screen as a user changes tilt angle. The tip may remain flat against the display screen even though the main body of the CID is tilted relative to the screen.
• The CID telescopes. Thus, inFIG. 4, if the user presses theCID400 against thedisplay surface404, the physical CID shortens in length, while the length of thevirtual portion406 of the CID appears to increase. The total apparent length of the CID, comprised of a physical portion and a virtual portion, remains the same (after taking into account foreshortening). Thus, the user experience is that, as the user holds the physical CID and pushes it toward a display screen, the end of the CID appears to be pushed through the display screen into the virtual 3D space beyond it. When the CID telescopes, it does so along its longitudinal axis, as indicated byarrow408 inFIG. 4B.
• Other movements are possible. For example, inFIG. 4A, a user may move thetip402 of the CID laterally along the display surface404 (e.g., along the x or y axes shown inFIG. 4A).
• FIG. 4B shows the CID being used to make a 3D drawing. A user holds thetip452 of aCID450 against adisplay surface454, and moves thevirtual end456 of theCID450 to draw virtual 3D lines458. In the example shown inFIG. 4B, the lines comprise a 3D drawing of a chair.
• FIG. 5A to 5C show a user holding a CID in one hand, and making gestures with the other hand. The gestures cause different 3D virtual shapes to be formed. The CID may be used to position these shapes.
• FIG. 5A shows a virtual square being created in this way. InFIG. 5A, a user holds aCID500 in one hand. On the user'sother hand510, the user wears aglove512.Fiducial markers514,516,518 are affixed to the exterior of the glove's thumb and first two fingers. With thisgloved hand510, the user makes a gesture. In this case, the gesture is an instruction to form a square. Avirtual square520 is displayed. Thevirtual tip504 of the CID is used to position thevirtual square520 thus formed. At least one camera is used to capture images of the gestures, and gesture recognition is performed by at least one processor.
• FIG. 5B shows a virtualrectangular cuboid550 being extruded in this way. InFIG. 5, the user has caused the cuboid to be extruded from the square shown inFIG. 5A.
• FIG. 5C shows avirtual ellipsoid560 being created in this way. The user is employing thevirtual end504 of the CID to position theellipsoid560 next to therectangular cuboid550.
• FIGS. 6A to 6F show different hand gestures that may be used as instructions for creating and manipulating various virtual shapes. For example, the CID system may recognize the gestures shown inFIGS. 6A,6B,6C,6D,6E and6F, respectively, as instructions to form a straight line, form a square, form an ellipse, extrude from an existing shape, lock a surface, or move in a specified direction.
• FIG. 7 shows a screen displaying a menu of shapes, from which a user may select in order to create different virtual 3D shapes. A menu ofshape icons700 appears on thedisplay screen704. The user may select any of these icons in order to create a desired shape. In the example shown inFIG. 7, the user selected thecylinder icon702 in order to form twovirtual cylinders706,708. Thevirtual end710 of the CID is used to position the cylinders in the virtual 3D space.
• FIG. 8A shows existing technology (prior art), in which a CAD drawing program has a 2D grid feature. When the grid feature is “on”, the cursor “snaps” to apoint800 on a 2D grid when the cursor comes sufficiently close to that point.
• FIG. 8B shows a 3D grid feature being used with a CID. As the virtual end of the CID comes close to apoint850 in a virtual 3D grid, the virtual end of the CID snaps to that point. This makes it easier to make 3D drawings with precise dimensions and precise alignment of the drawn objects.
• FIG. 9 shows a CID being used, first, to specify three points P1, P2, P3 to define avirtual 3D plane901, second, to draw on that plane after it has been brought up to thevirtual 2D surface902 of the display screen, and third, to reinsert the plane, with the drawing on it, into virtual 3D space atposition904. This has the advantage of allowing users to draw on a flat surface in 2D space, which may be easier because issues such as foreshortening may be avoided. Then the drawing on a flat plane may put back into 3D space (e.g., in a virtual orientation that is not parallel to the display surface).
• FIG. 10 shows how a user may tilt a CID back and forth. By rocking the physical CID back and forth while holding thetip1200 in place on adisplay surface1202, the user causes the virtual end of the CID to tilt back and forth.
• FIG. 11 shows how a user may move thetip1200 of a CID laterally along adisplay surface1202.
• FIG. 12A shows how the physical length of a CID may be shortened, while the virtual end of a CID lengthens, or vice versa. This may occur as a user presses atip1200 of the CID against adisplay screen1202, or as an actuator provides haptic feedback.
• FIG. 12B shows a system in which a user's head position is tracked by stereoinfrared cameras1210 and1212. The user wears an ear-mountedinfrared LED1214. At least oneprocessor1216 uses the camera images to determine the user's head position relative to the display screen, and based on that head position, to calculate the angle of the user's view into the virtual 3D space beyond the display screen.
• FIG. 12C shows a system in which a user'shead position1232 is tracked using asingle camera1230 and face recognition.
• FIG. 13 shows a prototype in which reflective IR tags on the CID are used to determine the length, position and orientation of the CID. Thereflective IR tags1300,1302 are affixed near the ends of the CID.
• InFIG. 13, a push-button1310 provides click and drag features. The push button is advantageous because it allows a user to indicate the beginning and ending of an interaction, and facilitates interactions such as selection and dragging of objects. The push button may be adapted to accept input from a user indicative of a variety of instructions.
• In addition,FIG. 13 shows certain internal hardware of a CID prototype. This hardware includes amicrocontroller1306 for, among other things, controlling avibration motor1304 for haptic feedback. It also includes a bluetooth communications unit1308 (including at least one transceiver) for communicating with at least one remote processor, which remote processor may in some cases be housed with or adjacent to the display screen. That remote processor may, for example, perform the rendering and other functions of the Java machine described inFIG. 26.
• FIG. 14 shows the same prototype asFIG. 13, with themounts1312,1314 for the IR reflective tags exposed.
• FIG. 15 shows an embodiment of this invention, in which amoveable shaft1508 transmits force from an actuator. The movement of theshaft1508 is measured by a linear potentiometer to determine the length of the CID. The linear potentiometer and actuator are housed inhousing1508. Mountingstages1500,1502 for IR reflective tags are affixed near the ends of the CID. In the example shown inFIG. 15, the IR reflective tags are used to determine orientation of the CID.
• FIG. 16 shows examples of how the shape of the virtual end of the CID may change. The samephysical CID1600 may at different times appear to project a virtual shape in the form of aserrated saw1602, aspoon1604, or apencil1606. The functionality of the CID may change appropriately. For example, thesaw1602 may be used for virtual cutting, thespoon1604 for scooping material away from a virtual object, and thepencil1606 for virtual drawing.
• FIGS. 17 is a diagram of a prototype extended to full length. In this prototype, acircuit1704 generates a magnetic field that is sensed by magnetic coils in a display screen (not shown inFIG. 17), in order to detect the tilt of the CID relative to the display screen and the 2D position of theCID tip1700 relative to that screen. The CID has telescoping parts, includinghousing part1702. As the CID telescopes (increasing or decreasing in length) a connectingrod1704 moves. That movement is sensed by alinear potentiometer1706. As the CID collapses, aspring1708 provides some resistance felt by the user.
• InFIG. 17A, apush button1718 is adapted to accept input from a user, including input regarding click and drag, and input regarding when an operation starts and ends. Amicrocontroller1710 controls avibration motor1716 for providing haptic feedback. Abluetooth communications unit1714 communicates with at least one remote processor, as described in more detail with respect toFIG. 13. Abattery1712 provides power for the electronics in the CID.
• FIG. 17B is a diagram of the same prototype as inFIG. 17A, with the prototype collapsed to a shorter physical length.
• FIGS. 18A and 18B are diagrams showing a magnetic sensing system for detecting the 2D position and tilt angle of the CID relative to a display surface. Amagnetic field1801 created by a resonance circuit or coil in thetip1800 of the CID induces voltage insensor coils1802 in the display surface. As the CID changes position or orientation, the magnetic field changes, and different patterns of inducedvoltages1804,1806 result. Based on these patterns, the 2D position and orientation of the CID relative to the screen may be determined. This method of magnetic sensing (in a context other than a CID) is described in U.S. Pat. No. 5,748,110, the entire disclosure of which is incorporated by reference herein.
• FIG. 19 is a diagram of a cross-section of a prototype, in which the length, orientation and position of the CID is determined by tracking the position ofreflective RF tags1920 and1922 on the CID. In this prototype, themicrocontroller1910, bluetooth communications unit1914,battery1912,push button1918,vibratory motor1916 andpad1900 at the tip of the CID correspond in structure and function toitems1700,1710,1714,1712,1718 and1716 inFIG. 17, which items are described above.
• FIGS. 20A and 20B show a linear potentiometer/actuator unit, in some embodiments of this invention. A connectingrod2048 moves as the CID telescopes in and out. The movement ofrod2048 is detected bylinear potentiometer2052,2050, in order to determine the length of the CID. Amotor2040 may be used to actuate movement ofrod2048, for haptic feedback. AnIC chip2030 is used to control themotor2040. The CID may telescope for reasons other than actuation bymotor2040. For example, it may increase in length due to force from a spring, or decrease in length as a user presses the CID against a display surface. IRreflective tags2020 and2022 may be mounted on the housing of the CID. Asrod2048 moves, the telescoping part of the housing to whichtag2020 is affixed moves with it.
• InFIG. 20A, the CID is extended to full physical length. InFIG. 20B, the CID is collapsed to a shorter length.
• FIG. 21 is a diagram of a model of a series elastic actuator (SEA) for haptic feedback in a CID. By using an SEA, accurate force feedback can be achieved dynamically regardless of the spring retraction. As shown inFIG. 21, an SEA may be implemented wherebytorque2100 from a motor is applied to amass2102, causing a motor shaft to move, which transmits force through aspring2103, resulting in an outputtedforce2104.
• In exemplary embodiments of this invention, haptic feedback is employed to heighten the illusion that the CID is directly interacting with virtual 3D objects in the virtual space beyond the display screen. For example, the CID may vibrate when the virtual end of the CID contacts a virtual object. Or, for example, the virtual end of a CID may be used to draw on a virtual tilted surface (i.e., a virtual surface whose apparent depth beyond the screen varies). As the virtual end of the CID moves from a lower point to a higher point4 on the tilted surface, a linear actuator may provide haptic feedback by lengthening the physical CID, so that it feels to a user as if the CID is rising as it goes up the tilted surface.
• FIG. 22 shows a cross-section of the end of a CID that is generally closest to the display screen. Sufficient space is left between thehousing2204 and asphere2202, so that thesphere2202 can tilt up to30 degrees from the longitudinal axis of the CID in any direction. This allows a user to change the orientation of the CID relative to a display screen, while leaving thepad2200 at the tip of the CID in a single place on the display screen. Thispad2200 at the tip of the CID may remain flat against the display while the tilt angle of the main body of the CID changes.
• The movement of the sphere relative to the housing of the CID may be optically sensed, in order to determine the tilt angle of the CID relative to the display surface. Alight source2208 emits light that reflects off of thesphere2202 and is detected by alight sensor2210. As thesphere2202 moves relative to thehousing2204, the pattern ofdots2206 on the surface of the sphere that are illuminated by thelight source2208 changes, thereby affecting the reflected light detected by thelight sensor2210. From these changes in reflected light, the orientation of the CID relative to the display surface may be determined.
• Alternately, rotation of the sphere may be mechanically sensed. For example, as shown inFIG. 23, the rotation of the sphere may mechanically causeelements2302,2304,2306 to move, and the movement of theseelements2302,2304,2306 may be measured.
• In addition, the orientation and/or position of the CID may be determined in other ways. For example, the tilt angle of the CID may be detected by gyroscopes and changes in lateral position or orientation of the CID may be detected with accelerometers.
• FIGS. 24A to 24D show how adisplay screen2400,2410,2420,2530 may be implemented in various form factors. The form factors shown inFIGS. 24A,24B,24C and24D, respectively, are acomputer monitor2402, a table-mounteddisplay2412, a wall-mounteddisplay2422, and ahandheld tablet computer2432.
• FIGS. 25A to 25D show some alternate ways of implementing 3D display for use with a CID.
• FIG. 25A shows lenticular lenslets for 3D display for use with a CID. An array oflenticular lenslets2500 in a display surface refracts light so that theright eye2504 and lefteye2508 of a user see different pixels, e.g.,pixels2502 and2506.
• FIG. 25B shows a parallax barrier for 3D display for use with a CID. Light from an array ofpixels2521 in a display surface passes through gaps in theparallax barrier2520. The pixels and parallax barrier are arranged so that different pixels, e.g.,2526 and2522, are seen by a viewer'sright eye2504 and lefteye2508.
• FIGS. 25C and 25D show integral imaging. Adisplay screen2802 displayselemental images2803. The light from theseelemental images2803 is refracted by an array oflenslets2800, causing areal image2808 to appear at aconvergence point2804 between the user'seye2806 and the lenslets.
• FIG. 26 is a flow chart for face tracking, 3D visual display and haptic feedback, in a prototype of this invention. Stereo IR cameras2614 capture images of user'sface position2628. A C++machine2612 communicates with the cameras using abluetooth unit2626. The C++machine checks if the head tracking is calibrated2644. If it is, the C++machine performsview transformation2666 and directs2684 a server to send the user's face position to a client. If the head tracking is not calibrated, calibration is performed, as described below.
• In the example shown inFIG. 26, the CID's tip position and orientation are detected usingmagnetic sensors2602. AJava machine2610 includes: aclient2628 that communicates with the server regarding face position, acommunications unit2608 for communicating with the magnetic sensors regarding the CID's tip position and orientation, abluetooth listener2606 for receiving communications from a chip in the CID, aninterface coordinator2642 that computes position of virtual objects in a computational coordinate, and avisual interface coordinator2664. TheJava machine2610 performs homographic3D view transformation2687 andrendering2690, and outputs signals fordisplay2692 of images. The Java machine also includes ahaptic interface coordinator2662 that checks2680 if haptic output is needed (i.e., whether the virtual end of the CID is touching a virtual object). If it is, instructions for haptic feedback are sent via abluetooth sender2622 to achip2604 in the CID. Thechip2604 includes anelectronics coordinator2620 for controlling a vibration motor orlinear actuator2640 that delivershaptic output2660, such as vibration, lengthening or shortening of the telescoping CID, or resistance to force applied by the user.
• IR head tracking may be calibrated by moving an infrared LED to four calibration points on the display screen, one at a time.
• In some embodiments, the CID interacts with a projected visual display. As shown inFIG. 27A, anoverhead projector2700 may project images of virtual objects on asurface2701. As a user presses aphysical CID2702 against thesurface2701, thephysical CID2702 collapses (shortens) and it appears to a user as if thevirtual end2704 pushes through thesurface2701 to the virtual space beneath thesurface2701.
• Alternately, the projection may be from below. As shown inFIG. 27B, aprojector2710 projects light that reflects off amirror2712 to form an image on adisplay screen2715 that is visible to a user viewing that screen from above. When theCID2714 is pressed against thedisplay screen2715, it appears to a user as if thevirtual end2716 of the CID goes through the screen into the virtual space beneath the screen.
• Alternately, as shown inFIG. 27C, apico projector2730 may be mounted on theCID2728 to project images onto asurface2732, and thevirtual end2734 of the CID may appear to push into the virtual space beneath that surface.
• In some embodiments, a CID may be used to control CNC milling or other subtractive manufacturing. As shown inFIG. 28, aprojector2800 may project images on asurface2802, such the top surface of a table remote from the CNC milling machine or even the top surface of material to be cut by the CNC milling machine. ACID2812 may be used to input and interact withvirtual images2804,2806,2808,2810 displayed on the surface by the projector. Digital information regarding those virtual 3D objects may be exported to aCNC milling machine2850. TheCNC milling machine2850 may remove material, leaving physical objects, e.g.2852,2854,2856,2858, that conform to this 3D information. Alternately, the CNC milling machine may drill cavities for a mold, which cavities conform to this 3D information. This approach is not limited to CNC milling, but may be used with any digitally controlled method of subtractive manufacturing.
• Item1216 inFIG. 12B is the one or more processors that may be employed for this invention, including for computing tip position, orientation and length of the CID, for face or head tracking, for face or gesture recognition, for rendering and visual display of images, for control of haptic feedback, for sound output, and for the other functions described inFIG. 26. These one or more processors may be located in various positions. For example, they may be housed in whole or part in a projector, a display screen, a CID, a CNC milling machine, or in one or more remote computers, including website servers.
• This invention may be implemented in many other ways, besides those described above. For example, gesture recognition may be implemented without fiducial markers or gloves. Or, for example, the push button unit may comprise multiple devices for accepting input from a user, including multiple buttons, dials, slider controls, piezoelectric sensors to measure the strength of the user's grip, or capacitive sensors for detecting touch input. Or, for example, gestures made by moving the CID in the air when it is not touching a display screen may be used to input user intent. Such gestures may be tracked, for example, by accelerometers, gyroscopes or other inertial measurement units.
Conclusion
• It is to be understood that the methods and apparatus which have been described above are merely illustrative applications of the principles of the invention. Numerous modifications may be made by those skilled in the art without departing from the scope of the invention. The scope of the invention is not to be limited except by the claims that follow.

Claims (1)

1. a collapsible input device.

Images