US20060284834A1 - Apparatus and methods for haptic rendering using a haptic camera view - Google Patents

Abstract

The invention provides systems and methods for using a “haptic camera” within a virtual environment and for using graphical data from the haptic camera to produce touch feedback. The haptic camera obtains graphical data pertaining to virtual objects within the vicinity and along the trajectory of a user-controlled haptic interface device. The graphical data from the camera is interpreted haptically, thereby allowing touch feedback corresponding to the virtual environment to be provided to the user.

Description

RELATED APPLICATIONS
• The present application is related to commonly-owned U.S. patent application entitled, “Apparatus and Methods for Haptic Rendering Using Data in a Graphics Pipeline,” by Itkowitz, Shih, Midura, Handley, and Goodwin, filed under Attorney Docket No. SNS-012 on even date herewith, the text of which is hereby incorporated by reference in its entirety; the present application is also related to commonly-owned international (PCT) patent application entitled, “Apparatus and Methods for Haptic Rendering Using Data in a Graphics Pipeline,” by Itkowitz, Shih, Midura, Handley, and Goodwin, filed under Attorney Docket No. SNS-012PC on even date herewith, the text of which is hereby incorporated by reference in its entirety; the present application claims the benefit of U.S. Provisional Patent Application No. 60/584,001, filed on Jun. 29, 2004, the entirety of which is incorporated by reference herein.
FIELD OF THE INVENTION
• The invention relates generally to haptic rendering of virtual environments. More particularly, in certain embodiments, the invention relates to the haptic rendering of a virtual environment using data from the graphics pipeline of a 3D graphics application.
BACKGROUND OF THE INVENTION
• Haptic technology involves simulating virtual environments to allow user interaction through the user's sense of touch. Haptic interface devices and associated computer hardware and software are used in a variety of systems to provide kinesthetic and/or tactile sensory feedback to a user in addition to conventional visual feedback, thereby affording an enhanced man/machine interface. Haptic systems are used, for example, in manufactured component design, surgical technique training, industrial modeling, robotics, and personal entertainment. An example haptic interface device is a six degree of freedom force reflecting device as described in co-owned U.S. Pat. No. 6,417,638, to Rodomista et al., the description of which is incorporated by reference herein in its entirety.
• A haptic rendering process provides a computer-based kinesthetic and/or tactile description of one or more virtual objects in a virtual environment. A user interacts with the virtual environment via a haptic interface device. Analogously, a graphical rendering process provides a graphical description of one or more virtual objects in a virtual environment. Typically, a user interacts with graphical objects via a mouse, joystick, or other controller. Current haptic systems process haptic rendering data separately from graphical rendering data.
• The graphical rendering of 3D virtual environments has been enhanced by the advent of 3D graphics application programming interfaces (APIs), as well as 3D graphics (video) cards. A programmer may create or adapt a 3D graphics application for rendering a 3D graphics virtual environment using the specialized libraries and function calls of a 3D graphics API. Thus, the programmer avoids having to write graphics rendering code that is provided in the API library. As a result, the task of programming a 3D graphics application is simplified. Furthermore, graphics standards have developed such that many currently-available 3D graphics applications are compatible with currently-available 3D graphics API's, allowing a user to adapt the 3D graphics application to suit his/her purpose. Examples of such 3D graphics API's include OpenGL, DirectX, and Java 3D.
• In addition to 3D graphics API's, 3D graphics cards have also improved the graphical rendering of 3D virtual objects. A 3D graphics card is a specialized type of computer hardware that speeds the graphical rendering process. A 3D graphics card performs a large amount of the computation work necessary to translate 3D information into 2D images for viewing on a screen, thereby saving CPU resources.
• While 3D graphics API's and graphics cards have significantly improved the graphical rendering of 3D objects, the haptic rendering of 3D objects in a virtual environment is a comparatively inefficient process. Haptic rendering is largely a separate process from graphical rendering, and currently-available 3D graphics applications are incompatible with haptic systems, since graphics applications are not designed to interpret or provide haptic information about a virtual environment.
• Furthermore, haptic rendering processes are generally computation-intensive, requiring high processing speed and a low latency control loop for accurate force feedback rendering. For example, in order to realistically simulate touch-based interaction with a virtual object, a haptic rendering process must typically update force feedback calculations at a rate of about 1000 times per second. This is significantly greater than the update rate needed for realistic dynamic graphics display, which is from about 30 to about 60 times per second in certain systems. For this reason, current haptic systems are usually limited to generating force feedback based on single point interaction with a virtual environment. This is particularly true for haptic systems that are designed to work with widely-available desktop computers and workstations with state-of-the-art processors.
• Thus, there is a need for increased efficiency in haptic rendering. Improvement is needed, for example, to facilitate the integration of haptics with currently-available 3D applications, to permit greater haptic processing speeds, and to enable the use of more sophisticated force feedback techniques, thereby increasing the realism of a user's interaction with a virtual environment.
SUMMARY OF THE INVENTION
• The invention provides systems and methods for using a “haptic camera” within a virtual environment and for using graphical data from the haptic camera to produce touch feedback. The haptic camera obtains graphical data pertaining to virtual objects within the vicinity and along the trajectory of a user-controlled haptic interface device. The graphical data from the camera is interpreted haptically, thereby allowing touch feedback corresponding to the virtual environment to be provided to the user.
• The efficiency of haptic rendering is improved, because the view volume can be limited to a region of the virtual environment that the user will be able to touch at any given time, and further, because the method takes advantage of the processing capacity of the graphics pipeline. This method also allows haptic rendering of portions of a virtual environment that cannot be seen in a 2D display of the virtual object, for example, the back side of an object, the inside of crevices and tunnels, and portions of objects that lie behind other objects.
• A moving haptic camera offers this advantage. Graphical data from a static camera view of a virtual environment can be used for haptic rendering; however, it is generally true that only geometry visible in the view direction of the camera can be used to produce touch feedback. A moving camera (and/or multiple cameras) allows graphical data to be obtained from more than one view direction, thereby allowing the production of force feedback corresponding to portions of the virtual environment that are not visible from a single static view. The interaction between the user and the virtual environment is further enhanced by providing the user with a main view of the virtual environment on a 2D display while, at the same time, providing the user with haptic feedback corresponding to the 3D virtual environment. The haptic feedback is updated according to the user's manipulation of a haptic interface device, allowing the user to “feel” the virtual object at any position, including regions that are not visible on the 2D display.
• The invention provides increased haptic rendering efficiency, permitting greater haptic processing speeds for more realistic touch-based simulation. For example, in one embodiment, the force feedback computation speed is increased from a rate of about 1000 Hz to a rate of about 10,000 Hz or more. Furthermore, the invention allows more sophisticated haptic interaction techniques to be used with widely-available desktop computers and workstations. For example, forces can be computed based on the interaction of one or more points, lines, planes, and/or spheres with virtual objects in the virtual environment, not just based on single point interaction. More sophisticated haptic interface devices that require multi-point interaction can be used, including pinch devices, multi-finger devices, and gloves, thereby enhancing the user's haptic experience. Supported devices include kinesthetic and/or tactile feedback devices. For example, in one embodiment, a user receives tactile feedback when in contact with the surface of a virtual object such that the user can sense the texture of the surface.
• In one aspect of the invention, a method is provided for haptically rendering a virtual object in a virtual environment. The method includes determining a haptic interface location in a 3D virtual environment corresponding to a haptic interface device in real space. A first virtual camera is positioned at the haptic interface location, and graphical data corresponding to the virtual environment is accessed from this first virtual camera. Additionally, the method comprises determining a position of the haptic interface location in relation to one or more geometric features of a virtual object in the virtual environment—for example, a surface, point, line, or plane of (or associate with) the virtual object—by using graphical data from the first virtual camera. The method also includes determining an interaction force based at least in part on the position of the haptic interface location in relation to the geometric feature(s) of the virtual object. In one embodiment, the interaction force is delivered to a user through the haptic interface device. In a preferred embodiment, the position of the first camera is updated as the haptic interface location changes, according to movement of the haptic interface device.
• The invention also provides a two-pass rendering technique using two virtual cameras. For example, the invention provides methods using a first virtual camera view dedicated for use in haptically rendering a 3D virtual environment and a second virtual camera view for graphically rendering the virtual environment for display. Accordingly, in one embodiment, the invention includes the steps of positioning a second virtual camera at a location other than the haptic interface location and accessing graphical data from the second virtual camera corresponding to the virtual environment. In one embodiment, the second virtual camera is at a fixed location, while the first virtual camera moves, for example, according to the movement of the haptic interface location.
• Preferred methods of the invention leverage the processing capability of the graphics pipeline for haptic rendering. For example, graphical data corresponding to the view(s) from one or more virtual cameras is accessed from a graphics pipeline of a 3D graphics application. In one embodiment, the step of determining a position of the haptic interface location using data from the first virtual camera includes determining a world-view transformation that maps coordinates corresponding to the haptic virtual environment (i.e. world coordinates) to coordinates corresponding to the first virtual camera (i.e. view coordinates). The world-view transformation can be customized for translating and rotating the camera to view the scene as if attached to the position of the haptic device's proxy in the virtual environment (i.e. the haptic interface location). Additional transforms may be determined and/or applied, including a shape-world transformation, a view-clip transformation, a clip-window transformation, a view-touch transformation, and a touch-workspace transformation.
• The invention also provides a method of determining what the view looks like from the “haptic camera.” Generally, in order to specify a 3D world-view transformation, a camera eye position and a look direction are needed. Thus, in one embodiment, the step of determining a world-view transformation includes determining an eye position and a look direction. To determine the eye position, the position of the haptic interface location (i.e. the virtual proxy position) is sampled. In order to avoid undesirable jitter, the eye position is preferably updated only when the virtual proxy moves beyond a threshold distance from the current eye position. To determine the look direction, a vector representing the motion of the haptic interface location is determined. Preferably, the look direction is determined by the motion of the proxy and optionally by the contact normal, for example, if in contact with a virtual object and constrained on the surface of the contacted object. For example, when moving in free space, the look direction is the normalized motion vector. When in contact with a virtual object, the look direction becomes a linear combination of the normalized motion vector and the contact normal.
• In one embodiment, a view volume associated with the first virtual camera is sized to exclude geometric elements that lie beyond a desired distance from the haptic interface location. This involves culling the graphical data to remove geometric primitives that lie outside the view volume of the first virtual camera. In one embodiment, hardware culling is employed, where primitives are culled by graphics hardware (i.e. a graphics card). In another embodiment, culling involves the use of a spatial partition, for example, an octree, BSP tree, or other hierarchical data structure, to exclude graphical data outside the view volume. Both hardware culling and a spatial partition can be used together. For example, where the number of primitives being culled by the graphics hardware is large, the spatial partition can reduce the amount of data sent to the hardware for culling, allowing for a more efficient process.
• The types of graphical data obtained from the first virtual camera include, for example, data in a depth buffer, a feedback buffer, a color buffer, a selection buffer, an accumulation buffer, a texture map, a fat framebuffer, rasterization primitives, application programming interface input data, and/or state data.
• As the term is used herein, a fat framebuffer is also known as and/or includes a floating point auxiliary buffer, an attribute buffer, a geometry buffer, and/or a super buffer. Fat framebuffers are flexible and allow a user to store a wide variety of different types of graphical data. A fat framebuffer can include, for example, vertex positions, normals, color, texture, normal maps, bump maps, and/or depth data. Fat framebuffers can be used as input in custom pixel and/or vertex shader programs that are run on graphics hardware (i.e. on the graphics card). In one embodiment, a fat framebuffer is used to capture vertex positions and normals. For example, in one embodiment, primitives are graphically rendered to a fat framebuffer, and pixel shading and/or vertex shading is performed using data from the fat framebuffer in the haptic rendering of a virtual environment. In one embodiment, a deferred shading process is used to render graphics primitives to a fat framebuffer.
• It is possible to use graphics hardware to graphically render virtual objects to a texture map instead of a buffer. Thus, throughout the specification, where graphical data is described as being stored in or read from a buffer, the data may alternately be stored in or read from a texture map.
• In one embodiment, determining the position of the haptic interface location using data from the first virtual camera includes performing an intersection test to determine an intersection point and intersection normal in screen space, and transforming the coordinates of the intersection point and intersection normal from screen space to object space. Alternatively, the graphical data can be used to determine the closest geometric feature, such as a point, line or plane, to the virtual proxy via a projection test. These geometric queries are important for haptic rendering of 1D, 2D, and/or 3D contacts and/or constraints.
• In another aspect, a system is provided for haptically rendering a virtual object in a virtual environment. The system comprises a graphics thread that generates a visual display of a virtual environment, a collision thread that uses input from the graphics thread to determine if a user-directed virtual proxy collides with a surface within the virtual environment, and a servo thread that generates force to be applied to a user in real space though a haptic interface device according to input from the collision thread.
• In one embodiment, the graphics thread refreshes the visual display at a rate within a range, for example, from about 5 Hz to about 150 Hz, or from about 30 Hz to about 60 Hz. Refresh rates above and below these levels are possible as well. In one embodiment, the collision thread performs a collision detection computation at a rate within a range, for example, from about 30 Hz to about 200 Hz, or from about 80 Hz to about 120 Hz. Computation rates above and below these levels are possible as well. In one embodiment, the servo thread refreshes the force to be applied through the haptic interface device at a rate within a range from about 1000 Hz to about 10,000 Hz. Force refresh rates above and below these levels are possible as well. In one embodiment, the servo thread includes a force shader.
• In yet another aspect, an apparatus is provided for providing haptic feedback to a user of a 3D graphics application. The apparatus comprises a user-controlled haptic interface device adapted to provide a user input to a computer and to transmit force to a user. The apparatus also includes computer software that, when operating with the computer and the user input, is adapted to determine force transmitted to the user. The force transmitted to the user is determined by a process that comprises determining a haptic interface location in a 3D virtual environment corresponding to a location of the haptic interface device in real space and positioning a first virtual camera substantially at the haptic interface location. Graphical data is then accessed using the first virtual camera. A position of the haptic interface location in relation to a surface of a virtual object in the virtual environment is determined using the graphical data from the first virtual camera. Finally, an interaction force is determined, based at least in part on the position of the haptic interface location in relation to the surface of the virtual object.
• There may be any number of cameras in a given scene. For example, each individual virtual object in a scene may have its own camera; thus, the number of cameras is unlimited. This allows a user to adapt the camera view to best suit individual objects, which allows for further optimization. For example, the camera position and view frustum for objects that are graphically rendered (and/or haptically rendered) using the depth buffer can be set differently than those rendered using the feedback buffer. In addition, there can be multiple haptic devices in a given scene. Each haptic device can have a different camera for each object, since the position and motion of the haptic devices will generally be different.
BRIEF DESCRIPTION OF DRAWINGS
• The objects and features of the invention can be better understood with reference to the drawings described below, and the claims. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the drawings, like numerals are used to indicate like parts throughout the various views.
• FIG. 1 is a block diagram featuring a method of haptically rendering one or more virtual objects in a virtual environment using data in a graphics pipeline, according to an illustrative embodiment of the invention.
• FIG. 2 is a schematic diagram illustrating a system for haptically rendering a virtual environment using data in a graphics pipeline, the diagram showing an interaction between a 3D graphics application, a graphics application programming interface (API), a 3D graphics card, and a haptics API, according to an illustrative embodiment of the invention.
• FIG. 3 is a schematic diagram illustrating a graphics pipeline of a 3D graphics application, according to an illustrative embodiment of the invention.
• FIG. 4A is a schematic diagram illustrating a system for haptically rendering a virtual environment using data in a graphics pipeline, the system including a graphics thread, a collision thread, and a servo thread, according to an illustrative embodiment of the invention.
• FIG. 4B is a schematic diagram illustrating the system ofFIG. 4A in further detail, according to an illustrative embodiment of the invention.
• FIG. 5 is a schematic diagram illustrating a servo thread of a haptics rendering pipeline, according to an illustrative embodiment of the invention.
• FIG. 6 is a schematic diagram illustrating a system for haptically rendering a virtual environment using data in a graphics pipeline, the diagram showing how third-party 3D graphics application software is integrated with the system, according to an illustrative embodiment of the invention.
• FIG. 7 is a block diagram featuring a method of delivering interaction force to a user via a haptic interface device, the force based at least in part on graphical data from a virtual camera located at a haptic interface location, according to an illustrative embodiment of the invention.
• FIG. 8A is a screenshot of a virtual object in a virtual environment as imaged from a fixed camera view, the screenshot indicating a haptic interface location, or proxy position, representing the position of a user in the virtual environment, according to an illustrative embodiment of the invention.
• FIG. 8B is a screenshot of the virtual object ofFIG. 8A as imaged from a moving camera view located at the haptic interface location shown inFIG. 8A, where graphical data from the images of either or both ofFIG. 8A andFIG. 8B is/are used to haptically render the virtual object, according to an illustrative embodiment of the invention.
• FIG. 9 is a block diagram featuring a 3D transformation pipeline for displaying 3D model coordinates on a 2D display device and for haptic rendering via a haptic interface device, according to an illustrative embodiment of the invention.
• FIG. 10 is a schematic diagram illustrating the specification of a viewing transformation for a haptic camera view, according to an illustrative embodiment of the invention.
• FIG. 11 is a schematic diagram illustrating the specification of a look direction for use in determining a viewing transformation for a haptic camera view when the position of a haptic interface location is constrained on the surface of a virtual object, according to an illustrative embodiment of the invention.
• FIG. 12 is a block diagram featuring a method for interpreting data for haptic rendering by intercepting data from a graphics pipeline via a pass-through dynamic link library (DLL), according to an illustrative embodiment of the invention.
• FIG. 13 is a schematic diagram illustrating a system for haptically rendering a virtual environment using data intercepted from a graphics pipeline of a 3D graphics application via a pass-through dynamic link library, according to an illustrative embodiment of the invention.
DETAILED DESCRIPTION OF THE INVENTION
• Throughout the description, where an apparatus is described as having, including, or comprising specific components, or where systems, processes, and methods are described as having, including, or comprising specific steps, it is contemplated that, additionally, there are apparati of the present invention that consist essentially of, or consist of, the recited components, and that there are systems, processes, and methods of the present invention that consist essentially of, or consist of, the recited steps.
• It should be understood that the order of steps or order for performing certain actions is immaterial so long as the invention remains operable. Moreover, two or more steps or actions may be conducted simultaneously.
• A computer hardware apparatus may be used in carrying out any of the methods described herein. The apparatus may include, for example, a general purpose computer, an embedded computer, a laptop or desktop computer, or any other type of computer that is capable of running software, issuing suitable control commands, receiving graphical user input, and recording information. The computer typically includes one or more central processing units for executing the instructions contained in software code that embraces one or more of the methods described herein. The software may include one or more modules recorded on machine-readable media, where the term machine-readable media encompasses software, hardwired logic, firmware, object code, and the like. Additionally, communication buses and I/O ports may be provided to link any or all of the hardware components together and permit communication with other computers and computer networks, including the internet, as desired. As used herein, the term “3D” is interpreted to include 4D, 5D, and higher dimensions.
• It is an object of the invention to leverage the processing power of modern 3D graphical rendering systems for use in the haptic rendering of a virtual environment containing, for example, one or more virtual objects. It is a further object of the invention to introduce a virtual camera in the virtual environment located at a haptic interface location, which can be moved by a user. The view volume of this “haptic camera” can be sized to exclude unnecessary regions of the virtual environment, and the graphical data can be used for haptically rendering one or more virtual objects as the user moves about the virtual environment.
• FIG. 1 is a block diagram100 featuring a method of haptically rendering one or more virtual objects in a virtual environment using data in a graphics pipeline of a 3D graphics application. The method shown inFIG. 1 includes three main steps—accessing data in a graphics pipeline of a3D graphics application102; interpreting data for use inhaptic rendering105; and haptically rendering one or more virtual objects in thevirtual environment110.
• A graphics pipeline generally is a series of steps, or modules, that involve the processing of 3D computer graphics information for viewing on a 2D screen, while at the same time rendering an illusion of three dimensions for a user viewing the 2D screen. For example, a graphics pipeline may comprise a modeling transformation module, in which a virtual object is transformed from its own object space into a common coordinate space containing other objects, light sources, and/or one or more cameras. A graphics pipeline may also include a rejection module in which objects or primitives that cannot be seen are eliminated. Furthermore, a graphics pipeline may include an illumination module that colors objects based on the light sources in the virtual environment and the material properties of the objects. Other modules of the graphics pipeline may perform steps that include, for example, transformation of coordinates from world space to view space, clipping of the scene within a three dimensional volume (a viewing frustum), projection of primitives into two dimensions, scan-conversion of primitives into pixels (rasterization), and 2D image display.
• Information about the virtual environment is produced in the graphics pipeline of a 3D graphics application to create a 2D display of the virtual environment as viewed from a given camera view. The camera view can be changed to view the same virtual environment from a myriad of vantage points. The invention capitalizes on this capability by haptically rendering the virtual environment using graphical data obtained from one or more virtual cameras. In one embodiment, the invention accesses data corresponding to either or both of aprimary view115 and ahaptic camera view120, where theprimary view115 is a view of the virtual environment from a fixed location, and thehaptic camera view120 is a view of the virtual environment from a moving location corresponding to a user-controlled haptic interface location. Thehaptic camera view120 allows a user to reach behind an object to feel what is not immediately visible on the screen (the primary view115).
• Information about the geometry of the virtual environment can be accessed by making the appropriate function call to the graphics API. Data can be accessed from one or more data buffers—for example, adepth buffer125, as shown in the block diagram ofFIG. 1, and a feedback buffer130 (or its equivalent). Use of this data for haptic rendering enables the reuse of the scene traversal and graphics API rendering state and functionality.
• Thedepth buffer125 is typically a two-dimensional image containing pixels whose intensities correspond to depth (or height) values associated with those pixels. The depth buffer is used during polygon rasterization to quickly determine if a fragment is occluded by a previously rendered polygon. The depth buffer is accessed by making the appropriate function call to the graphics API. This information is then interpreted instep105 of the method ofFIG. 1 for haptic use. Using depth buffer data provides several advantages. For example, depth buffer data is in a form whereby it can be used to quickly compute 3D line segment intersections and inside/outside tests. Furthermore, the speed at which these depth buffer computations can be performed is substantially invariant to the density of the polygons in the virtual environment. This is because the data in the depth buffer is scalar data organized in a 2D grid having known dimensions, the result of rasterization and occlusion processing.
• Other data buffers in the graphics pipeline include acolor buffer135, astencil buffer140, and anaccumulation buffer145. Thecolor buffer135 can store data describing the color and lighting conditions of vertices. Theaccumulation buffer145 can be used to accumulate precise intermediate rendering data. Thestencil buffer140 can be used to flag attributes for each pixel and perform logic operations as part of pixel fragment rendering. These buffers may be used, for example, to modify and/or map various haptic attributes—for example, friction, stiffness, and/or damping—to the pixel locations of the depth buffer. For example,color buffer data135 may be used to encode surface normals for force shading.Stencil buffer data140 can indicate whether or not to allow drawing for given pixels.Stencil buffer data140 can also be incremented or decreased every time a pixel is touched, thereby counting the number of overlapping primitives for a pixel. The stencil contents can be used directly or indirectly for haptic rendering. For example, it can be used directly to flag pixels with attributes for enabling and/or disabling surface materials, such as areas of friction. It can also be used indirectly for haptics by graphically rendering geometry in a special way for haptic exploration, like depth peeling or geometry capping.
• Encoding normals in the color buffer includes setting up the lighting of the virtual environment so that normals may be mapped into values in the color buffer, wherein each pixel contains four components <r,g,b,a>. A normal vector <x,y,z> can be stored, for example, in the <r,g,b> components by modifying the lighting equation to use only the diffuse term and by applying the lighting equation for six colored lights directed along the local axes of the object coordinate space. For example, the x direction light is colored red, the y direction light is colored green, and the z direction light is colored blue, so that the directional components of the pixels match their color components. Then the lighting equation is written as a summation of dot products scaled by the respective color of the light. This results in normal values which may be used, for example, for smooth force shading.
• Data contained in thedepth buffer125,feedback buffer130,color buffer135,stencil buffer140, and/oraccumulation buffer145, among other data buffers, may be altered by hardware such as a graphics card. A graphics card can perform some of the graphical data processing required to produce 2D screen views of 3D objects, thereby saving CPU resources. Data produced from such hardware-acceleratedgeometry modifications150 is used in certain embodiments of the invention. Modern graphics cards have the ability to execute custom fragment and vertex shading programs, enabling a programmable graphics pipeline. It is possible to leverage the results of such geometry modifications for purposes of haptic rendering. For example, view-dependent adaptive subdivision and view-dependent tessellation be used to produce smoother-feeling surfaces. Displacement mapping can result in the haptic rendering of surface details such as ripples, crevices, and bumps, which are generated onboard the graphics card.
• In one embodiment, an “adaptive viewport” is used to optimize depth buffer haptic rendering, wherein the bounds of the viewport are read-back from the graphics card. For example, the entire viewport may not be needed; only the portion of the depth buffer that contains geometry within the immediate vicinity of the haptic interface location may be needed. In an adaptive viewport approach, the bounds of the viewport that are to be read-back from the graphics card are determined by projecting the haptic interface location onto the near plane and by determining a size based on a workspace to screen scale factor. In this way, it is possible to ensure that enough depth buffer information is obtained to contain a radius of workspace motion mapped to screen space.
• Certain 3D graphics API's, for example, OpenGL, offer a mode of operation called feedback mode, which provides access to the feedback buffer130 (FIG. 1) containing information used by the rasterizer for scan-filling primitives to the viewport. In one embodiment, the method ofFIG. 1 includes the step of accessing thefeedback buffer130 and interpreting the data from the feedback buffer for haptic use. Thefeedback buffer130 provides access to the primitives within a view volume. The view volume may be sized to include only portions of the virtual environment of haptic interest. Therefore, haptic rendering of primitives outside the view volume need not take place, and valuable processing resources are saved.
• It is possible to simulate non-uniform surface properties using data in thefeedback buffer130 via groups of primitives, per vertex properties, and/or via texture mapping. In certain embodiments, the feedback buffer provides data that is more precise than depth buffer data, since primitives in the feedback buffer have only undergone a linear transformation, whereas the depth buffer represents rasterized primitives, thereby possibly introducing aliasing errors.
• Step105 of the method ofFIG. 1 is directed to interpreting the graphical rendering data accessed instep102 for haptic use. In one embodiment,step105 involves performing anintersection test160 to determine an intersection point and a normal in screen space, and transforming the intersection point coordinates and normal coordinates to objectspace165. The point and normal together define a local plane tangent to the surface of the virtual object. In one embodiment in which a depth values from adepth buffer125 are used, the intersection test ofstep160 is essentially a pixel raycast along a line segment, where the depth buffer is treated as a height map. A line segment that is defined in object space is transformed into screen space and tested against the height map to find an intersection. An intersection is found by searching along the line segment (in screen space) and comparing depth values to locations along the line segment. Once a crossing has been determined, a more precise intersection can be determined by forming triangles from the local depth values. This provides an intersection point and an intersection normal, where the intersection normal is normal to a surface corresponding to the screen space height map at the intersection point. Instep165, the intersection point and normal are transformed back into object space to be used as part of a haptic rendering method. Example haptic rendering methods are described in co-owned U.S. Pat. No. 6,191,796 to Tarr, U.S. Pat. No. 6,421,048 to Shih et al., U.S. Pat. No. 6,552,722 to Shih et al., U.S. Pat. No. 6,417,638 to Rodomista et al., and U.S. Pat. No. 6,671,651 to Goodwin et al., the disclosures of which are incorporated by reference herein in their entirety.
• In one embodiment in which screenspace rasterization primitives130 are accessed instep102 in the method ofFIG. 1, the intersection test ofstep160 also involves transforming a line segment from object space to screen space and performing a line intersection test against candidate primitives. An intersection point and intersection normal are found along the line segment and are transformed back into object space for haptic rendering.
• Step110 of the method ofFIG. 1 is directed to haptically rendering one or more virtual objects in the virtual environment using the interpreted data fromstep105. In one embodiment, the haptic rendering step includes determining a haptic interface location in the virtual environment corresponding to a user's position in real space (i.e. via a user's manipulation of a haptic interface device)170, locating one or more points on the surface of one or more virtual objects in the virtual environment (i.e. the surface point nearest the haptic interface location)175, and determining aninteraction force180 according to the relationship between the haptic interface location and the surface location(s). Thus, step110 may involve determining when a collision occurs between a haptic interface location (i.e. a virtual tool location) and a virtual object. In one embodiment, a collision occurs when the haptic interface location crosses through the surface of a virtual object. The interaction force that is determined instep180 may be delivered to the user through the haptic interface device. The determination and delivery of a feedback force to a haptic interface device is described, for example, in co-owned U.S. Pat. Nos. 6,191,796, 6,421,048, 6,552,722, 6,417,638, and 6,671,651, the disclosures of which are incorporated by reference herein in their entirety.
• FIG. 2 is a schematic diagram200 illustrating, in a simplified way, a system for haptically rendering a virtual environment using data in a graphics pipeline. The diagram shows an interaction between a3D graphics application202, a graphics application programming interface (API)205, a3D graphics card215, and ahaptics API210. Certain methods of the invention may be embodied in, and may be performed using, thehaptics API210, thegraphics API205, the3D graphics application202, and/or combinations thereof.
• A3D graphics application202 may be written or adapted to enable the user of the application to see a visual representation of a 3D virtual environment on a two-dimensional screen while “feeling” objects in the 3D virtual environment using a peripheral device, such as a haptic interface device. The graphics application makes function calls referencing function libraries in a graphics API265. The graphics API communicates with the3D graphics card215 in order to graphically render a virtual environment. A representation of at least a portion of the virtual environment is displayed on adisplay device220.
• Thesystem200 ofFIG. 2 permits a programmer to write function calls in the3D graphics application202 to call ahaptics API210 for rendering a haptic representation of at least a portion of the virtual environment. Thehaptics API210 accesses graphical rendering data from the 3D graphics pipeline by making function calls to the graphics API. The graphical data may include a data buffer, such as a depth buffer or feedback buffer. Thesystem200 interprets the graphical data to haptically render at least a portion of the virtual environment. The haptic rendering process may include determining a force feedback to deliver to the user via ahaptic interface device230. A haptic device API and ahaptic device driver225 are used to determine and/or deliver the force feedback to the user via thehaptic interface device230.
• Thehaptics API210 performs high-level haptics scene rendering, and thehaptic device API225 performs low-level force rendering. For example, the high-level haptics API210 provides haptic rendering of shapes and constraints and the low-levelhaptic device API225 queries device state, sends forces, and/or performs thread control, calibration, and error handling. The 3D graphics application may make direct calls to either or both thehaptics API210 and thehaptic device API225.
• FIG. 3 illustrates a3D graphics pipeline300, in which graphical data describing one or more 3D objects in a virtual environment is used to create a 2D representation for display on a two-dimensional screen. Graphical data corresponding to thescene geometry302, acamera view305, andlighting310, undergoes a series oftransformations315. The resultant primitives data then undergoes arasterization process320, producing 2D graphical data that may be stored in 2D buffers, for example, acolor buffer330 and adepth buffer335. The primitives data as it exists prior to rasterization can be accessed, for example, via afeedback buffer325. Methods of the invention use the graphical data in the3D graphics pipeline300, for example, thefeedback buffer325, thedepth buffer335, and thecolor buffer330, for haptically rendering the virtual environment, as described in more detail herein.
• FIG. 4A is a simplified schematic diagram illustrating components of asystem400 for haptically rendering a virtual environment using data in a graphics pipeline. Thesystem400 comprisescomputational elements402 including agraphics thread405, acollision thread410, and aservo thread415, as well as adisplay device420 and ahaptic interface device425. Thegraphics thread405 is adapted to generate a visual display of a virtual environment to be displayed on thedisplay device420. Thecollision thread410 determines if a user-directed virtual proxy (i.e. a haptic interface location) collides with a surface within the virtual environment, based on input from thegraphics thread405. Theservo thread415 determines (and may generate) a force to be applied to a user in real space via thehaptic interface device425 according to input from thecollision thread410.
• FIG. 4B is a schematic diagram427 illustrating the system ofFIG. 4A in further detail. Thegraphics thread405 is adapted to generate a visual display of a virtual environment. API commands430 are used to access graphical rendering data, including the depth buffer437 andfeedback buffer440. In one embodiment, this data is used for both haptic and graphical rendering. Additionally, the user (and/or the 3D graphics software programmer) may define custom shapes435 andcustom constraints442 independent of thegraphics API430. Custom shapes435 include, for example, NURBS shapes, SubDs, voxel-shapes, and the like. Custom constraints include, for example, constraint to surfaces, lines, curves, arcs, and the like.Standard force effects447 and user-definedforce effects450 may also be assigned in thegraphics thread405. Additional software, for example, third-party software, may be integrated with thegraphics thread405, for example, in a user-definedproxy module445. In certain embodiments, thegraphics thread405 refreshes thedisplay device420 at a rate, for example, within the range from about 10 Hz to about 150 Hz, within the range from about 20 Hz to about 110 Hz, or, preferably, within the range from about 30 Hz to about 60 Hz. Rates above and below these levels are possible as well.
• Thecollision thread410 ofFIG. 4B is adapted to determine whether a user-directed virtual proxy collides with a surface within the virtual environment. In one embodiment, the collision thread comprises three modules, including ashape collision renderer453, aconstraint collision renderer455, and aneffect renderer460. Theshape collision renderer453 is adapted to calculate the shapes in the virtual environment and to identify their collision with each other or with proxies. Theshape collision renderer453 may use data from the depth buffer437, thefeedback buffer400, and user defined shape data435. Similarly, theconstraint collision renderer455 may use data from the depth buffer437,feedback buffer440, and from user-definedconstraints442. Theeffect renderer460 may use data from the standardforce effects module447 and from the user-definedforce effects module450. One of the functions of theeffect renderer460 is to compose theforce shader480 in theservo thread415, so that theforce shader480 is able to simulate force effects at the typically higher servo loop rate. For example, theeffect renderer460 can start, stop, and manage parameters for theforce shader480. In certain embodiments, thecollision thread410 may perform a collision detection computation at a rate within the range from about 10 Hz to about 200 Hz, from about 80 Hz to about 120 Hz, or, preferably, at about 100 Hz. Rates above and below these levels are possible as well.
• Next, theservo thread415 generates a force to be applied to a user in real space via thehaptic interface device425 according to input from thecollision thread410. The force is calculated by using data from theshape collision renderer453 and from theconstraint collision renderer455. Data from these two renderers are used to calculate a local approximation, which is transmitted to thelocal approximation renderer465. Thelocal approximation renderer465 resolves a position/orientation transform for the proxy, which is used for producing a contact or constraint force. The proxy can be represented by the position of a single point, but can alternatively be chosen as having any arbitrary geometry. The local approximation transmitted to thelocal approximation renderer465 is a collection of geometry determined in the collision thread generally at a lower processing rate than the servo thread. This local approximation geometry may be used for several updates of the servo loop thread. The local approximation geometry generally serves as a more efficient representation for collision detection and resolution than the source geometry processed by the collision thread. The proxy position information is transmitted to aproxy shader470 and then to aproxy renderer475, along with the user-definedproxy information445 from the graphics thread.
• In one embodiment, aforce shader480 enables modification of a calculated force vector prior to transmitting the force vector to thehaptic interface device425. For example, rendered proxy data from theproxy renderer475, along with force vector data from theeffect renderer460, are used by theforce shader480 to calculate a modified force vector, which is then transmitted to thehaptic interface device425. Theforce shader480 is thus able to modify the direction and magnitude of the force vector as determined by preceding modules such as theproxy renderer475 and theeffect renderer460. Theforce shader480 may also have access to data from other modules in the schematic diagram427 ofFIG. 4B, such as thelocal approximation renderer465 and theproxy shader470. Theforce shader480 may be used for simulating arbitrary force effects. Examples of such force effects include inertia, viscosity, friction, attraction, repulsion, and buzzing.
• Theforce shader480 may also be used for modifying the feel of a contacted surface. For example, theforce shader480 may be used to simulate a smooth surface by modifying the force vector direction so that it is smoothly varying while contacting discontinuous surface features. As such, force discontinuities apparent when transitioning from one polygonal face to another may be minimized by theforce shader480 by aligning the force vector to an interpolated normal based on adjacent faces. Theforce shader480 may also be used for general conditioning or filtering of the computed force vector, such as clamping the magnitude of the force vector or increasing the magnitude of the force vector over time. In one embodiment, the force shader is used to reduce the magnitude and directional discontinuities over time, which can result from instabilities in the control system or mechanical instabilities in thehaptic interface device425.
• Theservo thread415 may refresh the force to be applied through thehaptic interface device425 at a rate within the range from about 500 Hz to about 15,000 Hz, from about 1000 Hz to about 10,000 Hz, or from about 2000 Hz to about 6000 Hz. Rates above and below these levels are possible as well.
• In one embodiment, a scheduler interface manages the high frequency for sending forces and retrieving state information from thehaptic interface device425. The scheduler allows the 3D graphics application to communicate effectively with the servo thread in a thread-safe manner and may add and delete operations to be performed in the servo thread. Furthermore, in one embodiment, a calibration interface allows the system to maintain an accurate estimate of the physical position of thehaptic interface device425. Calibration procedures may be manual and/or automatic.
• FIG. 5 is a schematic diagram500 illustrating a servo thread of an illustrative haptics rendering pipeline. Collision andconstraint resolution data502 from the virtual environment is transmitted from the collision thread to thelocal approximation renderer465. Thelocal approximation renderer465 calculates a proxy position, which is then transmitted to aproxy shader470 and then toimpedance control515, producing a force. The force is modified by theforce shader480, then transmitted to thehaptic interface device425 following application ofinverse kinematics525.Forward kinematics535 from thehaptic interface device535 is fed back to theforce shader480 and theimpedance controller515, and is transmitted to atransform shader540, which provides feedback to thelocal approximation renderer465 andproxy shader470.
• FIG. 6 is a schematic diagram600 illustrating a system for haptically rendering a virtual environment using data in a graphics pipeline of a 3D graphics application. The diagram600 shows how third-party 3D graphics application software is integrated with the system. The diagram600 illustrates the interaction between the3D graphics application602, ahaptics API610, and ahaptic device API625. Thegraphics application600 can make a function call to thehaptics API610. Thehaptics API610 then accesses data from the 3D graphics pipeline. Thehaptics API610 also transmits data to thehaptic device API625, which performs low-level force rendering.
• FIG. 7 is a block diagram700 featuring a method of delivering interaction force to a user via a haptic interface device, where the force is based at least in part on graphical data from a virtual camera located at a haptic interface location. The method includes determining a haptic interface location in a 3D virtual environment corresponding to the position of a haptic interface device inreal space702. The method further includes positioning a first virtual camera at thehaptic interface location705. The first virtual camera is usually implemented using matrix transformations that map 3D virtual objects in coordinate space into a 2D representation, so that the virtual environment, populated with the virtual objects, appears as if viewed by a camera. By modifying these transformations, the virtual camera view can be changed to view the same object from any of a plurality of vantage points. These transformations include a modeling transformation, a viewing transformation, a projection transformation, and a display device transformation. These are discussed in further detail with respect toFIG. 9 herein below. Furthermore, the position of the first camera is updated as the haptic interface location changes, according to the manipulation of the haptic interface device by the user.
• The method ofFIG. 7 next includes the step of accessing graphical data corresponding to the virtual environment as viewed from the first virtual camera at thehaptic interface location710. The accessed data is then used in the graphical rendering of the virtual environment, for example, according to methods described herein.
• The method ofFIG. 7 may optionally include the step of positioning a second virtual camera at a location other than thehaptic interface location715. The method would then comprise the step of accessing graphical data from the secondvirtual camera720. The accessed data may be used for graphical rendering, haptic rendering, or both. In one embodiment, the second virtual camera is used for graphical rendering, while the first virtual camera is used for haptic rendering. The second camera may move, or it may be static. In one embodiment, the second virtual camera is fixed while the first virtual camera is capable of moving. The second virtual camera operates using matrix transformations as described with respect to step705. The second virtual camera has associated with it a look direction and an eye position, independent of the look direction and eye position of the first virtual camera.
• FIG. 8A is ascreenshot800 of a virtual object (a teapot) in a virtual environment as imaged from a fixed camera view (i.e. the second camera view, as described with respect toFIG. 7). Thescreenshot800 shows ahaptic interface location805, representing the position of a user in the virtual environment. A “haptic camera” (first virtual camera) is located at the haptic interface location, which moves as a user manipulates a haptic interface device in real space.FIG. 8B is ascreenshot810 of the virtual object ofFIG. 8A as imaged from the moving haptic camera. As can be seen from thescreenshot810, additional detail is viewable from this vantage point. It is possible to haptically render the virtual object using the graphical data from the haptic camera. Efficiency is improved by limiting the information that is haptically rendered to only those parts of the virtual environment that can be “touched” by the user at any given time. Furthermore, geometry that is not visible from the second camera view (i.e. dedicated to providing a graphical display of the virtual environment) can be “felt” using graphical data from the haptic camera view. The user can feel behind the displayed teapot.
• The view volume of the haptic camera may be optimized so as to view only areas of the virtual environment the user will want to touch or will be able to touch at any given time. For example, the view volume of the first virtual camera, dedicated to haptic rendering, may be limited to objects within the vicinity and trajectory of the haptic interface. As a result, haptic rendering will only need to be performed for this limited view volume, and not for all the geometry that is viewed from the vantage point of a graphics-dedicated second virtual camera. The method thereby increases the efficiency of the haptic rendering process.
• Additionally, the method ofFIG. 7 comprises determining a position of the haptic interface location in relation to a surface of a virtual object in the virtual environment by using graphical data from either or both of the first virtual camera and the secondvirtual camera725. The method also includes determining an interaction force based at least in part on the position of the haptic interface location in relation to the surface of thevirtual object730. Finally, an interaction force is delivered to a user through thehaptic interface device735. The determination and delivery of an interaction force is described, for example, in U.S. Pat. Nos. 6,191,796, 6,421,048, 6,552,722, 6,417,638, and 6,671,651, the disclosures of which are incorporated by reference herein in their entirety.
• FIG. 9 is a schematic diagram900 illustrating a 3D transformation pipeline. 3D graphics applications generally perform a series of transformations in order to display 3D model coordinates on a 2D display device. These transformations include a shape-world transformation902, a world-view transformation905, a view-clip transformation910, and a clip-window transformation915. Additional transformations that are used to haptically render a virtual environment via a haptic interface device include a view-touch transformation920 and a touch-workspace transformation925. The transformations inFIG. 9 can be repurposed for rendering a scene from a virtual haptic camera viewpoint, thereby affording improved acquisition and utilization of graphics pipeline data.
• The shape-world transformation902 of the pipeline ofFIG. 9 transforms geometry describing a virtual object from its local coordinate space, or shape coordinates, into world coordinates, i.e., the main reference coordinate space for the 3D virtual environment. All objects in the virtual environment have a relationship to world coordinates, including cameras.
• The world-view transformation905 of the pipeline ofFIG. 9 maps world coordinates to view coordinates, the local coordinates of the virtual camera.FIG. 10 illustrates the relation of view coordinates (X_V, Y_V, Z_V), with an associated look direction and camera eye position, to world coordinates (X_W, Y_W, Z_W). The look direction ofFIG. 10 is preferably mapped to the z-axis of the world-view transform. The world-view transformation can be customized for translating and rotating the virtual camera so that it can view the scene as if attached to the position of the haptic device's virtual proxy.
• Furthermore, where the virtual camera is a haptic camera as described above, the camera eye position of the world-view transformation is sampled from the virtual proxy position. In order to avoid undesirable jitter, the camera eye position is preferably only updated when the virtual proxy moves beyond a threshold distance from the current eye position. In one embodiment, for example, the threshold distance is 2 mm.
• The look direction of the world-view transformation is determined by the motion of the proxy and optionally by the contact normal, for example, if the proxy is in contact with a virtual object in the virtual environment. When in contact with a virtual object, the proxy's position can be constrained to remain on the surface of the contacted virtual object.FIG. 11 illustrates thelook direction1110 when the virtual proxy is in contact with avirtual object1101. Additionally, the camera eye position is updated as soon as the proxy has moved beyond a threshold distance. This defines themotion vector1120 of the proxy. When moving in free space, the look direction is the normalizedmotion vector1120. However, when in contact with avirtual object1101, the look direction is a linear combination of the normalizedmotion vector1120 and the contact normal1105, as illustrated inFIG. 11. For example, where the haptic interface location (proxy position) is on the surface of the virtual object, as shown inFIGS. 8A and 8B, the look direction may be computed as a linear combination of the normalized motion vector and the contact normal. Thus, the haptic camera angle tilts to show more of what lies ahead, along the direction of motion.
• The world-view transformation905 ofFIG. 9 can be computed by forming a composite rotation-translation matrix that transforms coordinates from world coordinates into view coordinates, mapping the look direction to an-axis (preferably the z-axis), and mapping the camera eye position to the origin. An up vector, such as the y-axis, may be selected to keep the view consistently oriented.
• Another of the transformations in the 3D transformation pipeline ofFIG. 9 is the view-clip transformation910, also known as the projection transform. The view-clip transformation910 enables manipulations of the shape and size of the view volume. The view volume determines which geometry is lit and rasterized for display on the 2D display device. As a result, geometry that lies outside the view volume is usually excluded from the remainder of the graphics pipeline.
• When data from a virtual haptic camera is used for haptic rendering, the view volume may be sized so as to include only objects that are likely to be touched. In one embodiment, the size of the view volume is specified as a radius of motion in workspace coordinates of the haptic device which is transformed into view coordinates when composing the view-clip matrix. An orthographic view volume mapping centered around the origin is used with extents determined by the motion radius. By limiting the size of the view volume via the view-clip transformation910, it is possible to localize the geometry that is received by the graphic pipeline when haptically rendering the scene, thereby optimizing the haptic rendering process.
• Another of the transformations in the 3D transformation pipeline ofFIG. 9 is the clip-window transformation915, which converts clip coordinates into the physical coordinates of the display device so that an object in clip coordinates may be displayed on the display device. The clip-window transformation915 is specified by a 2D pixel offset and a width and height in pixels. By using the clip-window transformation915, it is possible to limit the amount of pixels used for rasterizing the geometry in the graphics pipeline. For optimal performance, it is not necessary to rasterize the localized contents of the view volume using the entire pixel buffer dimensions. There may be a tradeoff between performance and sampling error. For example, if the pixel buffer is too big, it will require more memory and copying time. However, if the pixel buffer is too small, it is possible that too many details will be lost for adequately realistic haptic rendering. The size of a display device buffer may be determined in consideration of the aforementioned tradeoff. In one embodiment, a width and height of 256 by 256 pixels for the display device buffer provides a sufficient compromise. Optimization of these dimensions is possible by considering the allowable time for pixel buffer read-back from the graphics card and the size of the smallest geometric feature in pixel coordinates.
• The view-touch transformation920 maps an object from view-coordinates into the touch coordinate space. The view-touch transformation920 is convenient for altering the alignment or offset of touch interactions with respect to the view. As a default, this transformation may be left as identity so that the position and alignment of touch interactions are consistent with the view position and direction. However, the view-touch transformation920 may be optionally modified to accommodate touch interactions with the scene in which the haptic device and display device are meant to be independent, for example, during use of a head-mounted display.
• The touch-workspace transformation925 maps an object in touch-coordinates into the local coordinate space of the haptic interface device. The haptic workspace is the physical space reachable by the haptic device. For example, the PHANTOMS Omni™ device, manufactured by SensAble Technologies, Inc., of Woburn, Mass., has a physical workspace ofdimensions 160×120×70 mm.
• The shape-world transformation900, the world-view transformation905, the view-clip transformation910, the clip-window transformation915, the view-touch transformation920, and/or the touch-workspace transformation925 may be structured for viewing a scene of a virtual environment from any of one or more virtual cameras. For example, these transformations may be structured for viewing a scene from a first virtual camera dedicated to haptic rendering, as well as a second virtual camera dedicated to graphical rendering. The processing capability of the graphics pipeline is leveraged for both graphical and haptic rendering.
• FIG. 12 is a block diagram1200 featuring an alternative method for interpreting data for haptic rendering, including the step of intercepting data from a graphics pipeline via a pass-through dynamic link library (DLL). InStep1202, data is intercepted from the graphics pipeline of a 3D graphics application using a pass-through dynamic link library (DLL). A graphics API generally uses a DLL file so that a 3D graphics application may access the functions in its library. A pass-through DLL may be named to match the name of the usual DLL file used by the graphics API, while the “real” graphics API DLL file is renamed. As a result, function calls from the 3D graphics application will call the pass through DLL, instead of calling the graphics API DLL. The pass-through DLL does not impede normal functioning of the 3D graphics application because all function calls are redirected by the pass-through DLL to the regular graphics API DLL.
• In order for the pass-through DLL to intercept data from the 3D graphics pipeline, logic is inserted in its code to respond to particular graphics API function calls. The pass-through DLL may also directly call functions of the graphics API, hence directly accessing the 3D graphics pipeline and the associated buffer data. Creating a pass-through DLL may require replicating the exported function table interface of the graphics API DLL. This may be accomplished by determining the signature of every function exported by the DLL. A binary file dumper can then be used to view the symbols exported by the DLL and access to the header file can be used for determining the number and type of the function arguments and return type.
• Instep1205 of the method ofFIG. 12, a subset of the accessed data is written to a memory buffer and a subset of data is read from this memory buffer. This memory buffer may be shared between the pass through DLL and a separate haptic rendering process.
• Inoptional step1210 of the method ofFIG. 12, a height map is determined using the accessed data. For example, if the depth buffer is accessed instep1200, the depth buffer itself may be treated as a height map. Such a height map may describe at least some of a surface of a virtual object in the virtual environment. Inoptional step1215, a mesh is generated using the height map determined instep1210. However, in a preferred embodiment, the haptic rendering method interprets a height field directly, as described elsewhere herein. Haptic rendering of a depth buffer is performed directly in screen space and in a local fashion (i.e. via a haptic camera). It is not necessary that the entire image be transformed and then processed to generate a mesh. In order to generate a mesh from depth buffer data, the data representing depth values and screen coordinate locations may be transformed from screen space to object space.
• FIG. 13 is a schematic diagram1300 illustrating an alternative system for haptically rendering a virtual environment using data intercepted from a graphics pipeline of a 3D graphics application via a pass-through dynamic link library. In one embodiment, a3D graphics application1300 is developed using a graphics API. When the3D graphics application1300 makes calls to the graphicsAPI DLL file1310, the calls are intercepted by a pass-throughDLL file1305. The pass-through DLL does not impede normal functioning of the 3D graphics application because all function calls are redirected by the pass through DLL to the regular graphics API DLL.
• The pass-throughDLL1305 may then make function calls to thegraphics API DLL1310, thereby accessing buffer data from the 3D graphics pipeline. Thegraphics API DLL1310 operates to render graphics on a display screen via a3D graphics card1315. However, the pass-throughDLL1305 may call the graphics API DLL to access the graphic rendering data from the 3D graphics pipeline and store this data inmemory buffer1320. The data may be read from thememory buffer1320 in a haptic rendering process to provide touch feedback based on the intercepted graphical data.
• Thus, thememory buffer1320 may be shared with ahaptic API1325. For example, thehaptic API1325 accesses the graphic rendering data in thememory buffer1320 and prepares it for low level haptic rendering by thehaptic device API1330. Thehaptic device API1330 then produces a force signal which a device driver uses to generate and transmit a force to a user via thehaptic interface device1335.
EQUIVALENTS
• While the invention has been particularly shown and described with reference to specific preferred embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (36)

1. A method for haptically rendering a virtual object in a virtual environment, the method comprising the steps of:

(a) determining a haptic interface location in a 3D virtual environment corresponding to a location of a haptic interface device in real space;

(b) positioning a first virtual camera substantially at the haptic interface location;

(c) accessing graphical data from the first virtual camera corresponding to the virtual environment;

(d) determining a position of the haptic interface location in relation to at least one geometric feature of a virtual object in the virtual environment using the graphical data from the first virtual camera; and

(e) determining an interaction force based at least in part on the position of the haptic interface location in relation to the at least one geometric feature of the virtual object.

2. The method ofclaim 1, wherein the at least one graphical feature of the virtual object comprises at least one of a surface, point, line, or plane.

3. The method ofclaim 1, further comprising the step of delivering the interaction force to a user through the haptic interface device.

4. The method ofclaim 1, wherein the position of the first camera is updated as the haptic interface location changes according to movement of the haptic interface device.

5. The method ofclaim 1, further comprising the steps of:

(f) positioning a second virtual camera at a location other than the haptic interface location; and

(g) accessing graphical data from the second virtual camera corresponding to the virtual environment.

6. The method ofclaim 5, wherein the second virtual camera is fixed and the first virtual camera moves.

7. The method ofclaim 1, wherein step (c) comprises accessing graphical data from a graphics pipeline of a 3D graphics application.

8. The method ofclaim 1, wherein step (d) comprises determining a world-view transform that maps world coordinates corresponding to the haptic virtual environment to view coordinates corresponding to the first virtual camera.

9. The method ofclaim 8, wherein the step of determining the world-view transform comprises determining an eye position and a look direction.

10. The method ofclaim 9, wherein determining the eye position comprises sampling the position of the haptic interface location and wherein determining the look direction comprises determining a vector representing a motion of the haptic interface location.

11. The method ofclaim 8, wherein step (d) further comprises at least one of the following:

determining a shape-world transformation;

determining a world-view transformation;

determining a view-clip transformation;

determining a clip-window transformation;

determining a view-touch transformation; and

determining a touch-workspace transformation.

12. The method ofclaim 1, wherein a view volume associated with the first virtual camera is sized to exclude geometric elements that lie beyond a desired distance from the haptic interface location.

13. The method ofclaim 1, further comprising the step of culling at least a portion of the graphical data from the first virtual camera to exclude data corresponding to geometric primitives that lie outside a view volume associated with the first camera.

14. The method ofclaim 13, wherein the culling step is performed using graphics hardware.

15. The method ofclaim 13, wherein the culling step is performed using a spatial partition.

16. The method ofclaim 15, wherein the spatial partition comprises a hierarchical data structure.

17. The method ofclaim 16, wherein the hierarchical data structure comprises at least one of an octree data structure and a BSP tree data structure.

18. The method ofclaim 13, wherein the culling step is performed using graphics hardware and a spatial partition.

19. The method ofclaim 1, wherein the graphical data from the first virtual camera comprises at least one of the following:

at least a portion of a depth buffer;

at least a portion of a feedback buffer;

at least a portion of a color buffer;

at least a portion of a selection buffer;

at least a portion of an accumulation buffer;

at least a portion of a texture map;

at least a portion of a fat framebuffer;

data from a pixel shading program;

data from a vertex shading program

rasterization primitives;

application programming interface input data; and

state data.

20. The method ofclaim 1, wherein the graphical data from the first virtual camera comprises at least a portion of a fat framebuffer.

21. The method ofclaim 20, wherein the fat framebuffer comprises at least one member of the group consisting of: vertex positions; normals; color; texture; normal maps; bump maps; and depth data.

22. The method ofclaim 20, wherein step (c) comprises performing at least one of a pixel shading and a vertex shading.

23. The method ofclaim 22, wherein the shading is performed using graphics hardware.

24. The method ofclaim 1, wherein the graphical data from the first virtual camera comprises at least a portion of a depth buffer.

25. The method ofclaim 1, wherein the graphical data from the first virtual camera comprises rasterization primitives.

26. The method ofclaim 1, wherein step (d) comprises:

performing an intersection test to determine at least one intersection point and at least one intersection normal in screen space; and

transforming coordinates of the at least one intersection point and the at least one intersection normal from screen space to object space.

27. The method ofclaim 26, wherein step (d) comprises defining for each intersection point a plane tangent to the surface of the virtual object at the intersection point.

28. The method ofclaim 26, wherein step (d) comprises performing a projection test to determine a geometric feature nearest the haptic interface location.

29. A system for haptically rendering a virtual object in a virtual environment, the system comprising:

a graphics thread that generates a visual display of a virtual environment;

a collision thread that determines if a user-directed virtual proxy collides with at least one geometric feature within the virtual environment, wherein the collision thread uses input from the graphics thread; and

a servo thread that generates force to be applied to a user in real space through a haptic interface device according to input from the collision thread, wherein the servo thread is in communication with the haptic interface device.

30. The system ofclaim 29, wherein the graphics thread refreshes the visual display at a rate within a range from about 5 Hz to about 150 Hz.

31. The system ofclaim 29, wherein the graphics thread refreshes the visual display at a rate within a range from about 30 Hz to about 60 Hz.

32. The system ofclaim 29, wherein the collision thread performs a collision detection computation at a rate within a range from about 30 Hz to about 200 Hz.

33. The system ofclaim 29, wherein the collision thread performs a collision detection computation at a rate within a range from about 80 Hz to about 120 Hz.

34. The system ofclaim 29, wherein the servo thread refreshes the force to be applied through the haptic interface device at a rate within a range from about 1000 Hz to about 10,000 Hz.

35. The system ofclaim 29, wherein the servo thread comprises at least one of a force shader and a proxy shader.

36. An apparatus for providing haptic feedback to a user of a 3D graphics application, the apparatus comprising:

a user-controlled haptic interface device adapted to provide a user input to a computer and to transmit force to a user; and

computer software that, when operating with the computer and the user input, is adapted to determine force transmitted to the user by:

(a) determining a haptic interface location in a 3D virtual environment corresponding to a location of the haptic interface device in real space;

(b) positioning a first virtual camera substantially at the haptic interface location;

(c) accessing graphical data from the first virtual camera corresponding to the virtual environment;

(d) determining a position of the haptic interface location in relation to at least one geometric feature of a virtual object in the virtual environment using the graphical data from the first virtual camera; and

(e) determining an interaction force based at least in part on the position of the haptic interface location in relation to the at least one geometric feature of the virtual object.

Images