US20190015991A1 - Simultaneous Kinematic and Hand-Eye Calibration - Google Patents

Abstract

Described are machine vision systems and methods for simultaneous kinematic and hand-eye calibration. A machine vision system includes a robot or motion stage and a camera in communication with a control system. The control system is configured to move the robot or motion stage to poses, and for each pose: capture an image of calibration target features and robot joint angles or motion stage encoder counts. The control system is configured to obtain initial values for robot or motion stage calibration parameters, and determine initial values for hand-eye calibration parameters based on the initial values for the robot or motion stage calibration parameters, the image, and joint angles or encoder counts. The control system is configured to determine final values for the hand-eye calibration parameters and robot or motion stage calibration parameters by refining the hand-eye calibration parameters and robot or motion stage calibration parameters to minimize a cost function.

Description

RELATED APPLICATIONS
• This application is a continuation of U.S. patent application Ser. No. 15/278,156, filed on Sep. 28, 2016, the entirety of which is incorporated herein by reference.
FIELD OF THE TECHNOLOGY
• The present technology relates generally to machine vision systems and, more specifically, to calibrating machine vision systems by simultaneously refining kinematic chain of robots/motion stages and hand-eye calibration parameters.
BACKGROUND
• In machine vision systems, one or more cameras can be used to perform vision system processes on an object or surface within an imaged scene. These processes can include inspection, alignment, and/or a variety of other tasks. The images captured by the one or more cameras can be processed by the machine vision system to determine information about features within the imaged scene, such as features' real-world locations, poses, etc. In some applications, a machine vision system can include a robot, motion stage, or other device whose pose and movement is controlled by the machine vision system (e.g., Vision Guided Robot (VGR) applications). For example, the machine vision system can analyze the images captured by the one or more cameras to determine how to move the robot or motion stage to move or interact with objects in the imaged scene. In some applications, the machine vision system can be calibrated to allow the machine vision system to perform vision tasks with increased accuracy and reliability.
SUMMARY
• The technology described herein can improve the accuracy and precision of existing machine vision systems by improving the kinematics modelling of the robot/motion stage and the calibration of the machine vision system.
• In one aspect, there is a machine vision system. The machine vision system includes a robot. The machine vision system includes a camera. The machine vision system includes a control system in communication with the robot and the camera. The control system is configured to move the robot to a plurality of poses, and for each pose of the plurality of poses: capture, with the camera, an image of one or more features on a calibration target and capture a plurality of robot joint angles. The control system is configured to obtain a plurality of initial values for a plurality of robot calibration parameters. The control system is configured to determine a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of robot calibration parameters, and the image and the plurality of robot joint angles captured for each of the plurality of poses. The control system is configured to determine a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the robot calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of robot calibration parameters to minimize a cost function.
• In some embodiments, the control system is further configured to use the plurality of final values for the plurality of hand-eye calibration parameters and the plurality of final values for the robot calibration parameters for a vision guided robot application. In some embodiments, the control system is further configured to move the robot using the plurality of final values for the robot calibration parameters.
• In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of robot joint angles captured for the pose. In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between a composite transformation and an identity transformation, the composite transformation being a composition of a first transformation from a first coordinate space to a second coordinate space and a second transformation from the second coordinate space to the first coordinate space, the first transformation and the second transformation determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, the plurality of observed features of the calibration target for the pose, and the plurality of robot joint angles captured for the pose. In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between a robot joint angle of the plurality of robot joint angles captured for the pose and an estimated joint angle for the pose, the estimated joint angle determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.
• In some embodiments, the control system is further configured to minimize the cost function using non-linear least squares. In some embodiments, the control system is configured to refine less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of robot calibration parameters. In some embodiments, the plurality of robot calibration parameters includes Denavit-Hartenberg parameters for the robot.
• In another aspect, there is a machine vision system-implemented calibration method. The method includes moving, by a control system, a robot to a plurality of poses, and for each pose of the plurality of poses: capturing, by a camera, an image of one or more features on a calibration target and capturing, by the control system, a plurality of robot joint angles. The method includes obtaining, by the control system, a plurality of initial values for a plurality of robot calibration parameters. The method includes determining, by the control system, a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of robot calibration parameters, and the image and the plurality of robot joint angles captured for each of the plurality of poses. The method includes determining, by the control system, a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the plurality of robot calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of robot calibration parameters to minimize a cost function.
• In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of robot joint angles captured for the pose. In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between a composite transformation and an identity transformation, the composite transformation being a composition of a first transformation from a first coordinate space to a second coordinate space and a second transformation from the second coordinate space to the first coordinate space, the first transformation and the second transformation determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, the plurality of observed features of the calibration target for the pose, and the plurality of robot joint angles captured for the pose. In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between a robot joint angle of the plurality of robot joint angles captured for the pose and an estimated joint angle for the pose, the estimated joint angle determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.
• In some embodiments, the method includes minimizing the cost function using non-linear least squares. In some embodiments, the method includes refining less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of robot calibration parameters. In some embodiments, the plurality of robot calibration parameters includes Denavit-Hartenberg parameters for the robot.
• In another aspect, there is a machine vision system. The machine vision system includes a motion stage. The machine vision system includes a camera. The machine vision system includes a control system in communication with the motion stage and the camera. The control system is configured to move the motion stage to a plurality of poses, and for each pose of the plurality of poses: capture, with the camera, an image of one or more features on a calibration target and capture a plurality of encoder counts for a plurality of actuators of the motion stage. The control system is configured to obtain a plurality of initial values for a plurality of motion stage calibration parameters. The control system is configured to determine a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of motion stage calibration parameters, and the image and the plurality of encoder counts captured for each of the plurality of poses. The control system is configured to determine a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the motion stage calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of motion stage calibration parameters to minimize a cost function.
• In some embodiments, the control system is further configured to use the plurality of final values for the plurality of hand-eye calibration parameters and the plurality of final values for the motion stage calibration parameters for a vision guided motion stage application. In some embodiments, the control system is further configured to move the motion stage using the plurality of final values for the motion stage calibration parameters.
• In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of motion stage calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of encoder counts captured for the pose. In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between an encoder count of the plurality of encoder counts captured for the pose and an estimated encoder count for the pose, the estimated encoder count determined based on a first plurality of values for the plurality of motion stage calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.
• In some embodiments, the control system is further configured to minimize the cost function using non-linear least squares. In some embodiments, the control system is configured to refine less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of motion stage calibration parameters. In some embodiments, the plurality of motion stage calibration parameters includes one or more of roller center position, roller radius, direction, step size, home encoder count, and striker plate's direction for an actuator.
• In another aspect, there is a machine vision system-implemented calibration method. The method includes moving, by a control system, a motion stage to a plurality of poses, and for each pose of the plurality of poses: capture, by a camera, an image of one or more features on a calibration target; and capturing, by the control system, a plurality of encoder counts for a plurality of actuators of the motion stage. The method includes obtaining, by the control system, a plurality of initial values for a plurality of motion stage calibration parameters. The method includes determining, by the control system, a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of motion stage calibration parameters, and the image and the plurality of encoder counts captured for each of the plurality of poses. The method includes determining, by the control system, a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the motion stage calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of motion stage calibration parameters to minimize a cost function.
• In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of motion stage calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of encoder counts captured for the pose. In some embodiments, the cost function measures, for each pose of the plurality of poses, a difference between an encoder count of the plurality of encoder counts captured for the pose and an estimated encoder count for the pose, the estimated encoder count determined based on a first plurality of values for the plurality of motion stage calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.
• In some embodiments, the method includes minimizing the cost function using non-linear least squares. In some embodiments, the method includes refining less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of motion stage calibration parameters. In some embodiments, the plurality of motion stage calibration parameters includes one or more of roller center position, roller radius, direction, step size, home encoder count, and striker plate's direction for an actuator.
• Other aspects and advantages of the technology will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating the principles of the technology by way of example only.
BRIEF DESCRIPTION OF THE DRAWINGS
• The foregoing and other objects, features, and advantages of the present technology, as well as the technology itself, will be more fully understood from the following description of various embodiments, when read together with the accompanying drawings, in which:
• FIG. 1 illustrates an exemplary machine vision system.
• FIG. 2 illustrates an exemplary machine vision system.
• FIG. 3 depicts a calibration scheme in accordance with the technology.
• FIG. 4 depicts a flowchart of a method of calibrating a machine vision system in accordance with the technology.
• FIG. 5 depicts a calibration scheme in accordance with the technology.
DETAILED DESCRIPTION
• Precisely calibrating machine vision systems for VGR applications can be critical. Existing approaches to calibrating such machine vision systems (e.g., machine vision systems with a robot, motion stage, or other device) can result in machine vision systems that do not have the requisite accuracy for many applications. For example, in many cases, calibrating the machine vision system based on the kinematic calibration parameter values (e.g., robot calibration parameter values or motion stage calibration parameter values) provided by robot or motion stage manufacturers can lead to inadequate accuracy of the machine vision system. The present technology can improve the accuracy and precision of machine vision systems for VGR applications by using efficient calibration techniques to refine kinematic calibration parameters and hand-eye calibration parameters. The technology can be used to determine more accurate calibration parameter values, including the hand-eye calibration parameters and kinematic calibration parameters, in a single refinement calculation based on measurements determined by the machine vision system. Beneficially, the technology can increase the accuracy of the machine vision systems through improved calibration without, for example, requiring additional calibration targets, additional cameras, or external measurement equipment.
• I. Machine Vision Systems
• FIG. 1 illustrates exemplarymachine vision system100.Machine vision system100 can be used, for example, for VGR applications.Machine vision system100 includesrobot105 with acamera110 mounted thereon, andcontrol system115.FIG. 1 illustrates a moving camera configuration (e.g.,camera110 is mounted on the arm of robot105). Still referring toFIG. 1,calibration target120 havingfeatures123 is shown.Camera110 can be an area-scan camera including a two-dimensional CCD camera sensor, a two-dimensional CMOS camera sensor or any other type of camera sensor for generating images.Control system115 can be any processing device (e.g., embedded machine vision processor, computing device, etc.).
• Control system115 is in communication withrobot105 andcamera110.Control system115 can receive image data fromcamera110 and control the pose ofrobot105. In some embodiments,Control system115 can be a part of a standalone computing device. In some embodiments,control system115 can be a part ofrobot105. In some embodiments,control system115 can be a part ofcamera110. In operation,control system115 can manipulate the pose ofrobot105, e.g., based on analysis of image data fromcamera110. For simplicity,control system115 is illustrated as a single component. In some embodiments, the control system can include a motion controller (e.g., a robot controller) and a machine vision processor working in concert. In some embodiments, the motion controller can be the machine vision system master. For example, the motion controller can communicate with and direct the actions of the machine vision processor. The motion controller can control the pose and movement of the robot and cause the machine vision processor to acquire images. The motion controller can provide pose information to the machine vision processor and cause the machine vision processor to perform the calibration described herein. In some embodiments, the machine vision processor can be the machine vision system master. For example, the machine vision processor can communicate with and direct the actions of the motion controller. The machine vision processor can cause the motion controller to control the pose and movement of the robot, and provide pose information to the machine vision processor. The machine vision processor can acquire images and perform the calibration described herein. In some embodiments, the motion controller and machine vision controller can be integrated. For example, one or more processor cores of a control system can control robot motion and one or more processor cores of the control system can perform machine vision processing such as calibration.
• In accordance with the technology, themachine vision system100 can be calibrated. As will be described in greater detail below,system100 can be calibrated to permit transformation of coordinates between the image coordinate space ofcamera110, the coordinate spaces of therobot105, and the coordinate space ofcalibration target120. Using the calibration,control system115 can directrobot hand160 to precise locations in the coordinate space ofcalibration target120 or in the robot base coordinate space.Control system115 can further determine the location of objects observed in image data fromcamera110 by using the calibration and information about the angles of the joints ofrobot105. Beneficially, the calibration permits machine vision system to control the motion ofrobot hand160 and use it to interact with and/or manipulate real-world objects (e.g., in the coordinate space of calibration target120).
• FIG. 2 illustrates exemplarymachine vision system200.Machine vision system200 can be used, for example, for VGR applications.Machine vision system200 includesmotion stage205,camera210, andcontrol system215. As illustrated,camera210 can be mounted abovemotion stage205 to image a top surface ofmotion stage205. In the illustrated embodiment,camera210 is mounted in a fixed way relative to base208 of motion stage205 (e.g., table209 ofmotion stage205 can move relative to camera210).Camera210 can be an area-scan camera including a two-dimensional CCD camera sensor, a two-dimensional CMOS camera sensor or any other type of camera sensor for generating images.Control system215 can be any processing device (e.g., embedded machine vision processor, computing device, etc.). Still referring toFIG. 2,calibration target220 havingfeatures223 is shown.
• Control system215 is in communication withmotion stage205 andcamera210. Control system can receive image data fromcamera210 and control the pose ofmotion stage205. In some embodiments,control system215 can be a part of a standalone computing device. In some embodiments,control system215 can be a part ofmotion stage205. In some embodiments,control system215 can be a part ofcamera210. In operation,control system215 can change the pose ofmotion stage205, e.g., based on analysis of image data fromcamera210. For simplicity,control system215 is illustrated as a single component. In some embodiments, the control system can include a motion controller (e.g., a motion stage controller) and a machine vision processor working in concert. In some embodiments, the motion controller can be the machine vision system master. For example, the motion controller can communicate with and direct the actions of the machine vision processor. The motion controller can control the pose and movement of the motion stage and cause the machine vision processor to acquire images. The motion controller can provide pose information to the machine vision processor and cause the machine vision processor to perform the calibration described herein. In some embodiments, the machine vision processor can be the machine vision system master. For example, the machine vision processor can communicate with and direct the actions of the motion controller. The machine vision processor can cause the motion controller to control the pose and movement of the motion stage, and provide pose information to the machine vision processor. The machine vision processor can acquire images and perform the calibration described herein. In some embodiments, the motion controller and machine vision controller can be integrated. For example, one or more processor cores of a control system can control motion stage motion and one or more processor cores of the control system can perform machine vision processing such as calibration.
• In accordance with the technology,machine vision system200 can be calibrated. As will be described in greater detail below,system200 can be calibrated to permit transformation of coordinates between the image coordinate space ofcamera210, the coordinate spaces ofmotion stage205, and the coordinate space ofcalibration target220. Using the calibration,control system215 can translate and/or rotatemotion stage205 to precise poses in the coordinate space ofcalibration target220 or in the coordinate space ofmotion stage205.Control system215 can further determine the location of objects observed in image data fromcamera210 by using the calibration and information about the actuators ofmotion stage105. Beneficially, the calibration permitsmachine vision system200 to control the motion ofmotion stage205 and use it to move real-world objects on its surface (e.g., in the coordinate space of calibration target220).
• It should be appreciated that the illustrated embodiments are merely exemplary machine vision system configurations. In some embodiments, the machine vision system can include other kinds of robots, other kinds of motion stages, or other devices whose movement can be controlled by the machine vision system. In some embodiments, the camera can be mounted to the robot or motion stage, or a fixture independent of the robot or motion stage (e.g., stationary camera scenarios and/or moving camera scenarios). In some embodiments, more than one camera can be used. In some embodiments, the robot or motion stage can interact with multiple workspaces.
• II. Calibration
• The technology can facilitate improved accuracy of machine vision systems by providing an efficient technique for refining calibration parameter values. Generally, calibration parameters can include hand-eye calibration parameters, as are well known in the art. In the case of machine vision systems with robots, calibration parameters can include robot calibration parameters. In the case of machine vision systems with motion stages, calibration parameters can include motion stage calibration parameters. Robot calibration parameters and/or motion stage calibration parameters can be referred to as kinematic calibration parameters. For robots, robot calibration parameters can include the Denavit-Hartenberg parameters (D-H parameters) used for describing, e.g., the links and joints of robots. As another example, UVW motion stages are motion stages that can, for example, include two actuators moving in the X direction, and one actuator moving in Y direction. For UVW motion stages, motion stage calibration parameters can include, for each actuator, the roller's center position, the roller's radius, the actuator's direction, the actuator's step size, the actuator's home encoder count, and the striker plate's direction. As still another example, XY Theta motion stages are motion stages that can, for example, include an actuator for linear translation along the X axis, an actuator for linear translation along the Y axis, and a theta actuator providing rotation. For XY Theta motion stages, motion stage calibration parameters can include, direction and step size for each of the X and Y actuators, and the roller's center position and radius, the actuator's direction and step size, the striker plate's direction for the theta actuator. More generally, the technology can be used for calibration of motion devices that can be modeled geometrically. For example, the technology can be used for calibration of motion devices whose geometry can be represented with calibration parameter values.
• A. Calibration for Machine Vision Systems with Robots
• The technology can refine calibration parameter values by minimizing the inaccuracies caused by robots' variance from their specifications and other conditions. In an exemplary embodiment of the technology, a robot (e.g., robot105) is moved to a plurality of poses, and for each pose the machine vision system captures an image (e.g., via camera110) of a calibration target (e.g., calibration target120) and the robot's joint angles at that pose. Initial calibration parameter values can be determined. For example, initial robot calibration parameter values can be taken from the robot's specification. Initial hand-eye calibration parameter values can be determined using the captured images and the initial robot calibration parameter values. Calibration functionality for determining the initial hand-eye calibration parameter values is commercially available, such as in the Cognex Vision Library® produced by Cognex Corporation. One or more initial calibration parameter values, including one or more robot calibration parameter values and one or more hand-eye calibration parameters, can then be refined to determine final calibration parameter values. For example, by using a nonlinear optimization method to minimize the overall discrepancy between the robot motion, which can be determined using the robot calibration parameters and the joint angles, and the motion observed by the machine vision system. The final calibration parameter values can then be used by the machine vision system to operate the robot during runtime. For example, the final robot calibration parameter values can be used to control the robot more accurately. The final hand-eye calibration parameter values can further improve the performance of the machine vision system at run time.
• FIG. 3 depictscalibration scheme300 in accordance with the technology. In some embodiments, the calibration scheme can be used for moving camera configurations. For example,calibration scheme300 can be used formachine vision system100 ofFIG. 1. InFIG. 3, the ovals represent coordinate spaces.FIG. 3 includes CalPlate3D coordinatespace305, RobotBase3D coordinatespace310, Joint1 coordinatespace315, Joint2 coordinatespace320, Joint3 coordinatespace325, Joint4 coordinatespace330, Joint5 coordinatespace335, Joint6 coordinatespace340, Hand3D coordinatespace345, Camera3D coordinatespace352, and Raw2D coordinatespace355. Arrows between ovals represent transformations between the illustrated coordinate spaces. The calibration parameter values used for the transformations can be determined and refined, as described with respect to the illustrative calibration methods below.
• In the illustrated example, CalPlate3D coordinatespace305 can be the coordinate space ofcalibration target120, and can be defined by the top surface ofcalibration target120. RobotBase3D coordinatespace310 can be the coordinate space ofbase125 ofrobot105. In some embodiments, RobotBase3D coordinatespace310 can be defined by the robot manufacturer. Joint1 coordinatespace315 can be the coordinate space ofjoint130 ofrobot105. Joint2 coordinatespace320 can be the coordinate space ofjoint135 ofrobot105. Joint3 coordinatespace325 can be the coordinate space ofjoint140 ofrobot105. Joint4 coordinatespace330 can be the coordinate space ofjoint145 ofrobot105. Joint5 coordinatespace335 can be the coordinate space ofjoint150 ofrobot105. Joint6 coordinatespace340 can be the coordinate space ofjoint155 ofrobot105. Hand3D coordinatespace345 can be the coordinate space ofhand160 ofrobot105. In some embodiments, Hand3D coordinatespace345 can be defined by the robot manufacturer. In some embodiments, Hand3D coordinatespace345 is a user defined offset from J6. Camera3D coordinatespace352 can be the coordinate space ofcamera110. The origin of Camera3D coordinatespace352 can be at the center of projection with the z axis pointing out along the optical axis of the lens ofcamera110. The x and y axes can be aligned with the rows and columns, respectively, of the camera sensor ofcamera110. Raw2D coordinatespace355 can be a 2D pixel-based coordinate system defined by the rows and columns of the camera sensor ofcamera110. The x axis can point from left to right along the image rows, and the y axis can point down along the columns of the camera sensor ofcamera110. It should be appreciated that coordinate spaces with names containing “3D” are three-dimensional spaces, and coordinate spaces with names containing “2D” are two-dimensional spaces.
• The transformations between the coordinate spaces can be rigid transformations (e.g., comprising a translation component and a rotation component) and/or transformations based on the robot calibration parameters ofrobot105. For example, the transformations between CalPlate3D coordinatespace305 and RobotBase3D coordinatespace310, between RobotBase3D coordinatespace310 and Joint1 coordinatespace315, between Hand3D coordinatespace345 and Camera3D coordinatespace352, and between CalPlate3D coordinatespace305 and Camera3D coordinatespace352, can be 3D rigid transformations. The transformations between Joint1 coordinatespace315 and Joint2 coordinatespace320, between Joint2 coordinatespace320 and Joint3 coordinatespace325, between Joint3 coordinatespace325 and Joint4 coordinatespace330, between Joint4 coordinatespace330 and Joint5 coordinatespace335, and between Joint5 coordinatespace335 and Joint6 coordinatespace340 can be based on the robot calibration parameter values describing the joints and links ofrobot105, and the corresponding joint angle values. In some embodiments, the transformations can be determined using a Denavit-Hartenberg matrix, where the robot parameters are d, θ, a, and αlpha, as is well known in the art. The transformation from Joint6 coordinatespace340 to Hand3D coordinatespace345 can be a rigid transformation along with a variable rotational transform based on the angle value of Joint6 ofrobot105. The transformation from CalPlate3D coordinatespace305 to Camera3D coordinatespace352, and the transformation from Camera3D coordinatespace352 to Raw2D coordinatespace355 can be based on a conventional camera calibration, as is well known in the art. As used herein, the notation CoordinateSpace1fromCoordinateSpace2 indicates the transformation from CoordinateSpace2 to CoordinateSpace1. For example, Joint1FromJoint2 denotes the transformation from Joint2 coordinatespace320 to Joint1 coordinatespace315.
• The present technology can be used to refine the calibration parameter values used to form the transformations described above.
• 1. Capturing Calibration Data
• FIG. 4 depicts flowchart400 of a method of calibrating a machine vision system in accordance with the technology. The depicted method can be performed bycontrol system115. Atstep405,control system115 captures images and robot joint angles when therobot105 is in a plurality of poses. At each pose,control system115 causescamera110 to capture an image of at least a portion ofcalibration target120 that contains one or more of features122.Control system115 further captures the angle of each joint ofrobot105 at the pose. For example,robot105 can report the angle of each joint to controlsystem115.
• 2. Determining Initial Calibration Parameter Values
• Atstep410,control system115 determines initial calibration parameter values. The initial calibration parameter values can be the values used to determine the transformations discussed with reference toFIG. 3 prior to refinement. As will be described in greater detail below, the values of these calibrations parameters can be refined in accordance with the present technology. In the illustrated example ofFIG. 3,control system115 can obtain the initial robot calibration parameter values forrobot105 based on the specification ofrobot105. For example, a robot manufacturer will typically provide these values. Using these initial calibration values and joint angles forrobot105 at a particular pose,control system115 can determine the following transformations forrobot105 at that pose: RobotBase3DFromJoint1, Joint1FromJoint2, Joint2FromJoint3, Joint3FromJoint4, Joint4FromJoint5, Joint5FromJoint6, and Joint6FromHand3D. For example, Joint1FromJoint2, Joint2FromJoint3, Joint3FromJoint4, Joint4FromJoint5, Joint5FromJoint6 can be determined using the Denavit-Hartenberg matrix based on the D-H values provided by the robot specification and the joint angle values ofrobot105 at the pose. RobotBase3DFromJoint1 can be determined based on the robot specification (e.g., the dimensions of robot105). Joint6FromHand3D can be determined based on the robot specification ofHand3D160 ofrobot105 and the joint angle value of Joint6. Joint6FromHand3D can provide the rigid transformation ofrobot hand160 in J6340 (e.g., the Joint 6 coordinate system).Control system115 can form the composite transformation RobotBase3DFromHand3D for a particular pose ofrobot105 based on these transformations. For example, the equation below provides RobotBase3DFromHand3D:
• 
  RobotBase3DFromHand3D=RobotBase3DFromJoint1*Joint1FromJoint2*Joint2FromJoint3*Joint3FromJoint4*Joint4FromJoint5*Joint5FromJoint6*Joint6FromHand3D
• Control system115 can perform a hand-eye calibration to determine initial hand-eye calibration parameter values. For example, control system can use the Cognex Vision Library to determine initial hand-eye calibration parameter values based on the captured images and the RobotBase3DFromHand3D transformation for a plurality of poses, along with the Camera3DFromCalPlate3D transformation. As a result of the hand-eye calibration,control system115 can determine Camera3DFromHand3D and RobotBase3DFromCalPlate3D. With the determination of Camera3DFromHand3D and RobotBase3DFromCalPlate3D, along with the previously determined transformations, coordinates in any coordinate space ofFIG. 3 can be transformed into coordinates in any other coordinate space ofFIG. 3.
• 3. Refining Calibration Parameters
• Atstep415,control system115 refines a plurality of the calibration parameters to determine more accurate values for those calibration parameters. To refine the calibration parameters,control system115 determines the calibration parameter values that minimize a cost function. Techniques for minimizing the cost function and determining final calibration parameter values can include non-linear least squares. One of skill in the art will appreciate that other techniques can be used.
• Embodiments of the technology can use a variety of cost functions. In some embodiments, the cost function can be the sum of the distances between observed locations of features (e.g., features on calibration target120) and the expected locations of those features based on the transformations discussed. In general, this can involve the machine vision system observing the location of a feature in a first coordinate space (e.g., observing with camera110) and applying a series of transformations to map that feature through a plurality of other coordinate spaces to the first coordinate space to calculate the expected location of that feature based on the calibration parameters. If the calibration parameters are completely accurate, the observed location and the expected location will be the same. In practice, there are typically some differences between the observed location and the expected location. In some embodiments, the cost function measures these differences for multiple features, multiple poses, and multiple images. For example,control system115 can determine the location of a plurality of observed calibration target features in Raw2D coordinatespace355 for each robot pose (ObservedFeatureRaw2D_ij).Control system115 can also determine the location of a plurality of mapped calibration target features in Raw2D coordinatespace355 for each robot pose
• 
  MappedFeatureRaw2D_ij=Raw2DFromCamera3D*CalPlate3DFromCamera3DAtPose_iEstimated⁻¹*CalibrationTargetFeature_j
• 
  where
• 
  CalPlate3DFromCamera3DAtPose_iEstimated=CalPlate3DFromRobotBase3D*RobotBase3DFromHand3DAtPose_i*Hand3DFromCamera3D
• • RobotBase3DFromHand3DAtPose_iis the estimated robot pose at pose i using the joint angle values at pose i and the initial calibration parameters values (or successively refined calibration parameter values).
    Control system115 can then compute the distance between the observed feature location and the mapped feature location. The cost function can be defined as follows:
• $\sum _{\begin{array}{c}\mathrm{For}\mathrm{each}\mathrm{camera},\mathrm{each}\mathrm{pose}i,\\ \mathrm{and}\mathrm{each}\mathrm{feature}j\end{array}}^{}{\left(\begin{array}{c}\mathrm{distance}(\mathrm{ObservedFeatureRaw}2D_{\mathrm{ij}},\\ \mathrm{MappedFeatureRaw}2D_{\mathrm{ij}})\end{array}\right)}^2$
• In some embodiments, the cost function measures the changes of the location of the calibration plate features after mapping using the transforms. For example,control system115 can determine the location of a plurality of features in CalPlate3D coordinatespace305.Control system115 can also map each feature location for the plurality features at each robot pose i of the plurality poses through the transformation CalPlate3DFromCalPlate3DAtPose_i, where:
• 
  CalPlate3DFromCalPlate3DAtPose_i=CalPlate3DFromRobotBase3D*RobotBase3DFromHand3DAtPose_i*Hand3DFromCamera3D*Camera3DFromCalPlate3DAtPose_i
• • Where RobotBase3DFromHand3DAtPose_iis the estimated robot pose at pose i using the joint angle values at pose i and the initial calibration parameters values (or successively refined calibration parameter values).
  • Camera3DFromCalPlate3DAtPose_iis obtained by using the observed features in Raw2D and camera calibration information.
    Control system115 can then compute the distance between the feature location and the mapped feature location. The cost function can be defined as follows:
• $\sum _{\begin{array}{c}\mathrm{For}\mathrm{each}\mathrm{camera},\mathrm{each}\mathrm{pose}i,\\ \mathrm{and}\mathrm{each}\mathrm{feature}j\end{array}}^{}\hspace{1em}({\mathrm{distance}\left(\left(\begin{array}{c}\mathrm{CalPlate}3\mathrm{DFrom}-\\ \mathrm{CalPlate}3{\mathrm{DAtPose}}_i*\\ \mathrm{featureCalPlate}3D_j,\\ \mathrm{featureCalPlate}3D_j\end{array}\right)\right)}^2$
• In some embodiments, the cost function can be defined without mapping feature points. If the calibration parameters are completely accurate, then CalPlate3DFromCalPlate3DAtPose_iwould be an identity transformation. Thus, the cost function can measure the difference between CalPlate3DFromCalPlate3DAtPose_iand the identity transformation. In some embodiments, the cost function can be defined as follows:
• $\sum _{\mathrm{For}\mathrm{each}\mathrm{camera},\mathrm{each}\mathrm{pose}i,}^{}{\left(\begin{array}{c}\mathrm{CalPlate}3\mathrm{DfromCalPlate}3{\mathrm{DAtPose}}_i·\\ \mathrm{translation}·\mathrm{length}+w*\\ \mathrm{absolute}\mathrm{value}\\ (\mathrm{CalPlate}3\mathrm{DfromCalPlate}3{\mathrm{DAtPose}}_i·\\ \mathrm{rotation}\mathrm{Angle})\end{array}\right)}^2$
• • Where:
  • CalPlate3DfromCalPlate3DAtPose_i.translation.length is the length of the translation component of the transformation CalPlate3DfromCalPlate3DAtPose_i;
  • CalPlate3DfromCalPlate3DAtPose_i.rotationAngle is the rotation amount of the transformation CalPlate3DfromCalPlate3DAtPose_i(e.g., the rotation can be represented as a rotation amount around a rotation axis); and
  • w is a weight (e.g., scale factor) for balancing the translation and rotation components.
• In some embodiments, the origin of Hand3D coordinatespace345 can be used as the basis for the cost function. A transformation Hand3DFromHand3DAtPose_ican be defined as follows:
• 
  Hand3DFromHand3DAtPose_i=Hand3DFromCamera3D*Camera3DFromCalPlate3DAtPose_i*CalPlate3DFromRobotBase3D*RobotBase3DFromHand3DAtPose_i
• 
  Where:
• 
  Hand3DFromCamera3D*Camera3DFromCalPlate3DAtPose_i*CalPlate3DFromRobotBase3D represents the inverse of RobotBase3DFromHand3D observed by the camera at pose i (Camera3DFromCalPlate3DAtPose_iis obtained by using the observed features in Raw2D and camera calibration information).
• Hand3DFromHand3DAtPose_iindicates the difference between the pose based on robot calibration parameters ofrobot105 and the estimated pose based on images fromcamera110. If the calibration parameters are completely accurate, then Hand3DFromHand3DPose_iwould be an identity transformation. The cost function can be defined as follows:
• $\sum _{\mathrm{For}\mathrm{each}\mathrm{camera},\mathrm{each}\mathrm{pose}i,}^{}{\left(\begin{array}{c}\mathrm{Hand}3\mathrm{DFromHand}3{\mathrm{DAtPose}}_i·\\ \mathrm{translation}·\mathrm{length}+w*\\ \mathrm{absolute}\mathrm{value}\\ (\mathrm{Hand}3\mathrm{DFromHand}3{\mathrm{DAtPose}}_i·\\ \mathrm{rotation}·\mathrm{angle})\end{array}\right)}^2$
• In some embodiments, the cost function can measure the difference between reported joint angles for the joints of robot105 (e.g., as measured by robot105) and the estimated joint angles of the corresponding joints ofrobot105. The cost function can estimate the joint angles for the joints ofrobot105 based on the captured images and the calibration parameter values. A transformation RobotBase3DFromHand3D estimated can be defined as follows:
• 
  RobotBase3DFromHand3D_estimated=RobotBase3DFromCalPlate3D*CalPlate3DFromCamera3DAtPose_i*Camera3DFromHand3D.
• The inverse kinematic equations can be used to compute the joint angles corresponding to RobotBase3DFromHand3D_estimated. The cost function can be defined as follows:
• $\sum _{\begin{array}{c}\mathrm{For}\mathrm{each}\mathrm{camera},\mathrm{each}\mathrm{pose}i,\\ \mathrm{each}\mathrm{joint}j\end{array}}^{}{\left(\mathrm{EstimateJointAngle}-\mathrm{ReportedJointAngle}\right)}^2\mathrm{OR}$$\sum _{\begin{array}{c}\mathrm{For}\mathrm{each}\mathrm{camera},\mathrm{each}\mathrm{pose}i,\\ \mathrm{each}\mathrm{joint}j\end{array}}^{}{\left(\mathrm{EstimateJointEncoderCount}-\mathrm{ReportedJointEncoderCount}\right)}^2$
• Beneficially, the cost function does not need to include w for balancing the translation and rotation components.
• 4. Enhancing Refinement
• In the illustrated example formachine vision system100, the calibration parameters that could be refined include 32 robot calibration parameters: 20 D-H parameters for the joints ofrobot105, 6 robot calibration parameters for RobotBase3DFromJoint1, and 6 robot calibration parameters for Joint6FromHand3D. The calibration parameters also include 6 parameters for Camera3DFromHand3D, and 6 parameters for RobotBase3DFromCalPlate3D bringing the total number of calibration parameters to 44 parameters.
• There can be redundancy in the calibration parameters, making it unnecessary to refine all of the calibration parameters. In some embodiments, Joint6FromHand3D can be combined with Hand3DFromCamera3D to reduce the number of calibration parameters being refined. In some embodiments, RobotBase3DFromJoint1 can be combined with CalPlate3DFromRobotBase3D. Combining the transformations can reduce the calibration parameters for refining to 32 (e.g., 6 from CalPlate3DFromJoint1, 6 from Joint6FromCamera3D, and 20 from Joint1FromJoint2, . . . , Joint5FromJoint6). In some embodiments, the D-H parameter θ offset and d offset from Joint1FromJoint2 can be combined into CalPlate3DFromJoint1, reducing the number of parameters for refining to 30. In some embodiments, only select calibration parameters are refined. For example, the offsets of the theta angles (e.g., the joint origins) can be refined. Beneficially, by reducing the number of calibration parameters for refining, the speed and accuracy of refining process can be improved. In some embodiments, refining can be done in groups of calibration parameters. For example, the calibration parameter values for Camera3DFromHand3D and RobotBase3DFromCalPlate3D can be fixed while refining the 20 robot calibration parameters. Then, the 20 robot calibration parameters can be fixed and the calibration parameters for Camera3DFromHand3D and RobotBase3DFromCalPlate3D can be refined.
• It should be appreciated thatmachine vision system100 is an exemplary machine vision system for practicing the technology. Other configurations are contemplated. In some embodiments, steps described as performed by the control system can be performed by a motion controller, a machine vision processor, and/or a motion controller in cooperation with a machine vision processor. For example, a motion controller can direct the robot through a series of poses and cause the machine vision processor to acquire images of the calibration target at each pose. The motion controller can provide to the machine vision processor information about the robot joint angles at each pose. The motion controller can cause the machine vision processor to perform the described calibration and refinement. As another example, a machine vision processor can cause the motion controller to direct the robot through a series of poses and provide to the machine vision processor information about the robot joint angles at each pose. The machine vision processor can acquire images of the calibration target at each pose. The machine vision processor can perform the described calibration and refinement. More generally, the control system can comprise a motion controller, a machine vision processor, or any combination thereof working in concert to implement the technology herein.
• In some embodiments, more than one camera can be used. In such machine vision systems, the cost function can be the combination of the cost functions for each camera. In some embodiments, a virtual 3D camera can be created. By using a virtual 3D camera, a single camera-based cost function can be used. Beneficially, by using a virtual 3D camera, the number of calibration parameters for refining can be smaller and the cameras' spatial relationship can be held constant during the refinement.
• In some embodiments, multiple work spaces can be used (e.g., the robot can move within and interact with objects in more than one work space). In implementations where the camera moves with the robot, then calibration can include placing a calibration target in each workspace and collecting calibration data (e.g., images and joint angles for a plurality of robot poses) for each workspace. Each additional workspace can result in six additional calibration parameters for refining. In some embodiments, the cost function can be the combination of cost functions for each work space. In some embodiments, one or more stationary cameras can be used. If the one or more cameras are stationary, and the calibration target is mounted on the robot, then the cost functions defined above can be used by exchanging Camera3D and CalPlate3D in the equations. In some embodiments, the technology can be used for refining the calibration parameters for a machine vision system in which the robot motion only moves in two dimensions. It should be appreciated that the cost functions can be changed to use only 2D information. In such cases, due to the limitation of the robot motions, the number of robot calibration parameters for refinement can be reduced for robust refinement.
• B. Calibration for Machine Vision Systems with Motion Stages
• It should be appreciated that the technology can refine calibration parameters for machine vision systems including motion stages by using approaches similar to those described above. The technology can refine hand-eye calibration parameters and motion stage calibration parameters in a single refinement calculation.
• FIG. 5 depictscalibration scheme500 in accordance with the technology. In some embodiments, the calibration scheme can be used for fixed camera configurations. For example,calibration scheme500 can be used formachine vision system200 ofFIG. 2. InFIG. 5, the ovals represent coordinate spaces.FIG. 5 includes Home2D coordinatespace510, Stage2D coordinatespace515, CalPlate2D coordinatespace520, Camera2D coordinatespace525, and Raw2D coordinatespace530. Arrows between ovals represent transformations between the illustrated coordinate spaces. The calibration parameter values used for the transformations can be determined and refined, as described with respect to the illustrative calibration methods below.
• In the illustrated example, Home2D coordinatespace510 can be the coordinate space in which other coordinate spaces and their relationships are described. Home2D coordinatespace510 can be orthonormal and defined by the X axis ofmotion stage205 and the center of rotation ofmotion stage205. For example, the origin of Home2D coordinatespace510 can be at its center of rotation whenmotion stage205 is at the home position. The x axis of Home2D coordinatespace510 can be aligned with the x axis ofmotion stage205. They axis of Home2D coordinatespace510 can be 90 degrees from the X axis and in the general direction of the y axis ofmotion stage205. Stage2D coordinatespace515 can be an orthonormal coordinate space that is attached to the center of rotation ofmotion stage205, and moves along withmotion stage205. CalPlate2D coordinatespace520 can be the coordinate space ofcalibration target220. This space can be moved around in the Home2D coordinatespace510 coordinates bymotion stage205. CalPlate2D coordinatespace520 can be an orthonormal coordinate system, although its length unit may have a non-identity scale factor from Home2D coordinatespace510. CalPlate2D coordinatespace520 and Home2D coordinatespace510 can have different handedness. Camera2D coordinatespace525 can be the physical orthonormal coordinate space forcamera210. The origin of Camera2D coordinatespace525 can be at a position in Home2D coordinatespace510 that corresponds to the center of the image window ofcamera210. Its x axis is parallel to the x axis of Raw2D coordinatespace530 at the origin of Camera2D coordinatespace525. The y axis can be 90 degrees from the x axis and in the general direction of the y axis of Raw2D coordinatespace530. Raw2D coordinatespace530 can be the pixel space coordinate system ofcamera210.
• The transformation between the coordinate spaces can be rigid transformations (e.g., comprising a translation component and a rotation component) and/or rigid plus handedness transforms (e.g. comprising a rigid transform plus a handedness component) and/or transformations based on the kinematic calibration parameters ofmotion stage205. For example, the transformations between Camera2D coordinatespace525 and Home2D coordinatespace510 and between Stage2D coordinatespace515 and CalPlate2D coordinatespace520 can be 2D rigid transformations. The transformations between CalPlate2D coordinatespace520 and Camera2D coordinatespace525 can be 2D rigid transformation. The transformations between Home2D coordinatespace510 and Stage2D coordinatespace515 can be based on, e.g., the actuators' encoder counts. The transformation from Camera2D coordinatespace525 to Raw2D coordinatespace530 can be based on a conventional camera calibration, as is well known in the art.
• 1. Capturing Calibration Data
• Again with reference toFIG. 4, the depicted method can be performed bycontrol system215 ofmachine vision system200. Atstep405, control system captures images and actuator encoder counts whenmotion stage205 is in a plurality of poses. At each pose,control system215 causes the camera to capture an image of at least a portion ofcalibration target220 that contains one or more features.Control system215 further captures the encoder count for each actuator of motion stage2015 at each pose.
• 2. Determining Initial Calibration Parameter Values
• Atstep410,control system215 determines initial calibration parameter values. The initial calibration parameter values can be the values used to determine the transformations discussed with reference toFIG. 5 prior to refinement. The values of these calibrations parameters can be refined in accordance with the present technology. In the illustrated example ofFIG. 4,control system215 can obtain the initial motion stage calibration parameter values formotion stage205 based on the specification ofmotion stage205. For example, a motion stage manufacturer will typically provide these values. Using these initial calibration values and the encoder counts at a particular pose,control system215 can determine the transformation Home2DFromStage2D.
• Control system215 can perform a hand-eye calibration to determine initial hand-eye calibration parameter values. For example,control system215 can use the Cognex Vision Library to determine initial hand-eye calibration parameter values based on the captured images and the Home2DFromStage2D transformation for a plurality of poses. As a result of the hand-eye calibration,control system215 can determine Stage2DFromCalPlate2D, CalPlate2DFromCamera2D, and Camera2DfromHome2D.
• 3. Refining Calibration Parameters
• Atstep415,control system215 refines a plurality of the calibration parameters to determine more accurate values for those calibration parameters. To refine the calibration parameters,control system215 determines the calibration parameters that minimize a cost function. In general, the cost functions can be based on comparingmotion stage205 motions (e.g., as determined from the actual encoder counts read from the motion stage) and the observed camera or calibration target motions (e.g., based on the images captured by the camera). Embodiments of the technology can use a variety of cost functions.
• In some embodiments, the cost function can measure the differences between the observed encoder count (e.g., as read from the motion stage) and the expected/estimated encoder count as calculated using the initial calibration parameter values (or successively refined calibration parameter values). For example, for each motion stage poses i, an estimated motion stage pose i′ (e.g., in Home2D coordinate space510) can be computed. Pose i′ can be computed as follows:
• 
  Posei′=Home2DFromCamera2D*Camera2DFromCalPlate2DAtPose_i*Plate2DFromStage2D
• • Where Camera2DFromCalPlate2DAtPose_iis computed from image features and the camera calibration information.
• Estimated encoder counts for pose i can be computed based on pose i′ and the initial calibration parameter values (or successively refined calibration parameter values).
• The cost function can be the sum of the squares of encoder count differences for every actuator, for every pose, and for every camera (in multiple camera machine vision systems). The cost function can be defined as follows:
• $\sum _{\begin{array}{c}\mathrm{for}\mathrm{each}\\ \mathrm{camera}\end{array}}^{}\sum _{\begin{array}{c}\mathrm{for}\mathrm{each}\\ \mathrm{pose}i\end{array}}^{}\sum _{\begin{array}{c}\mathrm{for}\mathrm{each}\\ \mathrm{actuator}j\end{array}}^{}{\left(\mathrm{Read}\mathrm{encoder}\mathrm{count}-\mathrm{Estimated}\mathrm{encoder}\mathrm{count}\right)}^2$
• In some embodiments, the cost function can measure the differences between the observed location of features in Raw2D and the expected (mapped) location of features in Raw2D. The transformation MappedFeatureRaw2D_ijcan be defined as follows:
• 
  MappedFeatureRaw2D_ij=Raw2DFromCamera2D*CalPlate2DFromCamera2DAtPose_iEstimated⁻¹*CalibrationTargetFeature_j
• 
  where CalPlate2DFromCamera2DAtPose_iEstimated=CalPlate2DFromStage2D*Home2DFromStage2DAtPose_i⁻¹*Home2DFromCamera2D
• Home2DFromStage2DAtPosei can be computed based on the encoder counts read frommotion stage205 at pose i and the initial calibration parameter values (or successively refined calibration parameter values).Control system115 can then compute the distance between the observed feature location and the mapped feature location. The cost function can be defined as follows:
• $\sum _{\begin{array}{c}\mathrm{For}\mathrm{each}\mathrm{camera},\mathrm{each}\mathrm{pose}i,\\ \mathrm{and}\mathrm{each}\mathrm{feature}j\end{array}}^{}{\left(\mathrm{distance}\left(\mathrm{ObservedFeatureRaw}2D_{\mathrm{ij}},\mathrm{MappedFeatureRaw}2D_{\mathrm{ij}}\right)\right)}^2$
• In some embodiments, the cost function can measure the differences between the location of features in CalPlate2D and the mapped location of features oncalibration target220. For example, Home2DFromStage2DAtPose_ican be computed based on the encoder counts read frommotion stage205 at pose i and the initial calibration parameter values (or successively refined calibration parameter values). The transformation CalPlate2DFromCalPlate2DAtPose can be computed as follows:
• 
  CalPlate2DFromCalPlate2DAtPose_i=CalPlate2DFromStage2D*Home2DFromStage2DAtPose_i⁻¹*Home2DFromCamera2D*Camera2DFromPlate2DAtPose_i
• Where Home2DFromStage2DAtPose_ican be computed based on the encoder counts read frommotion stage205 at pose i and the initial calibration parameter values (or successively refined calibration parameter values)
• Camera2DFromCalPlate2DAtPose_iis obtained by using the observed features in Raw2D and camera calibration information.
• The transformation CalPlate2DFromCalPlate2DAtPose_ican be used to map calibration target features at pose i. The cost function can then be defined as follows:
• $\sum _{\begin{array}{c}\mathrm{for}\mathrm{each}\\ \mathrm{cameracamera}\mathrm{ID}\end{array}}^{}\sum _{\begin{array}{c}\mathrm{for}\mathrm{each}\\ \mathrm{pose}i\end{array}}^{}\sum _{\begin{array}{c}\mathrm{for}\mathrm{each}\\ \mathrm{feature}j\end{array}}^{}({\mathrm{distance}\left(\left(\begin{array}{c}\mathrm{CalPlate}2\mathrm{DFrom}-\\ \mathrm{CalPlate}2{\mathrm{DAtPose}}_i*\\ \mathrm{featureCalPlate}2D_j,\\ \mathrm{featureCalPlate}2D_j\end{array}\right)\right)}^2$
• With respect to UVW motion stages, motion stage calibration parameters can include, for each actuator, the actuator's roller's center position, the roller's radius, the actuator's direction, the actuator's step size, the actuator's home encoder count, and the actuator's striker plate's direction. In some embodiments, a subset of the UVW motion stage's motion stage calibration parameters can be refined. For example, in some embodiments, the roller's radius is not refined. In some embodiments, the home encoder count is not refined. As noted above, refinement efficiency can be increased by reducing the number of calibration parameters being refined. With respect to XY Theta motion stages, motion stage calibration parameters can include, for each of the X and Y actuators, the actuator's direction and the actuator's step size. For the theta actuator, motion stage calibration parameters can include the actuator's roller's center position, the roller's radius, and the actuator's striker plate's direction. In some embodiments, a subset of the XY Theta motion stage's motion stage calibration parameters can be refined. For example, in some embodiments, the roller's radius is not refined.
• It should be appreciated thatmachine vision system200 is an exemplary machine vision system for practicing the technology. Other configurations are contemplated. In some embodiments, steps described as performed by the control system can be performed by a motion controller, a machine vision processor, and/or a motion controller in cooperation with a machine vision processor. For example, a motion controller can direct the motion stage through a series of poses and cause the machine vision processor to acquire images of the calibration target at each pose. The motion controller can provide to the machine vision processor information about the motion stage at each pose. The motion controller can cause the machine vision processor to perform the described calibration and refinement. As another example, a machine vision processor can cause the motion controller to direct the motion stage through a series of poses and provide to the machine vision processor information about the motion stage at each pose. The machine vision processor can acquire images of the calibration target at each pose. The machine vision processor can perform the described calibration and refinement. More generally, the control system can comprise a motion controller, a machine vision processor, or any combination thereof working in concert to implement the technology herein.
• In some embodiments, one or more stationary cameras can be used as described above. If the one or more cameras are moving cameras, that is the cameras are rigidly attached to the motion stage, and the calibration target is stationary, then the cost functions defined above can be used by exchanging Camera2D and CalPlate2D in the equations.
• The above-described techniques can be implemented in digital and/or analog electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The implementation can be as a computer program product, i.e., a computer program tangibly embodied in a machine-readable storage device, for execution by, or to control the operation of, a data processing apparatus, e.g., a programmable processor, a computer, and/or multiple computers. A computer program can be written in any form of computer or programming language, including source code, compiled code, interpreted code and/or machine code, and the computer program can be deployed in any form, including as a stand-alone program or as a subroutine, element, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one or more sites.
• Method steps can be performed by one or more processors executing a computer program to perform functions of the technology by operating on input data and/or generating output data. Method steps can also be performed by, and an apparatus can be implemented as, special purpose logic circuitry, e.g., a FPGA (field programmable gate array), a FPAA (field-programmable analog array), a CPLD (complex programmable logic device), a PSoC (Programmable System-on-Chip), ASIP (application-specific instruction-set processor), or an ASIC (application-specific integrated circuit), or the like. Subroutines can refer to portions of the stored computer program and/or the processor, and/or the special circuitry that implement one or more functions.
• Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital or analog computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and/or data. Memory devices, such as a cache, can be used to temporarily store data. Memory devices can also be used for long-term data storage. Generally, a computer also includes, or is operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. A computer can also be operatively coupled to a communications network in order to receive instructions and/or data from the network and/or to transfer instructions and/or data to the network. Computer-readable storage mediums suitable for embodying computer program instructions and data include all forms of volatile and non-volatile memory, including by way of example semiconductor memory devices, e.g., DRAM, SRAM, EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and optical disks, e.g., CD, DVD, HD-DVD, and Blu-ray disks. The processor and the memory can be supplemented by and/or incorporated in special purpose logic circuitry.
• The above described techniques can be implemented in a distributed computing system that includes a back-end component. The back-end component can, for example, be a data server, a middleware component, and/or an application server. The above described techniques can be implemented in a distributed computing system that includes a front-end component. The front-end component can, for example, be a client computer having a graphical user interface, a web browser through which a user can interact with an example implementation, and/or other graphical user interfaces for a transmitting device. The above described techniques can be implemented in a distributed computing system that includes any combination of such back-end, middleware, or front-end components.
• The components of a computing system can be interconnected by transmission medium, which can include any form or medium of digital or analog data communication (e.g., a communication network). Transmission medium can include one or more packet-based networks and/or one or more circuit-based networks in any configuration. Packet-based networks can include, for example, the Internet, a carrier internet protocol (IP) network (e.g., local area network (LAN), wide area network (WAN), campus area network (CAN), metropolitan area network (MAN), home area network (HAN)), a private IP network, an IP private branch exchange (IPBX), a wireless network (e.g., radio access network (RAN), Bluetooth, Wi-Fi, WiMAX, general packet radio service (GPRS) network, HiperLAN), and/or other packet-based networks. Circuit-based networks can include, for example, the public switched telephone network (PSTN), a legacy private branch exchange (PBX), a wireless network (e.g., RAN, code-division multiple access (CDMA) network, time division multiple access (TDMA) network, global system for mobile communications (GSM) network), and/or other circuit-based networks.
• Information transfer over transmission medium can be based on one or more communication protocols. Communication protocols can include, for example, Ethernet protocol, Internet Protocol (IP), a Peer-to-Peer (P2P) protocol, Hypertext Transfer Protocol (HTTP), Session Initiation Protocol (SIP), a Global System for Mobile Communications (GSM) protocol, Universal Mobile Telecommunications System (UMTS), 3GPP Long Term Evolution (LTE) and/or other communication protocols.
• Comprise, include, and/or plural forms of each are open ended and include the listed parts and can include additional parts that are not listed. And/or is open ended and includes one or more of the listed parts and combinations of the listed parts.
• One skilled in the art will realize the technology may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The foregoing embodiments are therefore to be considered in all respects illustrative rather than limiting of the technology described herein.
• The technology has been described in terms of particular embodiments. The alternatives described herein are examples for illustration only and not to limit the alternatives in any way. The steps of the technology can be performed in a different order and still achieve desirable results. Other embodiments are within the scope of the following claims.

Claims (30)

What is claimed is:

1) A machine vision system comprising:

a robot;

a camera; and

a control system in communication with the robot and the camera, the control system configured to:

move the robot to a plurality of poses, and for each pose of the plurality of poses:

capture, with the camera, an image of one or more features on a calibration target; and

capture a plurality of robot joint angles;

obtain a plurality of initial values for a plurality of robot calibration parameters;

determine a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of robot calibration parameters, and the image and the plurality of robot joint angles captured for each of the plurality of poses;

determine a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the robot calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of robot calibration parameters to minimize a cost function.

2) The machine vision system ofclaim 1, wherein the control system is further configured to:

use the plurality of final values for the plurality of hand-eye calibration parameters and the plurality of final values for the robot calibration parameters for a vision guided robot application.

3) The machine vision system ofclaim 1, wherein the control system is further configured to:

move the robot using the plurality of final values for the robot calibration parameters.

4) The machine vision system ofclaim 1, wherein the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of robot joint angles captured for the pose.

5) The machine vision system ofclaim 1, wherein the cost function measures, for each pose of the plurality of poses, a difference between a composite transformation and an identity transformation, the composite transformation being a composition of a first transformation from a first coordinate space to a second coordinate space and a second transformation from the second coordinate space to the first coordinate space, the first transformation and the second transformation determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, the plurality of observed features of the calibration target for the pose, and the plurality of robot joint angles captured for the pose.

6) The machine vision system ofclaim 1, wherein the cost function measures, for each pose of the plurality of poses, a difference between a robot joint angle of the plurality of robot joint angles captured for the pose and an estimated joint angle for the pose, the estimated joint angle determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.

7) The machine vision system ofclaim 1, wherein the control system is further configured to minimize the cost function using non-linear least squares.

8) The machine vision system ofclaim 1, wherein the control system is configured to refine less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of robot calibration parameters.

9) The machine vision system ofclaim 1, wherein the plurality of robot calibration parameters comprises Denavit-Hartenberg parameters for the robot.

10) A machine vision system-implemented calibration method comprising:

moving, by a control system, a robot to a plurality of poses, and for each pose of the plurality of poses:

capturing, by a camera, an image of one or more features on a calibration target; and

capturing, by the control system, a plurality of robot joint angles;

obtaining, by the control system, a plurality of initial values for a plurality of robot calibration parameters;

determining, by the control system, a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of robot calibration parameters, and the image and the plurality of robot joint angles captured for each of the plurality of poses;

determining, by the control system, a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the plurality of robot calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of robot calibration parameters to minimize a cost function.

11) The method ofclaim 10, wherein the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of robot joint angles captured for the pose.

12) The method ofclaim 10, wherein the cost function measures, for each pose of the plurality of poses, a difference between a composite transformation and an identity transformation, the composite transformation being a composition of a first transformation from a first coordinate space to a second coordinate space and a second transformation from the second coordinate space to the first coordinate space, the first transformation and the second transformation determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, the plurality of observed features of the calibration target for the pose, and the plurality of robot joint angles captured for the pose.

13) The method ofclaim 10, wherein the cost function measures, for each pose of the plurality of poses, a difference between a robot joint angle of the plurality of robot joint angles captured for the pose and an estimated joint angle for the pose, the estimated joint angle determined based on a first plurality of values for the plurality of robot calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.

14) The method ofclaim 10, further comprising minimizing the cost function using non-linear least squares.

15) The method ofclaim 10, further comprising refining less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of robot calibration parameters.

16) The method ofclaim 10, wherein the plurality of robot calibration parameters comprises Denavit-Hartenberg parameters for the robot.

17) A machine vision system comprising:

a motion stage;

a camera; and

a control system in communication with the motion stage and the camera, the control system configured to:

move the motion stage to a plurality of poses, and for each pose of the plurality of poses:

capture, with the camera, an image of one or more features on a calibration target; and

capture a plurality of encoder counts for a plurality of actuators of the motion stage;

obtain a plurality of initial values for a plurality of motion stage calibration parameters;

determine a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of motion stage calibration parameters, and the image and the plurality of encoder counts captured for each of the plurality of poses;

determine a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the motion stage calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of motion stage calibration parameters to minimize a cost function.

18) The machine vision system ofclaim 17, wherein the control system is further configured to:

use the plurality of final values for the plurality of hand-eye calibration parameters and the plurality of final values for the motion stage calibration parameters for a vision guided motion stage application.

19) The machine vision system ofclaim 17, wherein the control system is further configured to:

move the motion stage using the plurality of final values for the motion stage calibration parameters.

20) The machine vision system ofclaim 17, wherein the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of motion stage calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of encoder counts captured for the pose.

21) The machine vision system ofclaim 17, wherein the cost function measures, for each pose of the plurality of poses, a difference between an encoder count of the plurality of encoder counts captured for the pose and an estimated encoder count for the pose, the estimated encoder count determined based on a first plurality of values for the plurality of motion stage calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.

22) The machine vision system ofclaim 17, wherein the control system is further configured to minimize the cost function using non-linear least squares.

23) The machine vision system ofclaim 17, wherein the control system is configured to refine less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of motion stage calibration parameters.

24) The machine vision system ofclaim 17, wherein the plurality of motion stage calibration parameters comprises one or more of roller center position, roller radius, direction, step size, home encoder count, and striker plate's direction for an actuator.

25) A machine vision system-implemented calibration method comprising:

moving, by a control system, a motion stage to a plurality of poses, and for each pose of the plurality of poses:

capture, by a camera, an image of one or more features on a calibration target; and

capturing, by the control system, a plurality of encoder counts for a plurality of actuators of the motion stage;

obtaining, by the control system, a plurality of initial values for a plurality of motion stage calibration parameters;

determining, by the control system, a plurality of initial values for a plurality of hand-eye calibration parameters based on the plurality of initial values for the plurality of motion stage calibration parameters, and the image and the plurality of encoder counts captured for each of the plurality of poses;

determining, by the control system, a plurality of final values for the plurality of hand-eye calibration parameters and a plurality of final values for the motion stage calibration parameters by refining one or more of the plurality of hand-eye calibration parameters and one or more of the plurality of motion stage calibration parameters to minimize a cost function.

26) The method ofclaim 25, wherein the cost function measures, for each pose of the plurality of poses, a difference between an observed location of a feature in the image captured for the pose and an expected location of the feature, the expected location determined based on a first plurality of values for the plurality of motion stage calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of encoder counts captured for the pose.

27) The method ofclaim 25, wherein the cost function measures, for each pose of the plurality of poses, a difference between an encoder count of the plurality of encoder counts captured for the pose and an estimated encoder count for the pose, the estimated encoder count determined based on a first plurality of values for the plurality of motion stage calibration parameters, a second plurality of values for the plurality of hand-eye calibration parameters, and the plurality of observed features of the calibration target for the pose.

28) The method ofclaim 25, further comprising minimizing the cost function using non-linear least squares.

29) The method ofclaim 25, further comprising refining less than all of the plurality of hand-eye calibration parameters and less than all of the plurality of motion stage calibration parameters.

30) The method ofclaim 25, wherein the plurality of motion stage calibration parameters comprises one or more of roller center position, roller radius, direction, step size, home encoder count, and striker plate's direction for an actuator.

Images