Hand-eye calibration, three ways

Every grasp in my pick-and-place stack goes through one transform: camera to gripper. If it’s off by a centimetre, a grasp with a few millimetres of tolerance fails, and no amount of good perception fixes it.

So I built a scan → solve → evaluate pipeline that works for any robot and camera pair, and compared three ways of getting that transform. All numbers below are from a Franka FR3 with a RealSense D435.

The pipeline

1. 1
   Scan

   The arm runs an automated spiral around the target: 150 poses for training, 35 for testing.

2. 2
   Filter

   Drop views with reprojection error above 3.5 px.

3. 3
   Solve

   OpenCV’s five AX = XB solvers, plus my own solver: Adam on ‖AX − XB‖² in PyTorch.

4. 4
   Evaluate

   No ground truth needed: map the board into the world through each test pose and check how consistent it stays.

Method 1: the standard board

Solver	Rotation	Distance
Mine (PyTorch, Adam)	0.40°	7.77 mm
Park / Horaud	0.39° ± 0.19	7.80 ± 4.41 mm
Tsai	3.00°	21.88 mm

Rotation is fine. Distance is about 8 mm, and my gradient-descent solver landed within 0.03 mm of Park and Horaud.

When three different solvers agree on 8 mm, the solver isn’t the problem. The robot’s kinematics set the floor.

Method 2: take the robot out with a fixed camera

The idea: a second, fixed camera measures the gripper directly through an AprilTag on a 3D-printed holder, so robot kinematics drop out of the chain.

Best result: Horaud at 5.96 ± 3.33 mm, but rotation got worse, about 1°. The reason showed up in the projections. With the fixed camera about 2 m away, pushing the tag 1 cm farther in z moved the projected axes by only 1 px.

A low reprojection error is necessary, not sufficient. At that distance it can’t see depth errors.

Method 3: the “selfie”

If you know where the tag sits relative to the gripper (because you printed the holder), you don’t need a solver at all. The eye-in-hand camera looks at a tag at a known offset from the fingers, and the transform falls out of PnP plus the CAD geometry.

Repeatability of the gripper position over 11 z-rotations: 0.59 mm mean, 0.35 mm std, in x and y. That’s repeatability, not absolute accuracy, but it’s an order of magnitude tighter than the solver-based methods.

What I’d tell someone starting

• View the tag at an angle, not head-on. Frontal views make the pose ambiguous.
• A printed holder warps. If the geometry carries the accuracy, machine it.
• Check calibration with an independent measurement, not with the residual the solver minimised.

The same tools later calibrated the Viper 300s and the UR5e, including the GTC Paris setup the night before shipping.