Teaching a UR5e lab chores with AprilTags

Drawer, centrifuge knob, button, lid, rack. No world model, just a calibration chain and YAML pipelines. And the one inverse that cost 230 mm.

A rack of sample tubes rides an AMR through lab stations. At each station a UR5e with a Robotiq 2F-85 and a wrist RealSense does the manual part: open a drawer, turn a centrifuge knob, press a button, lift a lid, place the rack. An MQTT orchestrator triggers each skill.

I built the arm side: perception, calibration, the action engine, the station skills and the drivers. The orchestrator, UI and demo tooling were teammates’ work.

No world model, a calibration chain

Instead of a full scene model, every station carries AprilTags. A chain of transforms goes from tag to the station’s root to the affordance, the thing the gripper actually touches: a handle, a button, a knob.

1
Tag
Detected on the wrist camera.
2
Root
A fixed offset from tag to the station frame.
3
Affordance
Handle, button or knob pose in the station frame, set once with a keyboard annotation tool.

Moving a station means re-running a calibration, not editing code.

To make tag poses trustworthy I fused the RealSense’s two IR images: triangulate the tag centre, score the result with a four-term confidence, and fall back to a single view with a low confidence when only one camera sees the tag.

Skills as YAML

The skills are pipelines in a small YAML action engine: steps like approach, grasp, rotate, release, retreat. Station pipelines run 4 to 10 steps. The same pipelines run unchanged on URSim with a no-op gripper, which made it possible to test without the cell.

Pick-and-place debug view: approach, target and affordance poses in the robot frame.

The inverse that cost 230 mm

Hand-eye calibration uses a spiral scan around a ChArUco board. The first scans hit wrist singularities and protective stops; capping the elevation at 65° and shrinking the spiral fixed that.

The held-out error was still about 230 mm and 60°. Not noise. A wrong inv() in the chain. Removing it:

230 mm / 60° → 1.9 mm / 0.26°. One inverse.

I also pinned opencv-contrib below 5.0, because calibrateHandEye is gone there.

Where it is now

The arm code is extracted into a reusable package and ported to a second UR5e at another site; the drawer station runs there too. There are no task success rates or cycle times yet. That’s the next measurement.