Quantifying NVIDIA FoundationPose in the real world

I played with NVIDIA’s FoundationPose and built a small automated evaluation system to find out when it’s reliable enough for production use.

FoundationPose in 30 seconds

You give it a CAD model and an RGB-D frame. It samples candidate poses, then runs two stages: a coarse stage that scores many hypotheses, and a refinement stage that iteratively refines the best ones using RGB + depth. It’s trained on a large synthetic dataset, so it generalises surprisingly well out of the box.

Why my own evaluation

Before trusting it on a robot, I wanted numbers for a specific object, a specific camera and a specific lighting setup. Checking poses by eye is slow and subjective, so I built an automated pipeline that is easy to use, fast and quantitative.

The rig: a high-quality calibration board, the object, a Franka FR3 with a Zivid RGB-D camera (structured-light depth) on the end-effector. The arm moves to many random poses, always roughly looking at the object, and captures RGB-D at each one. Then I re-randomise the object pose and repeat to get several datasets.

Ground truth, roughly

The object is static while the camera moves, so:

1. 1
   Camera poses

   Get each frame’s camera pose relative to the calibration board.

2. 2
   Predict

   Run FoundationPose on every frame.

3. 3
   Common frame

   Map every predicted object pose into the board frame.

4. 4
   Consensus

   RANSAC-style consensus over all predictions gives one pseudo ground-truth pose.

5. 5
   Score

   Translation and rotation error of each frame against the consensus.

This only works when most poses are already good, but in that regime it gives a quick, automated metric per object, camera and lighting setup.

RGB vs depth

Two masking tests:

• Depth only (RGB zeroed): poses still very close to ground truth in many cases.
• RGB only (depth zeroed): often wrong, or rotation roughly right with translation way off.

FoundationPose leans heavily on depth. Good depth survives without colour; bad depth falls apart.

Update: two cameras and a public benchmark

I kept going until January 2025. Two additions.

RealSense vs Zivid. Five in-house objects, five recordings each, both cameras (3,688 RealSense and 3,584 Zivid images). Calibration reprojection error was 0.138 px on the RealSense and 0.098 px on the Zivid. Averaged over the objects, the Zivid’s pose spread was lower in distance (2.31 mm vs 3.40 mm), but its outlier rate was higher (12.5% vs 6.8%). The thin angle bracket is why: the Zivid’s wider baseline leaves its narrow side without depth, and FoundationPose needs depth.

HouseCat6D. On the 2,929 test images:

Metric	FoundationPose
3D IoU mAP @25 / 50 / 75	89.4 / 84.0 / 72.4
5°2cm / 5°5cm / 10°2cm / 10°5cm	57.0 / 63.4 / 65.3 / 74.6 %
Glass, IoU @25 / 50 / 75	34.5 / 7.1 / 2.9

That beats the published category-level methods (best IoU25 baseline: 83.7), but it isn’t a like-for-like comparison: FoundationPose gets each object’s CAD mesh and mask, the baselines are trained per category. The glass row is the same lesson as the masking test. No depth, no pose. Full inference took about two days.

Limitations

Depth dependence. Transparent, highly reflective or specular objects, anywhere the camera can’t measure depth, and the pose is unreliable too.

No sense of when it’s wrong. It scores many hypotheses and picks the best, but the score gap between a good pose and a very wrong one can be small, and the score scale changes per input. There’s no single threshold that says “safe to act”.

What I would add

If I trained it myself, I’d predict a score and an uncertainty per hypothesis already in the coarse stage. Filter high-uncertainty hypotheses early; refine the best remaining one only if its uncertainty is low, otherwise reject the frame. One actionable rule: only use poses that are both high-scoring and low-uncertainty. That would make it much safer to drop into a real manipulation pipeline.

Reference: FoundationPose