Training-free 6D pose with OnePose++

SfM on an ArUco board, LoFTR features per 3D point, PnP at runtime. Then rotation broke it, and se2-LoFTR fixed it.

In this project I explored object pose estimation with OnePose++ and LoFTR-style feature matching, and later switched to a rotation-invariant variant (se2-LoFTR) to fix a key failure mode.

Building the 3D model with SfM + LoFTR

The pipeline starts with structure-from-motion in COLMAP:

I place the object on an ArUco marker pattern, which gives accurate camera poses for free.
Those poses are priors for COLMAP to build a high-quality 3D point cloud of the object.
For matching between frames I use LoFTR, a transformer-based matcher that predicts dense correspondences, instead of classical descriptors.

For each reconstructed 3D point, I aggregate its LoFTR features from all frames where it is observed.

Recording the object on the ArUco board for SfM.

Pose at inference time

1
Dense features
Run LoFTR on the query image.
2
2D–3D matches
Compare each 3D point’s aggregated feature with the image features by cosine similarity; keep high-similarity matches.
3
PnP
Solve PnP with the camera intrinsics for the 6D pose.

Because there are many dense matches, this is naturally robust to occlusion and small deformations.

Tracking a bag of cashews through occlusion and handling.

The problem: rotation

It worked well until the object was rotated significantly. Then pose estimation often failed. My hypothesis: LoFTR features are not rotation-invariant.

To test it, I took one image, rotated it in steps of 22.5°, and matched the original against each rotated copy. LoFTR matches stayed good only up to about 45°. Beyond that, correspondences degraded quickly.

Same rotation sweep. Left: LoFTR. Right: se2-LoFTR.

One fix would be multi-view reconstructions merged with ICP, storing several features per 3D point to cover many viewing angles. In practice that adds a lot of compute and memory at inference.

Instead I switched to se2-LoFTR, a rotation-invariant LoFTR. On the same experiment, matches stayed stable across the full rotation range I tested.

With se2-LoFTR in the pipeline, pose estimation held up when objects were rotated arbitrarily.

Multiple objects

Each object is reconstructed separately (SfM + ArUco + se2-LoFTR). At inference, se2-LoFTR runs once on the query image, each object’s model is matched independently, and PnP gives one pose per object.

Several objects posed in the same scene.

Limitations

Textureless, reflective or finely repetitive objects break it. SfM often fails or reconstructs poorly, and small patches don’t carry distinctive features. This method suits objects with reasonable texture and geometric detail.

References: OnePose++ · LoFTR · se2-LoFTR