Metric depth without depth labels
Camera plus sparse LiDAR in, dense metric depth out, trained with view synthesis and the car's own speed. Pose quality turned out to limit everything.
Dense depth labels for driving data barely exist. LiDAR is sparse, and stereo has its own failure modes. So I trained a depth-completion network that takes an RGB image plus sparse LiDAR and outputs dense metric depth, with the dense supervised loss switched off.
How it learns without labels
- 1Predict depth
A ResNet-18 encoder with a light decoder takes a 4-channel RGB + sparse depth input at 192×640.
- 2Predict motion
A PoseNet estimates the camera motion between frames.
- 3Reproject
Warp neighbouring frames into the current one with depth and pose; the photometric error is the loss.
- 4Anchor the scale
A velocity loss ties PoseNet’s translation to the car’s measured speed, so the depth comes out in metres.
Evaluation uses no median scaling. The depth has to be right in metres on its own.
It grew out of an earlier self-supervised monocular depth model on A2D2 and KITTI, whose checkpoint initialised this one.
Pose limits everything
View synthesis is only as good as the camera motion behind it.
- Integrating vehicle bus signals drifted.
- A vehicle-to-camera axis convention bug quietly broke the warps until I found it.
- Trajectories from visual SLAM gave the best configuration.
Self-supervised depth is a pose problem in disguise.
The scene isn’t static
The photometric loss assumes a static world. The car directly ahead moves with you, so it looks static at infinite depth. I masked the central band of the image (37.5–62.5% of the width) where the ego-lane car lives.
LiDAR can fail
To keep the network from leaning on LiDAR, training randomly drops points, and in 50% of samples drops all of them. I also tested a mono-only fallback.
What I’d watch for
I found and fixed a validation leak and a shuffling bug along the way. Both made numbers look better than they were. If a depth result looks too good, check the split before the model.