Six years of shipping perception to cars

From 2017 to 2023 I worked on automotive perception at ESR Labs: segmentation, object detection, tracking, depth and camera-blindness detection, deployed on NVIDIA Drive PX / Pegasus and other embedded targets. Most of the code is internal, so this is a tour of what I’d still tell someone doing the same job.

Make networks cheap to try

My first project was a config-driven TensorFlow training framework: a new network is a small plugin class plus a JSON file, so architectures can be compared under the same data, losses and logging. I added a fast segmentation network of my own design, stereo depth networks and a freeze-graph → TensorRT exporter. In 2020 I did the same in PyTorch; every later project in that repo built on its base model.

One small win: optimising capsule-network routing (stop-gradient on early iterations, three iterations instead of four) gave about 10× faster training on traffic signs.

Read your labels as the right type

Two bugs I’ll never forget:

• Cityscapes disparity PNGs are 16-bit. Read as 8-bit, the ground truth was silently corrupted.
• Mapillary labels read as uint8 destroyed the class ids.

And horizontal flip is not a free augmentation for traffic signs: it changes what a direction sign means.

Rain detection: don’t trust single frames

For camera-blindness detection I built rain detection end to end: a dataset with dual labels from a rain and fog simulator (22,000 synthetic images with pixel-accurate rain masks), a classifier, then a segmentation model.

Frame scores oscillate with the wiper. Each wipe clears the glass for a moment. The fix was a sequence-level verdict: take the peak over a window instead of the mean, so wiper dips don’t cancel real rain.

Decide on sequences when the sensor has a cycle.

Profile before you optimise

Chrome-trace profiling of the deployed graph found the slow ops: transposed convolutions (replaced with upsample + conv), a tiled gather, and a py_func that blocked on the GIL (replaced with a hash table). TensorRT 7.1 also needed align_corners=True to match.

INT8 without losing accuracy

I measured post-training INT8 on a real-time segmentation net (Fast-SCNN, 16 Mapillary classes):

Setup	mIoU
TensorFlow FP32	0.5828
TensorRT INT8, no calibration	0.5328
TensorRT INT8, entropy calibration (100 images)	0.5828

Naive INT8 cost 5 points. Calibration got all of it back. That also made the quantisation regulariser I had been training with unnecessary.

One engine, three frameworks

On the C++ side I worked on a generic inference engine for the in-house ROS-like middleware: TensorFlow, LibTorch and TensorRT behind one interface, plus a hybrid that runs the backbone in TensorRT and the heads TensorRT can’t handle in TorchScript. Tensors know their device, so they’re copied only when they aren’t already on the inference GPU. Models became swappable by config, not by rewrite.

I also wrote a C++ Kalman filter library for ego-vehicle localisation, with partial update-only filters so one sensor’s errors don’t leak into the main filter.

Debug tools pay for themselves

When camera-to-map localisation drifted, I spent a week building a visual debugger: project curb segmentation onto the ground, score a particle grid against the map, show the heat-map. It separated detection errors from map errors from plain ambiguity (curbs look identical along a straight road), and found a bug where particles were projected from the car origin instead of the camera.

Negative results count

An ultrasound feasibility study tried to turn raw parking-sensor echoes into obstacle maps with five staged network variants. All of them fit the training scenes; none generalised to held-out scenes, and regularisation didn’t close the gap. That’s a result too: it said the data or the formulation was wrong, not the model size.

The thread through all of it: small, fast networks, measured on the target hardware. It’s the same thread in my robotics work now.