LLM-powered cobot manipulation through speech

Whisper, GPT-4, a robot-agnostic control layer, CNOS + FoundationPose and annotated grips. You talk; the arm picks.

I built a speech-driven cobot assistant: you talk to the robot, it understands what you want, looks at the scene, and picks and places objects, all orchestrated by an LLM.

Four parts: the LLM stack (cloud), a robot control layer, a perception stack, and a grip annotation tool.

The Franka FR3 cell with the test objects.

The LLM: brain and voice

Speech → text: Whisper transcribes the request.
Text → robot actions: GPT-4 maps the transcript to high-level robot commands, with access to the object database and the robot control API. It knows which objects exist and which actions are possible.
Text → speech: OpenAI TTS answers, so it feels conversational.

“Can you pick up the camera holder?” → Whisper → GPT-4 plans the perception and manipulation steps → “I’ll pick up the camera holder.”

Robot control: write it once

The control layer exposes abstract skills to the LLM: pick, place, drop, scan the scene. Inside it handles motion planning, collision checking, kinematics and execution.

It’s robot-agnostic. A new arm means implementing a common interface, not redesigning everything. The same logic runs on a Franka FR3, a Trossen Viper 300s and a UR5e. To the LLM, all three are “a robot that can pick and place”.

Perception: mesh in, pose out

Object-agnostic as long as there’s a 3D model:

CNOS detects objects by template matching, with templates rendered from each object’s mesh and texture.
FoundationPose gives the 6D pose from the CAD model.

Onboarding a new object: provide mesh and texture, render CNOS templates, add it to the database.

Grip annotation

For each object I define a set of grips once, in the object frame, with a dedicated annotation tool (from the side, from the top, on a handle). At runtime the grips are transformed into the world frame with the estimated pose, checked for reachability and collisions, and the best one is chosen for how the object lies.

The loop

1
Speak
“Pick up the camera holder.”
2
Transcribe
Whisper turns audio into text.
3
Plan
GPT-4 checks the object database and plans a sequence of skills.
4
Perceive
CNOS finds the object; FoundationPose estimates its pose.
5
Grip
Annotated grips are transformed into the scene; the best one wins.
6
Act
The control layer executes on whichever arm is connected.
7
Reply
TTS reports success, failure or asks for clarification.

Asking for the remote in Telugu, my mother tongue. It understood, but GPT replied in Hindi. Motion is slowed for testing.

An indirect request: I say I'm coughing. Cough drops are in the database, so it picks them up.

From the user’s side, they just talk to the cobot.

Stress test

To see how it holds up over hours, I ran a pick-and-drop loop: pick an object, drop it with a random yaw, find it again, repeat until 10 failures in a row.

Run 1: 133 of 153 picks (86.9%) in about 1 h 15 min, until the gripper fingers froze half-open.
Run 2: 302 of 342 picks (88.3%) in about 3 h 8 min.

Most failures were gripper errors, plus a few slips from small pose errors. Perception was rarely the weak link.

Keeping the LLM on a leash

The LLM is forced to answer with a function call, picks from a closed list of objects and actions, and anything it can’t parse becomes none. In 2026 I moved transcription and the LLM off the cloud: Whisper large-v3-turbo and Qwen3 8B on Ollama, running locally.

A teaser

I showed this demo at NVIDIA GTC Paris, and Jensen Huang stopped at the booth. That story is in the next post.

References: CNOS · FoundationPose · OpenAI API