Humanoid teleoperation: SenseGlove + HTC Vive

I’ve been experimenting with teleoperating a humanoid’s arms using Nova2 SenseGloves and HTC Vive trackers inside Isaac Lab 2.2. For now only the upper body: arms move, legs stay fixed.

There are teleop setups with Apple Vision Pro and Meta Quest, but I wanted to avoid a few things:

• Wearing a headset for long periods.
• Vision-based hand tracking: limited by camera streams, extra latency from the hand-tracking network, and it fails when fingers are occluded.

So I went sensor-based: gloves for finger and hand pose, trackers for hand position. Both stream at 100 Hz+ with minimal processing.

Three containers, one humanoid

To avoid dependency conflicts, everything is split into three Docker containers on a virtual bridge network:

1. 1
   SenseGlove

   Reads the Nova2 gloves, publishes hand joints over ROS 2.

2. 2
   HTC Vive

   Reads the trackers, publishes hand positions over ROS 2. Based on vive_ros2.

3. 3
   Isaac Lab

   Runs the humanoid in Isaac Lab 2.2, subscribes to both and drives the arms.

Vive integration, and a quick Viper test

vive_ros2 is built for Vive controllers. I first used it as-is for a quick test with my Viper 300s arm.

Then I extended it to trackers, used only for hand position. Trackers don’t label themselves left or right, so I added a startup calibration: hold them in a known pose, and the session assigns left and right consistently.

The setup

Gloves give finger bend and hand orientation; trackers give hand position, one per hand. Isaac Lab gets hand joint targets from the gloves and hand position targets from the trackers. Fingers and wrist come from the gloves; coarse position in space comes from the trackers.

A few details that mattered

• The glove SDK only spoke ROS 1, so a ros1_bridge container joins it to the ROS 2 world.
• vive_ros2 originally put both trackers on one topic. I split it into one topic per tracker.
• Calibration runs on the first frame after every reset, so the operator’s pose at that moment becomes the reference.
• Wrist orientation turned out better from the glove’s IMU than from the tracker.
• I dropped DexPilot retargeting for a direct interpolation over the 22 hand joints.

Issues and next steps

• Simulation slowdown. All three containers on one host slow Isaac Lab down. Plan: Isaac Lab on one machine, gloves and Vive on another, over the network.
• Nova2 gloves are bulky. You can’t fully close your fingers, which shows in long sessions. Plan: thinner, lighter Manus gloves.

Sensor-based teleop works for humanoid arms without headsets or occlusion-prone hand tracking.

Update, 2026: real robots

The next steps went from simulation to hardware:

• Unitree G1 from a PICO headset. I dockerised the open-source TWIST2 stack (PICO XR, XRoboToolkit, an ONNX whole-body policy) so it starts on the real G1 with one docker compose up. The stack itself is upstream; my part was packaging it and adding Inspire dexterous hand support, with grip and open commands sent only on a state change to spare the Modbus link.
• Manus gloves. Built an Isaac Lab 2.3 image with the Manus Core SDK plugin, the lighter gloves from the plan above.
• UR5e from PICO controllers (work in progress): Placo IK with a manipulability task, an IK thread at about 100 Hz and a separate servo thread.

Yes, the G1 path uses a headset. For whole-body teleop it was the fastest way to a working robot.