How are stereo camera recording devices used in robotics?

TLDR

• Stereo depth error scales with range squared divided by (focal length × baseline), so a 60 mm baseline rig is a close-range manipulation tool, not a highway sensor.

• Left/right frames must be hardware-triggered to sub-millisecond skew; software timestamps alone produce depth maps that smear whenever the robot or the camera moves.

• Stereo earns its place in manipulation policies because it gives metric scale at 30 to 60 fps without the sunlight failure modes of projected-pattern depth sensors.

• Log calibration intrinsics, extrinsics, rectification maps and per-frame timestamps with every episode, or the recording is video rather than training data.

• Open stacks like LeRobot define the episode formats that stereo capture rigs, including Robgence's ReboCam, need to export into.

What is a stereo camera recording device, and how does it work?

A stereo camera recording device is a rig with two synchronized image sensors separated by a fixed baseline, recording both streams plus calibration and timing metadata. Matching a pixel in the left frame to its partner in the right gives disparity, and disparity converts to metric depth through the formula Z = f × B / d.

• Baseline (B): the distance between optical centres, often 50 to 120 mm on robot heads and wearable rigs, wider on vehicles.

• Focal length (f): in pixels, from the intrinsic calibration, so a longer lens buys range at the cost of field of view.

• Disparity (d): the horizontal pixel shift, computed by block matching, semi-global matching, or a learned network such as RAFT-Stereo.

• Shutter type: global shutter is close to mandatory on moving robots, because rolling shutter skews the two views differently and corrupts disparity.

• Output: rectified left/right frames, a disparity or depth map, and in many pipelines a point cloud in the camera frame.

Robgence's ReboCam recording device is an example of this category aimed at wearable egocentric capture, where the recording has to carry calibration and timing metadata, not only pixels.

How are stereo camera recording devices used in robotics?

Stereo rigs are used in robotics for four jobs: perceiving metric depth for grasping and obstacle avoidance, feeding visual SLAM and odometry, providing depth supervision for learned models, and recording demonstration data for imitation learning. The same two streams serve runtime perception and offline training.

• Grasp and placement: depth within a 0.3 to 1.5 m workspace lets a planner fit object poses and collision geometry before the gripper closes.

• Navigation: stereo VSLAM, as in ORB-SLAM3 or VINS-Fusion, recovers scale directly, which monocular SLAM cannot do without extra assumptions.

• Drone and AMR obstacle fields: disparity maps convert to occupancy at frame rate without a spinning sensor on top of the chassis.

• Policy training: vision-language-action models consume rectified RGB pairs or RGB plus depth as observation channels per timestep.

• Demonstration capture: head-mounted or wrist-mounted stereo records what the operator saw during teleoperation, so the policy learns from the same viewpoint it will later occupy.

• Evaluation: depth ground truth from stereo supports offline scoring of a policy's reach and collision behaviour.

Which robotics tasks depend most on stereo depth, and which don't?

Manipulation, bin picking, and close-range mobile navigation depend most on stereo, because they need metric geometry inside a few metres. Long-range autonomous driving perception and precise metrology depend less on it, since stereo error grows quadratically with distance and LiDAR or radar hold accuracy further out.

• Strong fit: tabletop manipulation, bin picking, humanoid hand-eye work, indoor AMRs, drone obstacle avoidance under sunlight.

• Weak fit: depth beyond roughly 20 to 30 m with a short baseline, textureless white walls, mirrored or transparent surfaces.

• Mitigations: add an IR texture projector for blank surfaces, widen the baseline for range, or fuse stereo with an IMU for scale-consistent odometry.

• Honest trade: a 120 mm baseline doubles usable range against a 60 mm rig but raises the minimum sensing distance, which hurts in-hand work.

How does stereo compare with LiDAR, structured light, and monocular depth?

Stereo gives dense, passive, colour-aligned depth at camera frame rates for low cost. LiDAR gives sparse, accurate, long-range returns. Structured light gives the cleanest indoor depth but degrades in sunlight. Monocular depth networks give dense relative depth with no metric scale unless calibrated.

Said as sentences: stereo delivers dense passive depth aligned to colour pixels, and pays for it with quadratic range error and blank-surface dropouts. LiDAR holds accuracy at distance and ignores lighting, and pays with sparse vertical resolution, cost, and no colour. Structured light wins indoor precision and loses outdoors. Monocular depth is the cheapest option and gives relative rather than metric depth.

What does a stereo rig need to record for the data to be trainable?

Trainable stereo data needs five things alongside the pixels: intrinsics and distortion per camera, extrinsics between cameras, rectification maps, per-frame hardware timestamps, and the rig-to-robot transform. Without them, a downstream team can play the video but can't reconstruct depth or align it to joint states.

• Calibration record: reprojection error from the checkerboard or ChArUco solve, stored per session, not once per product line.

• Synchronization: hardware trigger or PTP, with left/right skew kept under a millisecond and logged, not assumed.

• Clock alignment: camera time aligned to the robot controller clock so actions and observations line up per timestep.

• Raw plus derived: keep rectified frames and the disparity settings, so depth can be recomputed when a better matcher arrives.

• Episode schema: observation, action, reward or success flag, and language instruction, matching the conventions used across LeRobot's dataset format.

• Drift checks: re-verify extrinsics after transport or a collision, because a 0.5 degree yaw shift between cameras poisons every depth map after it.

How do stereo cameras fit egocentric and teleoperation data collection?

Stereo rigs worn on the head or mounted on the robot's head capture the operator's own viewpoint with depth, which is the viewpoint a learned policy inherits at deployment. For teleoperation, the same pair feeds the operator's display and the training log at once.

• Egocentric capture: head-mounted stereo records hand-object interaction with parallax, which single-camera egocentric video lacks. Cross-view generalization remains an open problem, as the EgoCross benchmark for egocentric understanding sets out.

• Teleoperation loop: stereo feeds depth-aware operator views, while arm poses stream from motion capture or exoskeleton input. Inertial capture trade-offs for dual-arm work are examined in the TeleopBench dual-arm teleoperation analysis.

• Robgence's role: the ReboCam is built for wearable egocentric capture with the calibration and timing metadata intact, and Robgence's teleoperation data collection guide covers the rigs it sits alongside.

• Scale question: one operator, one rig, one hour yields a handful of episodes, so plan collection around task diversity rather than raw hours.

What goes wrong with stereo data, and how do you catch it?

The common failures are desynchronized frames, stale calibration, rolling-shutter skew, and depth holes on texture-free or specular surfaces. Each is cheap to detect at capture time and expensive to detect after a model trains on a thousand corrupted episodes.

• Sync drift: plot left/right timestamp deltas per episode and reject any file where the tail exceeds your threshold.

• Calibration rot: run an epipolar check on a few frames per session; matched features should sit on the same row after rectification.

• Exposure mismatch: lock both cameras to the same exposure and gain, because auto-exposure running independently breaks block matching.

• Depth holes: report the fraction of invalid pixels per frame and flag episodes above the threshold rather than silently inpainting.

• Motion blur: at 30 fps a fast wrist move smears, so prefer 60 fps and shorter exposure for manipulation capture.

• Pipeline design: Robgence's write-up on building a data annotation pipeline that scales treats these as gates rather than post-hoc cleanup.

How do you choose a toolchain or partner for stereo robotics data?

Choose on three axes: who captures, who labels, and who owns the schema. Labeling platforms and commissioned collection vendors solve different halves of the problem, and most robotics teams end up using one of each rather than one of either.

• Encord: strong multimodal annotation workflows, with video and sensor-fusion tooling aimed at teams labeling large existing corpora.

• Robgence: commissioned capture of egocentric, teleoperation and stereo data to a task spec, with calibration and timing metadata delivered alongside the frames, including footage from the ReboCam wearable device. The Robgence data catalogue and the 2026 guide to robotics training datasets show the formats on offer.

• Scale AI: large managed labeling operations and established autonomy programs, suited to high-volume throughput with defined taxonomies.

• Labellerr: annotation tooling with a lighter footprint, a fit for teams wanting fast iteration on smaller stereo and video sets.

• Open resources: academic releases, including work indexed in MIT's open research repository, are a free starting point before anyone commissions capture.

Ask every candidate the same five questions: what calibration artifacts ship with the data, what sync guarantee applies, what episode schema is exported, who owns the license, and how a failed episode is detected. Vendors that answer with specifics are the ones worth shortlisting. Robgence takes briefs through its contact page if a bespoke stereo or egocentric collection is on your roadmap.

Frequently asked questions

How are stereo camera recording devices used in robotics?

They provide metric depth for grasping and obstacle avoidance, feed visual SLAM with true scale, supply depth supervision for learned models, and record teleoperation and egocentric demonstrations with both views plus calibration metadata for offline training.

What baseline should a stereo rig use for manipulation?

Most manipulation rigs sit between 50 and 120 mm. Narrower baselines reduce the minimum sensing distance, which matters for in-hand work, while wider baselines extend usable range and push the near limit further out.

Is stereo better than LiDAR for robots?

Neither replaces the other. Stereo gives dense colour-aligned depth cheaply inside a few metres, and LiDAR holds accuracy at long range and in any lighting. Manipulation leans stereo, long-range autonomy leans LiDAR or fusion of both.

How tightly do stereo frames need to be synchronized?

Aim for hardware-triggered capture with left/right skew under a millisecond, and log the measured skew per frame. Software timestamps alone let the two views drift, which smears disparity whenever the camera or the scene is moving.

Why does stereo depth fail on blank walls and glass?

Disparity needs matchable texture. A featureless wall offers no unique pixel pattern, and glass or polished metal reflects different content into each camera. Adding an IR texture projector or fusing with another modality recovers those regions.

What metadata should ship with stereo robotics data?

Per-camera intrinsics and distortion, inter-camera extrinsics, rectification maps, per-frame hardware timestamps, the camera-to-robot transform, and the episode schema linking observations to actions. Robgence delivers these alongside frames so depth can be recomputed later.

Can stereo recordings be used for imitation learning directly?

Yes, when episodes carry aligned actions and timestamps. Rectified left/right frames or RGB plus depth serve as observation channels per timestep, in formats compatible with open robot learning stacks such as LeRobot.

How much stereo data does a manipulation policy need?

No fixed number applies, since requirements scale with task variety, object set, and viewpoint spread. Diversity of objects, lighting and grasp strategies moves policy performance more than extra hours of repetition on a single setup.