Egocentric data collection
First-person video and sensor data captured from the operator's perspective — the exact viewpoint a robot needs to learn manipulation, navigation, and object interaction in real environments.
Definition
Egocentric data collection records the world from the actor's own point of view. In robotics, this means mounting cameras and sensors on a human operator — typically on the head, chest, or wrist — so the captured footage mirrors the perspective a robot camera would have during deployment.
The resulting datasets include first-person RGB video, depth maps, hand-object interactions from the manipulator's viewpoint, and gaze direction. This is fundamentally different from third-person observation, where a static camera watches a scene from the side. Egocentric vision captures what matters for action: what is in reach, what the hands are doing, and how the scene changes as the operator moves through it.
For visuomotor policy learning — where a robot maps visual input directly to motor commands — egocentric data provides the exact input-output correspondence the model needs. The observation space during training matches the observation space during deployment.
Why it matters
Imitation learning algorithms learn a mapping from observations to actions. When training data is recorded from the same viewpoint the robot will use at inference time, there is no domain shift between training and deployment. Egocentric data eliminates the perspective transform that third-person data requires.
First-person cameras capture fine-grained details of grasping, tool use, and manipulation that are occluded in third-person views. Finger positions relative to objects, contact surfaces, and wrist orientation are all visible in ego video — information that is essential for dexterous manipulation policies.
Where a human operator looks during a task encodes implicit planning and attention. Head-mounted cameras capture this gaze signal, which maps directly to where a robot should direct its visual attention. This is a training signal that third-person cameras fundamentally cannot provide.
Ego vs. exo
For human-in-the-loop data collection workflows that target imitation learning and behavior cloning, egocentric video is the default capture modality because it removes the largest source of distribution shift between demonstration and deployment.
Capture hardware
Stereo RGB-D cameras mounted at eye level on a lightweight headband or helmet. Captures the operator's natural field of view with synchronized color and depth at 30–60 fps. Ideal for tasks where gaze direction and scene scanning are important signals — warehouse navigation, inspection, multi-step assembly.
Compact cameras strapped to the forearm or wrist, positioned to match a robot's wrist-mounted eye-in-hand camera. Captures close-range object interaction, grasp approach, and tool alignment. Best for fine manipulation tasks like circuit board assembly, food plating, or small-part insertion.
Multi-camera arrays on the torso provide a wider field of view that approximates a mobile robot's forward-facing sensor suite. Paired with IMU data for body motion tracking. Suitable for locomotion tasks, pushing/pulling, and tasks requiring full upper-body coordination.
What gets captured
Synchronized color and depth streams from the operator's viewpoint. Resolution up to 1280x720 at 30–60 fps. Depth maps provide metric distance for every pixel, enabling 3D scene reconstruction and spatial reasoning without additional LiDAR.
21-joint hand skeleton extraction at 30+ fps, synchronized with ego video. Tracks finger articulation, grip type, contact timing, and release events. Combined with wrist IMU data for 6-DoF hand motion. The primary signal for training dexterous manipulation policies.
Per-pixel metric depth from stereo or structured-light sensors. Enables point cloud generation and 3D object localization.
Visual-inertial odometry tracks the camera's position and orientation in space at all times, providing ego-motion ground truth.
Frame-level labels: grasp start, object contact, lift, transport, place, release. Temporal segmentation into discrete action primitives.
Bounding boxes, instance masks, and 6-DoF pose annotations for all task-relevant objects in each frame.
channels on one clock
3D keypoints per hand
IMUs at ~200 Hz
stereo FOV, 0.2–20 m depth
Imitation learning
Imitation learning trains a robot to replicate demonstrated behavior. The tighter the correspondence between the demonstration data and the robot's own sensory input, the better the learned policy generalizes. Egocentric data achieves this correspondence directly — the training images look like what the robot will see.
In behavior cloning, the policy maps observation to action at each timestep. With ego video, the observation is a first-person image plus depth, and the action is the recorded hand or arm motion. There is no viewpoint transformation, no camera calibration mismatch, and no occlusion of the end effector. This reduces compounding error, the primary failure mode of behavior cloning at deployment time.
For diffusion policy and transformer-based action models, large-scale egocentric datasets provide the diversity needed for generalization. Thousands of hours of first-person demonstrations across environments, objects, and operators give these models the distributional coverage to handle novel situations at test time.
Use cases
Egocentric video captures the spatial complexity of commercial kitchens — cluttered countertops, deformable ingredients, liquid handling. Head-mounted rigs record the operator's gaze as they sequence multi-step recipes, producing training data for food-prep robots that must handle soft objects and precise pouring.
First-person data from warehouse operators captures how experienced workers scan shelves, select grasp strategies based on object shape, and pack items into containers. Ego video paired with hand-pose tracking produces the observation-action pairs needed for autonomous pick-and-place in logistics.
Room cleaning, table bussing, linen folding, cart navigation through crowded hallways. These tasks require whole-body coordination and spatial awareness that egocentric capture preserves. Chest and head cameras record the full interaction context that static cameras would miss.
Our approach
Operators learn the task before recording begins. They perform it the way a skilled worker would, not as someone encountering it for the first time. This produces demonstrations with consistent quality, natural variability, and correct task completion — the attributes that make imitation learning data actually usable.
We configure capture rigs to approximate the specific robot platform's camera placement — eye-in-hand, head-mounted, or chest-forward. This minimizes the visual domain gap between demonstration and deployment, improving policy transfer without additional fine-tuning.
All data is collected in the actual environments where robots will operate — production facilities, commercial kitchens, hotel rooms, warehouses. The lighting, objects, spatial layout, and background clutter are real. This eliminates the distribution shift that lab-collected data introduces.
Data Explorer
Egocentric datasets collected by Humaid are available in the robotics data explorer. Browse first-person video recordings alongside synchronized third-view cameras, hand pose estimation overlays, 3D body tracking, and temporal action segmentation — with frame-level playback controls for precise inspection.
The explorer currently hosts egocentric datasets across household tasks (kitchen, laundry, folding, office work) and manufacturing assembly — each with 60+ metadata properties per sequence and 11 downloadable file types per recording. Explore egocentric datasets (opens in a new tab).
FAQ
Get started
Tell us your target task, robot platform, and environment. We will design a capture protocol, deploy operators, and deliver calibrated egocentric datasets ready for your training pipeline.