How to Collect Real-World Robotics Data for Training AI Models

The sim-to-real gap is not a single gap — it is a collection of dozens of small discrepancies that compound multiplicatively. Each one might seem minor in isolation. Together, they make simulation-only training insufficient for any system that must operate reliably in uncontrolled environments.

Material properties are the first and most persistent gap. A simulated steel bearing has uniform friction, predictable reflectance, and zero surface contamination. A real steel bearing on a factory floor has machining oil residue that changes its grip friction by 30-50%, surface scratches that alter its visual appearance under directional lighting, and manufacturing tolerances that make every unit slightly different in mass distribution. A behavior cloning policy trained exclusively on simulated grasps of that bearing will fail when the real bearing slips in the gripper because the friction coefficient was wrong by a factor of two.

Consider a specific example: bin picking from a parts tray in an automotive assembly line. The tray contains fifty stamped metal brackets, each with slight burrs from the stamping process. Some brackets are nested together. Oil from the stamping press coats every surface. The overhead fluorescent lights create specular reflections that move as the robot arm occludes different light sources. None of this exists in simulation. Domain randomization can approximate some of it — randomizing friction coefficients, adding noise to visual textures — but it cannot replicate the correlated structure of real-world variation. Oil residue is not random noise; it pools in concavities and sheets off convex surfaces in patterns determined by the part geometry and the stamping process.

Warehouse environments present a different class of real-world variation. A single fulfillment center may handle tens of thousands of SKUs. Each product has a different shape, weight, surface texture, deformability, and packaging material. Shrink-wrapped bundles deform under grip force. Polybag items shift their center of mass unpredictably. Cardboard boxes with worn edges look different from fresh ones. Barcode labels peel, fade, and wrinkle. A model trained on clean CAD renders of five hundred products will encounter visual and physical variation on its first day in production that exceeds anything simulation could have generated.

Contact dynamics represent perhaps the hardest gap to close. When a robot inserts a USB connector into a port, the contact forces follow a complex profile determined by the connector geometry, the port spring mechanism, alignment error, and insertion speed. Simulating this accurately requires a contact model that most physics engines do not implement — and even those that do require material parameters that must be measured empirically for each specific connector-port pair. Collecting real-world demonstrations of the insertion captures the ground-truth force profile that a teleoperation data collection setup can record at 500 Hz through a wrist-mounted force-torque sensor. No simulation parameter sweep can substitute for this.

There are three primary strategies for collecting real-world robotics data, and each produces a fundamentally different type of dataset with different downstream uses.

Strategy 1: Egocentric human demonstrations. A trained operator wears a sensor rig — typically head-mounted or wrist-mounted RGB-D cameras, hand pose tracking gloves, and an IMU — and performs the target task from a first-person perspective. The resulting data captures what a robot would see if it were performing the task, along with the human hand trajectory and grasp strategy. This approach is fast to set up, requires no robot hardware during collection, and produces large volumes of demonstration data quickly. It is ideal for tasks where the human hand and a robot gripper share similar approach strategies — picking objects from shelves, clearing a cluttered countertop, sorting items into bins. The limitation is the embodiment gap: human hand kinematics do not map directly to most robot end-effectors, so the action labels require retargeting before they can train a behavior cloning policy. Egocentric data collection for robotics works best when the primary training signal is the visual observation trajectory rather than the exact motor commands.

Strategy 2: Teleoperation through the robot. An operator controls the target robot directly using a leader-follower arm, VR controller, or SpaceMouse. The data is recorded in the robot's native action space — joint positions, end-effector poses, gripper states — with no retargeting required. This is the gold standard for training behavior cloning and diffusion policies because every timestep contains a matched observation-action pair in the deployment configuration. The trade-off is throughput: teleoperation is slower than human demonstration, requires the target robot hardware, and demands operator training on the specific control interface. For contact-rich tasks like inserting a USB connector, tightening a screw, or assembling snap-fit components, teleoperation is the only collection strategy that captures the correct force profiles.

Strategy 3: Autonomous exploration with human correction. The robot executes a partially trained policy autonomously, and a human operator intervenes when the robot fails or deviates from the desired behavior. The operator's corrections are recorded as additional training data. This approach is most useful for fine-tuning an already-capable policy — correcting failure modes, extending the operational distribution, and handling edge cases that initial training data did not cover. It requires a functional base policy and a real-time intervention mechanism, making it inappropriate for initial data collection but valuable for iterative improvement.

Most production programs use a combination of all three. Egocentric demonstrations provide the initial visual priors and task structure. Teleoperation produces the precise action-labeled data for policy training. And autonomous exploration with correction closes the gap between initial deployment and production reliability.

The sensor stack determines the upper bound of what your collected data can teach a model. An incorrectly configured or under-specified sensor setup produces data that looks complete but lacks the information a policy needs to generalize. Getting this right requires understanding both the task requirements and the failure modes of consumer-grade hardware in production environments.

Cameras. For wrist-mounted perspectives, the Intel RealSense D405 is a strong choice: it has a close-range depth mode (minimum 7 cm) that captures the near-field geometry around the gripper during grasp approach and contact. Its compact form factor (42 mm width) allows mounting without significant occlusion of the workspace. For external workspace cameras, the RealSense D435 provides a wider field of view and effective depth sensing from 30 cm to 3 meters. Most setups use one wrist camera and two external cameras positioned to minimize occlusion of the manipulation zone.

Consumer-grade cameras — webcams, phone cameras, GoPros — fail in production environments for specific reasons. They lack hardware-synchronized depth, produce rolling shutter artifacts during fast arm motion, have auto-exposure algorithms that hunt under industrial lighting, and provide no intrinsic calibration parameters. The resulting data has variable focal lengths between frames, inconsistent depth alignment, and motion blur during the critical contact moments that matter most for training.

Force-torque sensing. For contact-rich tasks — assembly, insertion, polishing, deformable object manipulation — a 6-axis force-torque sensor mounted at the robot wrist is essential. The ATI Nano25 or OnRobot HEX-E are common choices, sampling at 1 kHz or higher. Force data captures contact events that cameras cannot see: the moment a grasped object begins to slip, the insertion force profile of a connector seating into a port, the compliance required when pressing a gasket into a groove. Without force-torque data, a policy trained for connector insertion has no feedback signal for detecting successful contact versus jamming.

IMU and motion tracking. For egocentric collection where the operator is moving freely, a 9-axis IMU provides gravity-aligned orientation data that disambiguates camera viewpoint from body pose. For teleoperation setups, the robot's joint encoders provide equivalent proprioceptive data. Mobile manipulation tasks — pushing a cart, opening a door, navigating around obstacles — benefit from an additional IMU on the robot base to capture whole-body dynamics.

Synchronization. Every sensor stream must share a common time base. The standard approach is PTP (Precision Time Protocol, IEEE 1588) for networked devices, providing sub-microsecond synchronization across cameras, force sensors, and robot controllers on the same network. For devices that lack PTP support, a hardware trigger line can synchronize capture events. Software timestamps — the system clock at the moment a frame is received — introduce variable latency of 1-20 ms depending on system load, which is unacceptable for correlating force events with visual observations during fast contact transitions.

Calibration. Extrinsic calibration (the spatial relationship between each sensor and the robot base frame) must be established at the start of each collection session and verified periodically. A calibration target — an ArUco board or checkerboard — is placed in the workspace and imaged from multiple viewpoints. The resulting transforms are stored as session metadata. Calibration drift of even 3-5 mm over the course of a multi-hour collection session can corrupt grasp pose labels and render point cloud fusion unreliable. Automated drift detection — comparing ArUco reprojection errors against a threshold at regular intervals — is a necessary part of any production robotics data collection platform.

The single most important principle in real-world data collection is: collect where you deploy. Data collected in a university lab — clean surfaces, controlled lighting, standardized objects — does not transfer to a manufacturing floor with oil stains, fluorescent flicker, and worn fixtures. The visual distribution, the physical properties, and the edge case frequency are all different. A model trained on lab data and deployed in a factory will encounter out-of-distribution inputs on its first hour of operation.

Environment selection means physically setting up collection in the target deployment site, or in an environment that replicates its key properties with high fidelity. If the robot will operate in a cold storage warehouse at 2°C, collect data at 2°C — camera behavior, material stiffness, and operator dexterity all change at low temperatures. If the robot will work next to a vibrating conveyor, collect data next to that conveyor — the vibration introduces motion blur and IMU noise that must be present in training data for the policy to handle it.

Protocol design translates the deployment task into a structured collection specification. A protocol defines: the task objective (what constitutes a successful episode), the initial state distribution (how objects are arranged at the start of each episode), the episode boundary conditions (what triggers the start and end of recording), the operator instructions (speed constraints, approach angle diversity requirements, error recovery procedures), and the success/failure criteria (precise conditions for labeling an episode as successful, failed, or ambiguous).

Task decomposition is critical for complex tasks. Clearing a cluttered countertop is not a single task — it is a sequence of subtasks: identify target object, plan approach, grasp, lift, transport, place in bin, return to home position. Each subtask has its own success criteria, failure modes, and variation requirements. A well-designed protocol specifies randomization at each stage: vary the number of objects on the counter (3-12), vary their positions and orientations, vary which objects are partially occluded, and require operators to vary their approach trajectories across episodes rather than repeating the same motion pattern.

Edge cases deserve explicit protocol coverage. What does the operator do when two objects are stuck together? When an object is too heavy for a single grasp attempt? When an object rolls off the counter during approach? These scenarios are not rare in production — they occur in 5-15% of real operations. If the training data contains no examples of recovery from these situations, the deployed policy will freeze or produce dangerous behavior when they occur. The protocol must specify how operators handle each edge case and ensure that edge case episodes are collected intentionally, not just when they happen to occur.

The quality of collected data is bounded by the quality of the operators who produce it. This is not a platitude — it is a measurable fact. In controlled studies, trained operators produce demonstrations with 40-60% less variance in approach trajectory, 2-3x fewer failed grasps, and significantly more diverse object orientations compared to untrained operators performing the same task. The resulting datasets train policies that generalize better because the demonstrations are consistent enough for the model to extract the underlying task structure, while being diverse enough to cover the operational distribution.

Operator selection and training. For manufacturing data collection, operators with domain experience — people who have worked on assembly lines, operated CNC machines, or performed manual quality inspection — produce fundamentally better demonstrations than operators recruited from general-purpose crowdsourcing platforms. They understand the materials, the tolerances, and the failure modes. A manufacturing operator instinctively varies their grasp approach based on part geometry; a crowdworker follows the written instruction literally and produces monotonous demonstrations that lack the implicit knowledge the model needs.

Training involves three phases: (1) task familiarization — operators practice the task without recording until they achieve consistent completion, (2) protocol training — operators learn the specific episode structure, randomization requirements, and edge case procedures, and (3) calibration validation — operators complete a standardized set of calibration episodes that are scored against reference demonstrations. Only operators who pass calibration validation contribute data to the training dataset.

Consistency metrics. During collection, operator consistency is tracked across multiple dimensions: task completion rate, episode duration distribution, approach trajectory diversity (measured by end-effector path variance across episodes), and protocol adherence (correct initial state randomization, correct episode boundary marking). Operators whose metrics deviate from the population norm receive real-time feedback and, if necessary, re-training.

Fatigue management. Teleoperation is physically and cognitively demanding. Operator performance degrades measurably after 60-90 minutes of continuous collection: task completion times increase, trajectory variance decreases (operators default to a single comfortable motion pattern), and error rates rise. Collection schedules must include mandatory breaks — typically 15 minutes per hour of active collection — and session duration limits. Fatigue-related quality degradation is invisible in individual episodes but becomes apparent in aggregate statistics, which is why continuous monitoring matters.

Real-time QA monitoring. During active collection, a QA pipeline runs in parallel: checking sensor calibration status, verifying frame synchronization, flagging episodes where the operator deviated from protocol, and tracking per-operator quality metrics. Episodes that fail automated QA checks are flagged immediately rather than discovered during post-processing, which allows operators to correct issues before they affect an entire session. Episode rejection criteria are explicit: any episode with a calibration drift above threshold, a sensor dropout exceeding 100 ms, or a protocol violation is rejected and must be re-collected.

Raw sensor recordings are not training data. The gap between what comes off the sensors and what a model training pipeline consumes is significant, and bridging it requires a structured transformation process.

Storage format. Raw recordings are stored in MCAP, which supports multi-stream recording with per-channel timestamps, schema-aware message serialization, and efficient random access. Each episode is a single MCAP file with a sidecar JSON metadata file containing the session context: operator ID, environment configuration, calibration parameters (intrinsic and extrinsic for each camera), protocol version, and episode-level labels (success/failure, task variant, edge case flags). MCAP files are immutable after recording — no in-place editing — which provides a clean audit trail from raw data through all downstream transformations.

Annotation. Raw episodes require temporal segmentation — marking the boundaries of each task phase (approach, contact, manipulation, transport, place, release) with frame-accurate timestamps across all synchronized streams. Action labels are applied at the phase level (grasp type, manipulation strategy) and at the timestep level (discrete action commands for behavior cloning). Object identity is tracked across episodes when the same physical objects appear in multiple collections. This annotation requires human-in-the-loop data collection expertise — automated annotation tools can propose boundaries, but human reviewers must verify them, especially at contact transitions where frame-level accuracy matters for policy training.

Format conversion. Annotated datasets are converted from MCAP to the formats that training pipelines consume. HDF5 is the most flexible — it supports arbitrary nested data structures and is compatible with any Python training codebase. Each episode becomes an HDF5 group containing datasets for each sensor stream, with attributes storing metadata and calibration. RLDS (Reinforcement Learning Datasets) provides a TensorFlow-native format with explicit episode structure, step-level observations and actions, and metadata at every level. It integrates directly with TensorFlow Datasets for distributed loading. LeRobot format is increasingly adopted by teams training diffusion policies and action-chunking transformers — it provides native compatibility with the LeRobot training framework including dataset visualization, replay, and statistics computation.

Documentation. Every delivered dataset includes: sensor specifications and calibration parameters, annotation schema with definitions for every label, episode counts broken down by task variant and success/failure, distribution statistics (trajectory lengths, force profiles, object categories), and known limitations or caveats (e.g., reduced depth quality on reflective surfaces, under-representation of certain edge cases).

Versioning. As collection campaigns produce additional batches, datasets are versioned with semantic versioning. Each version increment includes a changelog specifying what was added, what was re-annotated based on quality review, and what was removed during QA. Training experiments reference specific dataset versions, enabling reproducibility. This versioning infrastructure is essential for any team collecting manufacturing robotics data across multiple production sites over months-long campaigns — without it, tracing a model regression to a specific data batch is impossible.