Annotation is where raw sensor recordings become training data. In robotics, this is not the same problem as labeling images for object detection or drawing bounding boxes for autonomous driving. Robotics annotation involves temporal segmentation of manipulation episodes that span tens of seconds, action labeling across multi-phase task sequences, grasp type classification that requires understanding 3D contact geometry, and contact event marking in force-torque data that is invisible in RGB streams. The annotation schema directly determines what a behavior cloning or diffusion policy model can learn.
Get it wrong and your model learns noise — action boundaries that are off by ten frames, grasp labels that conflate power grasps with precision pinches, success flags on episodes where the object was dropped and re-grasped. Get it right and you have a competitive advantage that is nearly impossible to replicate, because high-quality annotated robot training data requires domain expertise, careful tooling, and systematic quality assurance that cannot be shortcut by throwing more annotators at the problem.
This guide covers the annotation types that matter for robot learning, the QA pipeline that catches real errors, and the operational practices that maintain quality at scale. Everything here is based on what works in production data pipelines, not theoretical recommendations.
Why Robotics Annotation Differs from Computer Vision Annotation
Computer vision annotation operates on static images or short video clips. An annotator looks at a frame, draws a bounding box around a cat, and moves on. The temporal dimension, when it exists, is limited to tracking objects across consecutive frames. The semantic content is visual: shape, color, texture, position in 2D space.
Robotics annotation is fundamentally different in four ways. First, it operates on temporal structure. A single manipulation episode is not a collection of independent frames — it is a sequence of phases with causal relationships. The approach phase determines the grasp phase; the grasp phase determines the lift phase. Annotators must understand this structure to place boundary frames correctly. An annotator who treats each frame independently will produce boundaries that are temporally incoherent.
Second, robotics annotation requires multi-stream synchronization. A single episode includes RGB video, depth images, joint position trajectories, end-effector pose, and force-torque readings — all at different frame rates. An annotator marking a grasp event needs to identify the frame in the RGB stream where fingers close, verify that the corresponding force-torque data shows a contact force increase, and check that the joint position data shows the gripper closing. Annotation on a single stream misses events that are only visible in other modalities.
Third, the semantics are action-based rather than appearance-based. In computer vision, a bounding box around a screwdriver tells the model what a screwdriver looks like. In robotics, the annotation must capture what the operator is doing with the screwdriver — approaching it (from which direction), grasping it (with which grasp type), inserting it (with what force profile), and turning it (with what torque sequence). These are physical actions with dynamics that cannot be inferred from static appearance.
Fourth, many critical events are invisible in RGB data. Contact events — the moment a fingertip touches an object surface, slip during a grasp, the seating of a connector during insertion — produce barely perceptible visual changes but generate clear signals in force-torque data. An annotator working only from video will miss these events entirely, or place their timing incorrectly. This is why robotics annotation must be multimodal: the annotator needs synchronized access to all sensor streams simultaneously.
Consider a concrete example: labeling the grasp type for a bin-picking episode. In a 2D image, a hand closing around a bolt might look like either a precision pinch or a lateral grasp, depending on the camera angle. But the force-torque data shows the contact force distribution — a precision pinch produces opposing forces along the fingertip axis, while a lateral grasp produces force primarily along one finger's lateral surface. Without the force-torque context, the grasp label is a guess. With it, the label is a measurement.
Essential Annotation Types for Robot Training
The annotation types required for a robotics dataset depend on the learning algorithm, but six types cover the needs of most current approaches including imitation learning, behavior cloning, and diffusion policy training.
Temporal segmentation divides each episode into phases with precise start and end frames. For a pick-and-place task, the standard phases are: idle/home, approach, pre-grasp alignment, grasp closure, lift, transport, descent, placement, release, and retract. Each phase boundary should be accurate to ±2 frames at the recording frame rate. For a 30fps recording, this means ±66ms precision. The boundary criteria must be explicit: "grasp closure begins at the frame where the gripper command transitions from open to closing" rather than "when the fingers start to close," which is visually ambiguous. Temporal segmentation enables phase-conditioned policies that can learn different control strategies for different task phases — approach requires coarse visual servoing, while insertion requires fine force-controlled motion.
Action labels describe what the operator is doing at each timestep or phase. These are drawn from a predefined taxonomy that must be finalized before annotation begins. The taxonomy should be hierarchical: top-level actions (pick, place, inspect, adjust) with sub-actions (approach-from-top, approach-from-side, approach-from-angle under pick). Labels must be mutually exclusive within a hierarchy level and collectively exhaustive — every frame in the episode belongs to exactly one action at each level.
Grasp type classification labels the physical grasp strategy used in each manipulation event. Standard categories include: power grasp (full-hand wrap), precision pinch (thumb and index fingertips), tripod grasp (thumb, index, and middle fingertips), lateral pinch (thumb pad against index lateral surface), and key grasp (thumb pad against curled index finger). For datasets collected through human-in-the-loop data collection, where human demonstrators perform grasps, the grasp type distribution reflects natural human strategies and provides rich supervision for grasp planning models.
Object identity and state tracking assigns a unique identifier to each object instance and tracks its state across frames and episodes. State includes pose (position and orientation), condition (intact, damaged, deformed), and relevant physical properties (weight class, surface finish). Object tracking must persist across episodes when the same physical objects are reused, enabling the model to learn object-specific manipulation strategies.
Success/failure classification with failure mode coding labels each episode outcome. Success criteria must match the protocol definition exactly. Failure modes should be categorized: missed grasp (no contact), slip (contact lost during transport), collision (unintended contact with non-target objects or environment), incorrect placement (outside tolerance zone), and incomplete task (task abandoned or timed out). Failure episodes are training data — they provide negative examples that help the model learn what not to do.
Contact event marking identifies specific timesteps where physically significant contact events occur: first contact with the target object, grasp establishment (when force exceeds a stability threshold), force peaks during manipulation, slip events (detected as sudden force drops or tangential force spikes), and placement contact. These events are primarily identified from force-torque sensor data, which is why multi-stream annotation access is essential.
Annotation Tools: What Exists and What's Missing
The robotics community has largely been adapting computer vision annotation tools for a problem they were not designed to solve. CVAT, Label Studio, and VGG VIA are capable tools for image and video annotation. They support bounding boxes, segmentation masks, keypoints, and temporal video segments. But they lack the specific features that robotics annotation demands.
The core limitation is synchronized multi-stream playback. A robotics annotation interface needs to display the RGB video, the depth visualization, the force-torque time series, and the joint position trajectory simultaneously, with a shared timeline cursor. When the annotator scrubs to a specific frame, all streams must update together. Existing tools display one video at a time; force-torque and joint data require custom visualization plugins that are rarely available.
The second limitation is temporal labeling at the timeline level. CVAT supports temporal segments in video, but the interface is designed for tracking objects across frames, not for labeling task phases. An ideal robotics annotation interface has a timeline at the bottom of the screen — similar to a video editor — where the annotator drags to create phase segments, assigns labels from a dropdown taxonomy, and adjusts boundaries with frame-level precision. The timeline should show waveform overlays for force-torque data so the annotator can identify contact events without switching views.
The third limitation is 3D visualization. Grasp type annotation benefits enormously from a point cloud viewer that shows the hand-object contact surface. A 2D RGB image shows the grasp from one viewpoint; a point cloud shows the full 3D contact geometry. Some annotations — distinguishing a tripod grasp from a three-finger wrap, for instance — are nearly impossible from a single 2D view but obvious in 3D.
Some teams build custom annotation tools internally. This works for specific projects but creates maintenance burden and limits reuse. The ideal state is a robotics-native annotation platform that supports MCAP and HDF5 import, multi-stream synchronized playback, configurable label taxonomies, and export to training formats including RLDS and LeRobot. This tooling gap is one reason robotics data collection platforms that include annotation capabilities provide significant value — the tooling is purpose-built for the data type.
Building a QA Pipeline That Catches Real Errors
Quality assurance for robotics data operates at three tiers: automated, manual, and statistical. Each tier catches different error types, and all three are necessary for production-quality datasets.
The automated tier runs on every episode immediately after collection and annotation. It includes: sensor dropout detection (any stream with gaps exceeding one frame interval is flagged), calibration validity verification (ArUco marker reprojection error computed at episode start and end, flagged if exceeding 3 pixels), frame synchronization check (maximum timestamp offset between any two streams, flagged if exceeding 5ms), label format validation (every required annotation field is populated, values fall within the defined taxonomy, temporal segments cover the full episode without gaps or overlaps), and metadata completeness check (calibration files, operator ID, environment descriptor, protocol version all present). Automated checks should run as part of the ingestion pipeline — data that fails any check enters a quarantine queue, not the main dataset.
The manual tier involves human review of sampled episodes. Protocol adherence review: a trained reviewer watches a random sample (minimum 10% of episodes per batch) and scores adherence to the collection protocol — correct start/end states, proper edge case handling, appropriate demonstration speed. Annotation spot-checks: the reviewer compares the annotated phase boundaries and labels against their own independent judgment, flagging discrepancies exceeding the defined tolerance. Edge case flagging: episodes that involve unusual situations (novel object orientations, recovery from near-failures, workspace boundary interactions) are flagged for additional review because they are the most likely to have incorrect annotations.
The statistical tier analyzes aggregate patterns across batches. Label distribution analysis compares the actual distribution of action labels, grasp types, and success/failure ratios against the protocol-defined targets. Significant deviations indicate collection bias or annotation drift. Inter-annotator agreement, measured by Cohen's kappa for categorical labels and mean absolute frame difference for temporal boundaries, should be computed on the overlap set (episodes annotated by multiple annotators). Kappa below 0.75 on action labels or mean boundary difference above 5 frames indicates the taxonomy needs clarification. Operator consistency metrics track per-operator success rates, episode durations, and approach angle distributions to detect bias introduced by individual operator habits.
The output of the QA pipeline is a per-episode quality score that aggregates results across all three tiers. Episodes below the acceptance threshold are routed for re-collection or re-annotation. The QA results feed back into protocol refinement and annotator training — they are not just a filter but a continuous improvement mechanism.
Annotator Training and Calibration
The quality of robotics annotations is directly proportional to the domain expertise of the annotators. A generic crowd worker can draw accurate bounding boxes on images because the task requires only visual pattern matching. Robotics annotation requires understanding what a grasp type means physically, why a particular approach angle matters for task success, and how force-torque data relates to contact events that are barely visible in video. This expertise must be built through structured training.
An annotator training protocol should include three phases. The orientation phase covers the task domain: what the robot is doing, why each manipulation phase matters, how different grasp types affect task outcomes, and what the sensor streams represent. For warehouse robotics data, this means understanding warehouse pick-and-place operations — object types, bin layouts, packing constraints. For manufacturing data, it means understanding part tolerances, assembly sequences, and quality requirements. This domain knowledge is what enables annotators to make informed judgments on ambiguous cases.
The calibration phase uses a shared set of benchmark episodes that have been annotated by expert annotators. New annotators independently annotate these episodes, and their labels are compared against the expert reference. Discrepancies are reviewed one-on-one: why did you place the grasp boundary at frame 142 instead of frame 137? What made you choose lateral grasp instead of precision pinch? These discussions build a shared understanding of the taxonomy and boundary criteria. Annotators who cannot achieve inter-annotator agreement above 0.8 after calibration should receive additional training before annotating production data.
The production phase begins with close supervision. For the first batch, every annotated episode is reviewed by a lead annotator. Feedback is immediate and specific: "The approach phase ends when the end-effector velocity drops below 5cm/s, not when it starts to decelerate." As consistency improves, review frequency decreases to the standard sampling rate. But ongoing spot-checks continue for the lifetime of the annotation project.
Specialization matters. An annotator who has spent three months labeling manufacturing assembly tasks will produce higher-quality labels on similar tasks than a generalist who switches between domains. When possible, assign annotators to task domains and let them build deep expertise. The investment in annotator training pays dividends across the entire dataset — a well-trained annotator working at moderate speed produces more value than three untrained annotators working quickly.
Scaling Annotation Without Losing Quality
Scaling annotation is primarily an organizational challenge. The technical work — building tools, defining taxonomies, computing agreement metrics — is necessary but not sufficient. The harder problem is maintaining consistent quality as the number of annotators, episodes, and task domains grows.
Annotation guidelines as living documents are the foundation. The initial guidelines will be incomplete — edge cases that were not anticipated will surface during production annotation. When an annotator encounters an ambiguous case and asks a question, the answer should be documented in the guidelines within 24 hours, with a specific example and clear ruling. Every annotator must work from the current version of the guidelines. Version the guidelines and track which version was in effect when each episode was annotated. If a guideline change affects the interpretation of previously annotated episodes, those episodes must be flagged for re-review.
Lead annotator review provides the quality backbone. For every batch of annotations, a lead annotator — someone with deep domain expertise and extensive calibration experience — reviews a sample. The sample size depends on the team's current agreement metrics: 20% when onboarding new annotators, 10% at steady state, 5% for highly experienced teams. The lead annotator has authority to reject annotations and return episodes for re-annotation. Their feedback goes directly to the annotator and is recorded for trend analysis.
Model-assisted pre-annotation accelerates the process without sacrificing quality. A trained model (even an imperfect one) generates preliminary temporal segments and action labels. Human annotators then correct these pre-annotations rather than starting from scratch. This is faster — correction takes roughly 40% of the time of annotation from scratch — and can improve consistency because annotators are adjusting from a common starting point rather than making independent judgments. However, pre-annotation introduces a risk of anchoring bias: annotators may accept incorrect pre-annotations that they would not have produced independently. Counter this by periodically including episodes without pre-annotation in the review stream and comparing annotation quality between pre-annotated and clean episodes.
Batch-level quality gates prevent delivery of substandard data. Every batch must pass automated QA, achieve minimum inter-annotator agreement on the overlap sample, and receive lead annotator approval before it enters the delivered dataset. Batches that fail are held for re-annotation, not patched. The cost of re-annotation is lower than the cost of training a model on noisy labels and spending weeks debugging why performance plateaued.
Annotation quality determines model quality — there is no shortcut. Humaid's annotation pipeline is built specifically for robotics data: synchronized multi-stream playback, temporal segmentation with frame-level precision, action labeling from task-specific taxonomies, and a three-tier QA pipeline that catches errors before they reach your training pipeline. If you need robot training data with annotations you can trust, the annotation infrastructure matters as much as the collection infrastructure.