Manufacturing Robotics Data Collection: Challenges and Solutions

The sim-to-real gap gets most of the attention in robotics research, but the lab-to-factory gap is equally severe and receives far less discussion. Teams that collect manipulation data in university labs or clean-room R&D facilities consistently discover that their models fail when deployed on production floors. The reasons are specific and measurable.

Object properties differ fundamentally. Lab datasets typically use 3D-printed parts, off-the-shelf items from YCB or similar benchmarks, or simplified proxies for real manufacturing components. A 3D-printed M8 bolt weighs differently than a cold-forged steel M8 bolt, has a different surface finish (layer lines vs. machined surfaces), different reflectance properties (matte PLA vs. oily zinc-plated steel), and different dimensional tolerances (±0.2mm for FDM printing vs. ±0.02mm for CNC machining). A model trained on the 3D-printed version learns visual features, grasp forces, and placement tolerances that do not transfer to the real part.

Lighting conditions are incomparable. Labs have controlled, consistent overhead lighting — typically diffuse LED panels providing even illumination. Factories have high-bay fluorescent or metal halide lights that flicker at line frequency (60Hz in North America, 50Hz in Europe), creating periodic intensity variation that affects camera exposure. Windows and loading dock doors create variable natural light that changes with time of day and weather. Adjacent machinery casts hard shadows that move as equipment operates. Welding arcs from nearby stations produce intense, intermittent illumination. An RGB model trained under uniform lab lighting will produce degraded feature representations under these conditions.

Surface contamination is the norm, not the exception. Manufacturing parts are covered in cutting oil, hydraulic fluid residue, and metal shavings. Workbenches have coolant puddles and swarf accumulation. These contaminants affect every sensor modality: oil films change surface reflectance and reduce depth sensor accuracy, metal shavings create unexpected contact points during grasping, and contaminated surfaces have different friction coefficients than clean surfaces. A bin-picking model trained on clean parts in a lab fails when confronted with stamped steel parts that have burrs, oil film, and residual cutting fluid.

Scale requirements differ by orders of magnitude. A lab demonstration might require 95% success over 20 trials. A manufacturing deployment requires 99.5% success over thousands of cycles per shift, with failure modes that do not damage parts, tooling, or adjacent equipment. The tail of the distribution — the 0.5% of cases that are most difficult — contains the exact scenarios that lab data fails to capture: parts with maximum tolerance stackup, unusual orientations from tumble feeding, and environmental transients.

Manufacturing encompasses hundreds of task types, but five categories account for the majority of current robotics data collection demand. Each has specific data requirements that shape the collection protocol.

Bin picking is the most common manufacturing robotics task. Parts arrive in bins, totes, or conveyors in random orientations and positions. The robot must identify individual parts, plan a collision-free grasp, extract the part from potential entanglement with neighboring parts, and present it at a known orientation for downstream processing. Data requirements: RGB-D from multiple viewpoints (wrist and overhead), force-torque for grasp quality assessment, varied part orientations (minimum 50 unique orientations per part type), entanglement scenarios (parts interlocked, stacked, bridging), and surface condition variation (oily, dry, corroded). Typical dataset size: 2000-5000 episodes per part type for robust performance.

Assembly and insertion tasks include peg-in-hole, snap fits, screw driving, and connector mating. These are contact-rich tasks where sub-millimeter precision is required. Data requirements: high-resolution depth data (Zivid or equivalent for initial alignment), high-rate force-torque (1000Hz minimum to capture insertion dynamics), joint torque for compliant insertion, and detailed temporal segmentation of contact phases — approach, search, chamfer contact, sliding insertion, and seating confirmation. Force-torque data is the primary training signal for the insertion phase; visual data is secondary once contact is established.

Weld inspection requires visual assessment of weld quality under challenging lighting conditions. Data requirements: high-resolution RGB with controlled lighting (often requiring dedicated ring lights or structured illumination to reveal surface defects), multiple viewing angles per weld (minimum three: top, 30° left, 30° right), labeled defect categories (porosity, undercut, overlap, spatter, crack, incomplete fusion), and reference images showing acceptable vs. rejectable welds for each quality standard. This task is vision-dominated and benefits from large, diverse image datasets.

Machine tending involves loading raw material into CNC machines, lathes, or presses, waiting for the machining cycle, and unloading the finished part. Data requirements: precise placement data (parts must be loaded into fixtures with ±0.5mm accuracy), force-torque for fixture seating verification, integration timing with the machine cycle (the robot must coordinate with the machine's start/stop signals), and handling of hot parts (thermal considerations for gripper design and sensor selection). The data collection platform must interface with the machine's PLC for cycle synchronization.

Palletizing arranges parts or boxes on pallets according to defined layer patterns. Data requirements: weight measurement or estimation for each item (affects placement stability), layer pattern definitions, edge-of-pallet handling (partial rows, layer transitions), and varied box sizes for mixed-product pallets. This task is more structured than bin picking — the patterns are predefined — but the physical execution requires precise placement and force-controlled stacking to prevent toppling. Data collection for palletizing in warehouse environments shares many of these requirements.

The factory environment creates data collection challenges that simply do not exist in research labs. Each challenge has engineering solutions, but they must be anticipated and planned for — discovering them during a collection campaign wastes production floor access time that is expensive and difficult to reschedule.

Vibration from presses, conveyors, compressors, and vehicle traffic propagates through the floor and into any equipment mounted on it. Camera mounts that are rigid in a lab develop micro-oscillations on a factory floor that blur images at long exposure times and shift extrinsic calibrations. Force-torque sensors pick up background vibration that obscures contact events. IMU data contains machinery-frequency components that must be filtered. Solutions: vibration-isolated camera mounts (rubber-damped or pneumatically isolated), high-shutter-speed camera settings (1/500s or faster for RGB), bandpass filtering of force-torque data (high-pass above machinery vibration frequency, typically 5-50Hz), and increased calibration verification frequency.

Temperature variation affects calibration through thermal expansion. A factory that operates at 15°C during winter mornings and 35°C during summer afternoons experiences 20°C thermal swing that causes metal camera mounts to expand by measurable amounts. Aluminum expands at 23μm/m/°C — a 0.5m camera arm expands by 0.23mm over a 20°C range. This is below the accuracy threshold for coarse manipulation but significant for precision tasks. Solutions: temperature monitoring with calibration correction, or recalibration when temperature change exceeds 5°C from the calibration reference.

Restricted access and safety zones limit where sensors and operators can be positioned. Robotic workcells have safety fencing with interlocked gates. CNC machines have enclosures. Press brakes have light curtains. Data collection equipment cannot obstruct safety systems, and human operators conducting demonstrations must follow all lockout/tagout procedures when entering robotic workcells. Solutions: pre-plan sensor placement with the facility's safety engineer, use remote camera positions outside safety zones where possible, and schedule data collection during designated maintenance windows when safety interlocks can be legitimately disengaged under controlled conditions.

Production schedule constraints are often the most difficult challenge. A production line running three shifts has zero tolerance for interruption. Data collection must happen during scheduled downtime — shift changes, preventive maintenance windows, weekend shutdowns — or on parallel equipment that is not currently in production. A typical manufacturing data collection campaign might have four-hour collection windows available twice per week. This means the collection protocol must be maximally efficient: sensors pre-calibrated, operators pre-trained, episode structure optimized for the available time.

Contamination and durability affect sensor longevity. Cutting oil mist coats camera lenses within hours. Metal shavings accumulate on magnetic surfaces. Coolant spray reaches surprisingly far from its source. Solutions: IP-rated sensor enclosures (minimum IP54 for dusty environments, IP65 for coolant-exposed areas), sacrificial protective windows on camera lenses (replaced between sessions), and sealed connectors for all cables. Force-torque sensors are generally more robust — the ATI Mini45 is rated for industrial environments — but cables and connectors remain vulnerable.

Manufacturing data collection requires sensor hardware that survives industrial conditions while maintaining the accuracy needed for training data. The standard configurations differ from lab setups in specific ways.

Camera mounting must address vibration, contamination, and restricted access simultaneously. Wrist-mounted cameras (RealSense D405) benefit from being inside the robot's safety zone and moving with the end-effector, providing consistent viewpoint relative to the task. But they are more exposed to contamination and collision risk. External cameras (D435 or industrial equivalents like FLIR Blackfly S) can be positioned outside safety zones but require fixed, vibration-resistant mounts. For manufacturing, use solid-state mounts — aluminum brackets bolted to structural steel, not clamp-on adjustable arms — with vibration-damping bushings. Include alignment pins so that if a camera is removed for cleaning, it returns to the same position within 0.5mm.

Force-torque sensors for manufacturing should be rated for the expected force range with appropriate overload protection. A bin-picking application with parts up to 5kg needs a sensor rated for at least 100N with overload tolerance of 5-10x. An assembly application with tight insertion forces needs higher resolution and lower range — the ATI Nano17 resolves 0.003N across a 12N range. Mount the sensor between the robot flange and the gripper with a rigid, precisely machined adapter plate — any compliance in the mounting corrupts force measurements.

Industrial Ethernet (EtherCAT, PROFINET, or EtherNet/IP) provides deterministic data transport that USB and consumer Ethernet cannot guarantee. In a factory with multiple welding robots, variable-frequency drives, and industrial wireless networks, electromagnetic interference can corrupt USB data transfers. Industrial Ethernet protocols include error detection and correction at the link layer, and many force-torque sensors (ATI, Schunk) offer industrial Ethernet interfaces with hardware-timestamped data. For cameras, GigE Vision over industrial Ethernet provides reliable image transport with hardware triggering support.

Calibration frequency must account for thermal drift, vibration, and the fact that access for recalibration may be limited to scheduled windows. A pragmatic approach: full calibration at the start of each collection session, automated verification (ArUco marker check) every 30 minutes during collection, and full recalibration if any verification fails. Log the ambient temperature at each calibration and verification point. Over time, build a thermal calibration model that predicts calibration drift as a function of temperature, enabling corrections without physical recalibration.

Successful manufacturing data collection requires collaboration with plant operations staff. This is an operational challenge as much as a technical one, and teams that underestimate it waste their limited floor access time.

Scheduled collection windows must be negotiated with production management weeks or months in advance. Identify the available windows: shift changeovers (typically 15-30 minutes, sufficient for short collection bursts), scheduled maintenance windows (2-8 hours, ideal for focused campaigns), weekend shutdowns (full access but may lack production-representative conditions), and parallel equipment (if a second identical cell exists, it may be available full-time). Build the collection protocol around the available window duration — if you have four-hour windows, design episodes and batch sizes that fit within that constraint with setup and teardown time included.

Working with plant personnel is essential. The production engineer who designed the assembly process understands tolerances, failure modes, and edge cases that no external robotics team will discover independently. The maintenance technician knows which fixtures have wear, which machines produce parts at the edge of tolerance, and where vibration is worst. The safety manager defines the boundaries of acceptable data collection activities. Build relationships with all three before the first collection session. Conduct a walkthrough of the collection protocol with plant staff and incorporate their feedback — they will identify practical issues that are invisible from a conference room.

Understanding the production process before collection is non-negotiable. Document the complete process flow: raw material input, machining operations, assembly sequence, quality checkpoints, and final packaging. Identify which process steps the robot will perform and which steps precede and follow. Understand how parts arrive at the robot station — by conveyor, by tote, by hand — and how they leave. This context determines the data collection protocol: initial conditions (how parts are presented), task boundaries (when the robot's responsibility starts and ends), and success criteria (what quality standard the output must meet).

Operator training for manufacturing extends beyond the standard collection protocol. Operators conducting human-in-the-loop data collection on a manufacturing floor must be trained on facility-specific safety requirements: PPE (steel-toed boots, safety glasses, hearing protection), robotic cell entry procedures, lockout/tagout for energy isolation, emergency stop locations, and hazard communication for chemicals in the workspace. They must understand the manufacturing process well enough to demonstrate it correctly — a demonstration that assembles components in the wrong sequence produces training data that teaches an illegal assembly order. This often means training operators alongside experienced production workers who can verify technique correctness.

The unique advantage of manufacturing data collected on the production floor is that the training distribution matches the deployment distribution by construction. There is no sim-to-real gap if the training data comes from the same environment, lighting, parts, and fixtures that the robot will encounter during production. This is the strongest argument for on-site collection despite its operational complexity.

But the path from collected data to deployed model still requires careful engineering. Format delivery must match the customer's training pipeline. Most manufacturing robotics teams use either custom PyTorch dataloaders (for which HDF5 is the natural storage format) or frameworks that support RLDS or LeRobot format. The data delivery package should include: the episodes in the target format, a dataloader implementation or configuration file, calibration data for each sensor, environment documentation (CAD models of the workcell, fixture drawings, part specifications), and a quality report summarizing the QA metrics for the delivered batch.

Integration with existing automation determines how the trained model will be deployed. Most manufacturing facilities use PLCs (Programmable Logic Controllers) for production sequencing, industrial robots controlled by vendor-specific controllers (FANUC, ABB, KUKA, Universal Robots), and MES (Manufacturing Execution Systems) for production tracking. A learned manipulation policy must interface with this existing infrastructure — receiving start signals from the PLC, sending completion signals back, and reporting error states through the standard alarm system. The data collection protocol should capture these integration points: what signals trigger the task start, what signals indicate task completion, and what happens when the task fails.

The data collected on the production floor also provides a baseline for continuous improvement. As the model is deployed and encounters edge cases that cause failures, those failures are collected as new data — adding to the training set and enabling iterative model improvement without returning to the lab. This is the flywheel effect of production-floor data collection: the deployment environment generates its own training data, and each model iteration is trained on a more representative dataset than the last.

For teams building their first manufacturing robotics dataset, start with the highest-volume, most repetitive task at the target facility. Bin picking of a single part type is ideal: the task is well-defined, the success criteria are clear, and the data requirements are well-understood. Achieve reliable performance on one task before expanding to assembly, inspection, or multi-part operations. Each task requires its own protocol, its own sensor configuration, and its own annotation schema — trying to collect everything at once leads to compromised quality across the board.