Why floor pickup is difficult
Floor pickup is difficult because the robot sees the object from a moving base, approaches from imperfect poses, and must handle everyday variation without damaging the home.
A closed gripper does not prove the robot picked anything up. Lumi needs grasp verification before moving to the bin.
- shape
- size
- surface material
- orientation
- distance
- occlusion
- lighting
- graspability
A closed gripper does not prove the robot picked anything up.
Demonstration collection
The manipulation path begins with structured demonstrations instead of pretending the robot already knows every household object.
Remote leader/follower teleoperation is being prepared to make data collection more practical, but it still needs supervised physical validation.
Laptop-connected leader arm
Used by the operator to guide pickup, recovery, and placement demonstrations.
Jetson-connected follower arm
Mounted on Lumi and controlled locally during supervised demonstrations.
Jetson
Owns camera streams, follower-arm safety, dataset recording, and learned-policy inference.
Useful data variations
- left / center / right object positions
- near and far reach
- multiple object rotations
- different safe household objects
- successful grasps
- failed grasps
- recovery actions
- different lighting and background conditions
Near-term learned-policy roadmap
Stage 1 - Structured pickup setup
Known workspace, safe objects, fixed camera locations, and known bin location.
Stage 2 - Teleoperation demonstrations
Operator records approaches, grasps, lifts, failed attempts, and recovery behavior.
Stage 3 - First learned policy
ACT-style imitation learning predicts short sequences of arm and gripper actions from camera observations and robot state.
Stage 4 - Floor-pickup generalization
Increase object diversity, position variation, viewpoint variation, and recovery examples.
Stage 5 - Pretrained policy research
Evaluate pretrained robot-policy or VLA-style approaches only after Lumi has high-quality task data and stable hardware interfaces.
Pick, place, and verify sequence
- Detect object
- Estimate pickup point
- Move to pre-grasp pose
- Descend
- Close gripper
- Lift
- Verify grasp
- Navigate to bin
- Place object
- Verify release
- Retry or mark failed
This page owns
- pre-grasp positioning
- grasp behavior
- lifting
- grasp verification
- bin placement
- release verification
- recovery behavior
Planning and navigation remain responsible for target selection and moving the robot base between object and bin.
Learned policy versus safety system
A learned policy can propose arm actions. It does not bypass safety. The Jetson-side safety layer still enforces joint limits, velocity limits, workspace limits, watchdog behavior, and emergency-stop priority. ACT-style imitation learning is the initial baseline; VLA-style approaches are a later research direction, not Lumi's current deployed pickup method.