Teleoperation-Driven and Keyframe-Based Generalizable Imitation Learning for Construction Robots

Yan Li; Songyang Liu; Mengjun Wang; Shuai Li; Jindong Tan

doi:10.1061/JCCEE5.CPENG-5884

Advancing Construction Machine Autonomy on Earth and Beyond.

Photo: Built Robotics / Unsplash

The robot digs under human control.

Teleoperation.We developed this robotic excavation testbed to collect operator demonstrations.

Demonstration collection.Robot states and visual observations are recorded together for learning.

Challenges to autonomous excavation

Data challenge

Repeated motions add little useful guidance.The recording grows without showing a new way to complete the task.

Poor demonstrations make reliable learning harder.

Learning challenge

The robot moves, but does not reliably complete a dig.Limited demonstrations and complex visual observations make it difficult to learn which actions will achieve the excavation goal.

Our solution learns from keyframes and visual goals.

Keyframe selection and action-bound visual representations guide a high-level subgoal policy and a low-level action policy — Keyframes and visual goals guide two policies: one chooses subgoals, the other generates actions.

Calibrated point cloud aligned with the robot’s working space — Align the observed scene with the robot’s coordinate frame.

Current terrain contour — Current and target contours describe the desired terrain change.

Desired terrain contour with an excavation depression — Current and target contours describe the desired terrain change.

The learned policy improves excavation success.

Selected keyframes marked as red points along a recorded robot trajectory — **Keyframes preserve the essential motion.**The selected poses provide a more compact demonstration for training, with fewer redundant frames.

Excavation with a specified terrain goal.In simulation, the full policy improves task success over the compared methods, including the version without the policy that learns motions between subgoals.

Simulation comparison

Simulation success-rate comparison with a specified terrain goal; the proposed method outperforms the compared methods — Brown curve: proposed method.

Excavation with a goal learned from vision.The visual policy improves task success over the compared methods without requiring a manually specified final goal. Linking visual features to robot actions improves learning.

Simulation comparison

Simulation success-rate comparison with a learned visual goal; the proposed method outperforms the compared methods — Brown curve: proposed method.

Extending our research toward lunar excavation.

A real JAXA lunar-rover research prototype with four tracked units, a single articulated digging arm, and a bucket in a simulated lunar terrain test facility — JAXA–Yanmar single-arm excavation research, ground experiment, 2023. Courtesy of JAXA.

We aim to extend demonstration-based excavation learning to robotic arms working with lunar regolith, where communication delays make continuous remote control difficult.

Publication

✳ 2024 Best Paper Award ↗