Embodied Continual Learning

Learning New Tasks via Reusable Skills: Skill-Compositional Experts for Embodied Continual Learning

SCE enables robots to continually acquire new tasks by decomposing demonstrations into reusable skills and composing them through dual execution-and-transition experts.

Shuaike Zhang1, Shaokun Wang1, Haoyu Tang2, Jianlong Wu1,3, Liqiang Nie1,3

1Harbin Institute of Technology, Shenzhen
2Shandong University
3Shenzhen Loop Area Institute

Paper Code coming soon
Overview comparing embodied continual learning with reusable skill composition in SCE.
Closed-loop control amplifies the impact of old-task feature drift, while SCE composes new tasks from a skill base.

Abstract

Embodied Continual Learning (ECL) aims to enable robots to continually acquire new manipulation tasks while retaining previously learned behaviors under closed-loop control. In ECL, feature drift can propagate through sequential decision-making under closed-loop control, turning representation changes into compounding behavioral deviations on previously learned tasks. A key challenge in ECL lies in structured skill reuse across continually evolving tasks, since existing methods primarily focus on skill learning without explicitly organizing them for coherent task execution.

To address this issue, we propose SCE, a Skill-Compositional Experts framework for ECL. SCE builds a skill base via Compositional Skill Grounding (CSG), which decomposes task demonstrations into reusable skills. Based on this, Dual Execution-and-Transition Experts (DETE) enable new task learning through skill composition, where one branch ensures skill execution and the other supports transitions between skills for coherent behavior. Experiments on LIBERO benchmarks and real-world manipulation tasks show that SCE improves retention and overall task performance. Further feature drift analyses and ablation studies verify the effectiveness of our method.

Method

CSG builds the skill base; DETE composes skills

Compositional Skill Grounding

CSG partitions each demonstration into temporally coherent segments using robot-state dynamics, then uses VLM-grounded induction to assign skill labels and update the skill base.

Execution Expert Branch

DETE routes each input to a skill-specific expert through a skill distribution, giving each skill dedicated capacity and reducing interference with unrelated skills.

Transition Expert Branch

A transition-aware router models cross-skill dependencies around skill boundaries, and adaptive fusion balances execution and transition outputs during closed-loop action generation.

Overall SCE framework with CSG and DETE modules.
Overall framework of SCE. Demonstrations are decomposed into reusable skills, and DETE augments a pretrained VLA action decoder for skill-specific execution and coherent cross-skill composition.

Results

Stronger retention and final success across simulation and real-world tasks

The reported metrics follow the paper protocol: FWT measures new-task performance, NBT measures forgetting on previously learned tasks, AUC summarizes performance across stages, and Final SR reports final-stage average success.

LIBERO-Goal 83.9

Final SR with 4.3 NBT and 82.9 AUC.

LIBERO-Long 73.9

Final SR with 2.5 NBT and 70.2 AUC.

Real World 80.0

Final SR with 0.6 NBT and 79.1 AUC.

Main results: mean ± standard deviation over three independent training seeds. † Prior results reported by Stellar VLA.
Method LIBERO-Goal LIBERO-Long
FWT NBT AUC Final SR FWT NBT AUC Final SR
UniVLA†87.5 ± 0.165.7 ± 6.135.2 ± 4.720.5 ± 4.770.3 ± 5.631.3 ± 4.043.6 ± 2.417.1 ± 2.0
MoDE†69.7 ± 1.823.2 ± 2.946.8 ± 2.941.6 ± 4.871.1 ± 1.546.4 ± 1.132.9 ± 1.419.9 ± 1.4
T-Stellar†76.6 ± 3.419.1 ± 3.657.5 ± 1.157.0 ± 2.080.3 ± 1.638.0 ± 2.548.6 ± 2.632.2 ± 4.4
TS-Stellar†71.8 ± 4.025.9 ± 2.747.5 ± 2.547.7 ± 1.484.7 ± 1.440.8 ± 0.651.4 ± 1.439.7 ± 1.6
IsCiL†66.8 ± 1.37.8 ± 2.560.9 ± 2.848.5 ± 2.065.8 ± 0.331.5 ± 0.341.4 ± 0.117.3 ± 0.3
LOTUS†46.8 ± 4.337.7 ± 3.918.5 ± 1.510.8 ± 1.132.3 ± 2.32.8 ± 2.729.4 ± 0.531.8 ± 2.3
Sequential81.7 ± 0.774.1 ± 0.624.3 ± 0.47.5 ± 0.161.7 ± 0.758.1 ± 0.314.4 ± 0.73.7 ± 0.6
ER74.7 ± 0.417.3 ± 0.360.5 ± 0.344.9 ± 0.173.3 ± 1.422.5 ± 0.455.2 ± 1.250.9 ± 0.9
Task-MoE84.0 ± 0.514.4 ± 1.272.3 ± 0.466.0 ± 1.173.5 ± 0.812.1 ± 0.564.4 ± 0.661.1 ± 2.9
SCE86.5 ± 0.34.3 ± 0.482.9 ± 0.483.9 ± 0.671.7 ± 1.92.5 ± 1.870.2 ± 0.273.9 ± 1.0
Representative real-world manipulation snapshots for three SCE tasks.
Real-world tasks include Box-to-Board, Board-to-Bowl, and Box-to-Bowl manipulation with shared behavioral elements such as object retrieval, transfer, placement, and lid opening.
Real-world continual learning: 30 trials per observed task, method, and stage.
Method FWT NBT AUC Final SR
Sequential55.645.628.710.0
ER52.223.937.222.2
Task-MoE64.48.359.858.9
SCE80.00.679.180.0

Videos

Real-World Video Demos

The videos show successful SCE rollouts and representative failure cases from baseline methods. All videos are played at 2x speed.

SCE success demos

RW-1: Box-to-Board

SCE moves the block from the box to the wooden board.

RW-2: Board-to-Bowl

SCE opens the lid and transfers the block from the board into the bowl.

RW-3: Box-to-Bowl

SCE composes the learned manipulation skills to move the block from the box into the bowl.

Baseline failure cases

Task-MoE: RW-3 Failure

The policy hesitates between gripper-state changes and upward motion, causing execution to remain stuck around these actions.

ER: RW-2 Failure

The policy moves the cover aside but does not proceed to transfer the block from the wooden board into the bowl.

Analysis

Why SCE Works

Old-task feature cosine distance and shared-state rollout distance, with 95% bootstrap confidence intervals.
SCE preserves old-task representations and rollout behavior. Feature cosine distances are normalized to SCE within each feature type. Rollout curves compare acquisition and final checkpoints from identical simulator states; shading denotes pointwise 95% bootstrap confidence intervals across tasks. Lower is better in both panels.
Ablations on LIBERO-Goal. EEB: Execution Expert Branch; TEB: Transition Expert Branch.
MethodFWTNBTAUCFinal SR
SCE w/o CSG72.47.066.163.6
SCE w/o TEB61.4-6.367.169.4
SCE w/o EEB80.66.575.080.0
SCE86.24.382.583.4

CSG organizes reusable skills, while the two expert branches provide complementary execution and transition modeling. Full SCE achieves 82.5 AUC and 83.4 Final SR in the ablation setting.