Grasping objects
Walk up, reach, grasp and lift, from onboard depth: three objects on a table and one on a low box.
FIBER
Action‑Aligned Latent Planning for Versatile Humanoid Loco‑Manipulation
Overview
What FIBER does, the question it answers, and how it works.
How do we abstract the interface between high-level planning and whole-body control, without abstracting away the physical responses that realize interaction?
feedback only on variables chosen in advance
feedback only on variables chosen in advance
the planner must model high-rate feedback itself
Our answer: plan in an action-aligned latent space.
FIBER’s planner outputs latent commands for a frozen behavior foundation model (BFM). Each latent is defined by the actions it applies to the robot, so the BFM keeps closing the loop at 50 Hz while the planner replans at 5 Hz. One perceptive planner, trained entirely offline, performs box pickup, door opening and object grasping on a real Unitree G1 from onboard depth.
Real-world rollouts
Box pickup, door opening and object grasping on a Unitree G1, from onboard depth. Experts, latent projectors and the planner are trained entirely in simulation.
Picking up thrown boxes
A person throws the box to a random spot; the robot walks over and picks it up.
Picking up a box from another box
Whole-body pickup of a box resting on a second box, at three placements.
Opening doors
Two heavy doors opened by leaning in with the whole body, and a light door.
Steering with waypoints
Waypoints guide the robot to the box and through whole-body motions while it carries.
18 real-world rollouts. Videos play while they are on screen.
Method
Independently acquired experts are projected into a frozen BFM’s latent space through the actions they induce; one perceptive planner then learns from their latent rollouts, entirely offline.
FIBER overview
What should we plan for humanoid whole-body intelligence?
Action-aligned latents.
Paused
- 1
Experts learn each skill on their own
Box pickup, door opening and object grasping each get their own expert, with their own observations, rewards and training. They do not need to share anything.
- 2
Project each expert into the frozen BFM’s latents
Along each expert’s rollouts, find the latent sequence z1…zH whose BFM actions match the expert’s local response at every visited state. The adaptive-horizon BFM itself stays frozen.
- 3
Train one perceptive planner, offline
Roll out the projected skills to collect latent trajectories, then train a single flow-matching planner on all of them, conditioned on depth, proprioception and the command. The experts are no longer needed.
- 4
Deploy: slow planning, fast feedback
The planner replans latents at 5 Hz; the frozen BFM turns them into whole-body actions at 50 Hz, reacting to every new state.
Simulation
The same planner and frozen BFM in MuJoCo, given language and waypoint commands.
One start, eight goalsA single pickup, branched into eight carries; only the waypoint in the steering command differs. Routes are drawn on the floor.
“pick up the box”“carry the box”+ waypoint
Picking up boxes
“pick up the box”
Three box sizes on the floor and three support heights, each approached from 1.2 m away.
Six pickups, one plannerSix independent rollouts shown side by side, from the smallest box on the floor to the highest support.
Placing and sorting
“pick up the box”“carry the box”“put the box down on the raised platform”
Put-down commands set a box down on the floor, a low step or a raised platform. Chained one after another, they sort several boxes in one continuous rollout.
Sorting by color, with chained commandsThree colored boxes to three matching destinations — the clay 42 cm platform, the teal 30 cm step and the blue floor ring — in one continuous rollout. Each box’s destination is set by its waypoint and put-down command; the planner sees only depth.
Whole-body carrying
“carry the box”+ waypoint+ height
The steering command also carries a body height. Lowered from the default 0.75 m, the planner crouches while it walks, or kneels and stands back up, holding the box throughout.
Under a low beamPick up the box, then crouch-walk under a beam 1.12 m above the floor with the height command at 0.60 m, and stand up again on the far side. Standing, the robot is 1.32 m tall.
Grasping objects
“pick up the object from the table”
The grasp expert is trained on the GRAIL tabletop library (2,991 demonstrations of 685 objects). The 128 objects of the paper’s grasp evaluation, with their own textures.
The evaluation objectsAll 128 objects, each turning in place.
Different objects, same plannerIndependent tabletop grasps of eight different objects, shown side by side.
Doors and skill chains
“open the door”“walk”“pick up the box”“carry the box”
A light and a heavy door, and the paper’s skill chain in one rollout.
Open the door, pick up the box, carry it back, place it downThree cameras on one rollout. A 3 s settle on the frozen BFM’s stand latent separates crossing and pickup, as in the playground.
Push recovery
“carry the box”+ waypoint
Shoves on the pelvis (red arrows) while carrying, slowed to 0.3× around each push. The translucent blue robot is the same moment without the push.
Three pushes, one carry180 N from the left, 180 N from the right, then 190 N from behind, each for 0.15 s. Between planner updates the frozen BFM reacts at 50 Hz and keeps the box.
Interactive playground
MuJoCo physics and the planner run in your browser. Pick a task, place a waypoint, and watch it act.
Try FIBER in your browser
Give the latent planner a command and watch the frozen BFM carry it out in MuJoCo physics.
Pick up a box, open a door, grasp an object, or walk and carry to a waypoint.
The web version runs a few-step distilled student planner (3 flow steps, INT8 weights, replanning at 1 Hz) instead of the paper's planner (10 Heun steps at 5 Hz), so success rates may be lower than reported. About 677 MB of models load on first launch. Desktop Chrome with WebGPU recommended.MuJoCo physics + FIBER ONNX inference in your browserOpen in a new tab ↗
Results
Box size and placement, door geometry, and grasped objects vary; FIBER is compared with replay, tracking and planning baselines.
128
variations per task
Box size and placement · door geometry · grasped objects
Mean simulation success across pickup, door opening, and grasping.
FIBER is the proposed unified policy; LPP Replay and the kinematic planner are comparison baselines.
Simulation success across tasks
Success rate (%) · with approach · higher is better
Pickup box
Task-specific policies
Reference-prompted execution
Perceptive planning
Open door
Task-specific policies
Reference-prompted execution
Perceptive planning
Grasp object
Task-specific policies
Reference-prompted execution
Perceptive planning
Mean · with approach
Task-specific policies
Reference-prompted execution
Perceptive planning
Success rates in percent. Pickup and door opening include the approach phase. † Task-specific policy. * Uses privileged object states instead of egocentric observations. Mean uses with-approach settings. Action Planner predicts low-level action trajectories directly instead of BFM latents.
AH-BFM analysis
Adaptive-horizon behavioral foundation model
A single backward representation B(s) is accumulated with different discount factors. Low γ produces a local successor feature, while high γ preserves information from farther states; adaptive training retains both outputs in one BFM.
Fγ(sₜ) = Σₖ₌₀∞ γᵏ B(sₜ₊ₖ)
Single shared representation
B(s)
B(st)→
B(st+1)→
B(st+2)→
B(st+3)
Exact weights wk = γk on one shared state axis
γ = 0.8
γ = 0.98
Each marker is computed exactly as γk; lines only connect the discrete samples.
k048121620
γ = 0.81.00000.40960.16780.06870.02810.0115
γ = 0.981.00000.92240.85080.78470.72380.6676
Different Fγ outputs
One B(s) remains shared across both horizons.
0.1020Joint-MAE ↓
0.1950.5 s BE-EMD ↓
0.4221 s BE-EMD ↓
0.5902 s BE-EMD ↓
| Method | Joint-MAE ↓ | BE-EMD ↓ | |||
|---|---|---|---|---|---|
| 0.2 s | 0.5 s | 1 s | 2 s | ||
| SONIC | 0.1315 | — | — | — | — |
| BFM γ0.8 | 0.1107 | 0.038 | 0.200 | 0.495 | 0.792 |
| BFM γ0.98 | 0.1327 | 0.048 | 0.213 | 0.451 | 0.601 |
| AH-BFM | 0.1020 | 0.049 | 0.195 | 0.422 | 0.590 |
Multi-horizon BFM tracking and behavior embedding performance on 100STYLE.
Paper
Read the paper and cite FIBER.
Learning a single perceptive policy for diverse humanoid loco-manipulation requires unifying high-level task decisions without losing the state-dependent physical responses that make each interaction succeed. This makes the representation between high-level planning and whole-body control critical: kinematic or contact targets provide compact abstractions but do not directly encode interaction-specific control responses, while directly predicting low-level actions requires the planner to model high-frequency closed-loop feedback. We propose FIBER, a framework that uses the executable latent space of a frozen Behavior Foundation Model (BFM) as a shared planning representation for heterogeneous interaction skills. Our Latent Policy Projection (LPP) maps independently acquired task-specific policies into this representation by identifying latent commands whose induced actions match their local control responses, enabling lower-rate planning while retaining the BFM's high-frequency state feedback. After projection, we aggregate closed-loop latent rollouts from all skills and train a single perceptive flow planner entirely offline. We further introduce AH-BFM, an adaptive-horizon BFM trained with multi-γ Forward-Backward (FB) learning to support both precise control and temporally extended whole-body behavior. We evaluate FIBER on box pickup, door opening, and object grasping in simulation and on a real Unitree G1 humanoid. Across 128 diverse variations per task in simulation, FIBER achieves 42.0% mean success, more than tripling projected latent replay (11.6%) and substantially outperforming kinematic planning (3.5%). Real-world experiments further demonstrate sim-to-real transfer, skill chaining, and conditional steering through the same unified planner.
BibTeX
@article{lin2026fiber,
title = {Action-Aligned Latent Planning for Versatile Humanoid Loco-Manipulation},
author = {Lin, Yutang and Shi, Guanya and Liu, C. Karen},
year = {2026}
}