From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents

Yue ZhangHan LinZun WangYidong HuangZaid KhanMohit Bansal UNC Chapel Hill

arXiv Code BibTeX

WEDGE: Weakness-guided Environment Design, Generation, and Evaluation

The agent fails

LLM mines the weakness

15 successes15 failures

LLM

The LLM proposes a feature; it is validated on every episode.

Find the weak regime

Below the agent’s average is the weak regime: we generate there.

Generate environments (new layouts)

New layouts; red: path before the Sofa is seen; blue: the agent’s rollout.

Adapt

Same episode after training. Left: route to the Bed; right: what the agent sees.

re-diagnose · next round

Why policy-conditioned

Different agents fail for different reasons

Same task, three pretrained agents, three different weakness profiles. Each cell is how much an agent's success rate drops between the easy third and the hard third of its diagnostic episodes for that property. The biggest drop per agent is its top weakness.

Success-rate drop, easy third → hard third (percentage points)

Weakness mining

LLM-guided weakness discovery

Sample successful and failed episodes, ask the LLM for a feature that separates them, and validate it on the whole diagnostic set. Rejected candidates go back with their reason; validated ones become the mined weakness.

1Contrasting episodes

15 successes and 15 failures.

2LLM writes an executable weakness

3Validated on every episode

Coverage

Non-redundant

Predictive

easy third

–hard third

–

Incremental

rejected → resample 15 + 15 and propose again

?Now you

available fields

s scene:
p planner:
cat target category · Math available

Weakness-conditioned generation

Each weakness maps to the factor that controls it

Target visibility is controlled by where the object sits, so we re-place it. Blind-path ratio and target-room remoteness are layout properties, so we generate new layouts. Candidates are kept only inside the agent's weak regime.

Adaptation

Training on the generated environments fixes the failures

Validation episodes the pretrained agent failed and the adapted agent solved after one round. Left: before. Right: after. Paths are logged positions over the simulator’s top-down render.

pretrained after one round target start

Transfer

The gains generalize to real HM3D scans

Uni-NaVid was diagnosed and adapted in ProcTHOR only. On held-out HM3D episodes the pretrained model failed and the adapted model solved, here are its route on the scan’s navigation map and what it saw along the way. The route is drawn as the video plays.

pretrained after one round target start

Results

Experimental results on policy-specific adaptation and generalization

For each agent: the validated weakness, adaptation on ProcTHOR, and evaluation back on HM3D. Baselines use the same recipe and budget with environments or examples chosen at random or by failure alone.

Table 1: success rate and SPL for DD-PPO, Uni-NaVid and Gemini under pretrained, random, hard-mined and WEDGE settings on ProcTHOR and HM3D

Citation

BibTeX

@article{zhang2026wedge,
  title   = {From Failures to Learnable Worlds: Policy-Conditioned Environment Generation for Navigation Agents},
  author  = {Zhang, Yue and Lin, Han and Wang, Zun and Huang, Yidong and Khan, Zaid and Bansal, Mohit},
  journal = {arXiv preprint},
  year    = {2026}
}