Long-Horizon / Traceable Credit Assignment
Round 2/10
vs previous: Round 1
patch source: main-agent autodiagnosed
Round 2: dimension evidence scan
Autoresearch loop: 主 agent 读取数据和指标 → 诊断具体问题 → 决定 patch → 改代码/评测 → 生成本轮可视化。当前小说场景 AHEAD intentionally disabled。
Idea tested
Use delayed-feedback text and per-turn feature tags to identify candidate culprit dimensions.
Diagnosed issues
Long-horizon round 2 dominant issue: hard_distractor_temporal_ambiguity.
culprit_step_accuracy changed +0.6500 vs previous round.
Selected next patch
Add temporal utility-drop windows so early weak distractors do not steal credit from the real culprit.
Data used this round
TRACE-USER-Bench synthetic delayed-feedback traces
One row per multi-turn episode: turn trace, utility/implicit feedback, delayed comment, gold culprit step/dimension, counterfactual repairs.
sample_count: 120 · splits: {"culprit_dimensions": {"lore_density": 27, "pacing": 35, "trope_misunderstanding": 32, "female_agency": 26}}
Evaluation protocol
Rank culprit steps from the full trace, compare top prediction and reciprocal rank to gold, then estimate counterfactual repair gain and confidence calibration.
Metric definitions
| Metric | Definition |
|---|---|
culprit_step_accuracy | Exact-match accuracy for the delayed-feedback culprit step. |
culprit_dimension_accuracy | Accuracy for the causal preference/content dimension. |
credit_mrr | Mean reciprocal rank of the gold culprit step. |
repair_gain | Mean expected utility gain from the selected counterfactual repair. |
credit_calibration | One minus confidence error; higher means confidence tracks correctness. |
Metrics vs previous
| Metric | Value | vs previous |
|---|---|---|
| culprit_step_accuracy | 0.6500 | +0.6500 |
| culprit_dimension_accuracy | 0.8917 | +0.1000 |
| credit_mrr | 0.8167 | +0.5304 |
| repair_gain | 0.1461 | +0.0701 |
| counterfactual_repair_gain | 0.1461 | +0.0701 |
| credit_calibration | 0.6277 | +0.2477 |
Concrete problems found
- trace_0008
step_miss:gold=4:pred=2, dimension_miss:gold=trope_misunderstanding:pred=female_agency那个误会梗埋得太早，后面解释让我出戏。
- trace_0017
step_miss:gold=6:pred=4, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0034
step_miss:gold=4:pred=2, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0043
step_miss:gold=4:pred=2, dimension_miss:gold=pacing:pred=trope_misunderstanding读到后面才发现前面铺垫太拖，剧情推进慢了。
- trace_0050
step_miss:gold=2:pred=1, dimension_miss:gold=trope_misunderstanding:pred=pacing那个误会梗埋得太早，后面解释让我出戏。
Top failure examples
- trace_0008 — step_miss:gold=4:pred=2, dimension_miss:gold=trope_misunderstanding:pred=female_agency
那个误会梗埋得太早，后面解释让我出戏。 - trace_0017 — step_miss:gold=6:pred=4, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0034 — step_miss:gold=4:pred=2, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0043 — step_miss:gold=4:pred=2, dimension_miss:gold=pacing:pred=trope_misunderstanding
读到后面才发现前面铺垫太拖，剧情推进慢了。 - trace_0050 — step_miss:gold=2:pred=1, dimension_miss:gold=trope_misunderstanding:pred=pacing
那个误会梗埋得太早，后面解释让我出戏。
Why zero-valued metrics appear
本轮没有需要特别解释的 0 指标。
Research-state update
本轮将失败模式写回下一轮方法假设：Add temporal utility-drop windows so early weak distractors do not steal credit from the real culprit.