Long-Horizon / Traceable Credit Assignment
Round 5/10
vs previous: Round 4
patch source: main-agent autodiagnosed
Round 5: trace verifier + repair planner
Autoresearch loop: 主 agent 读取数据和指标 → 诊断具体问题 → 决定 patch → 改代码/评测 → 生成本轮可视化。当前小说场景 AHEAD intentionally disabled。
Idea tested
Verify culprit-step candidates with counterfactual repairs and calibrated confidence.
Diagnosed issues
Long-horizon round 5 dominant issue: multi_cause_trace_ambiguity.
repair_gain changed +0.0137 vs previous round.
Selected next patch
Keep CREDIT-TRACE verifier; next expansion should add multi-cause labels from real reader logs, not AHEAD.
Data used this round
TRACE-USER-Bench synthetic delayed-feedback traces
One row per multi-turn episode: turn trace, utility/implicit feedback, delayed comment, gold culprit step/dimension, counterfactual repairs.
sample_count: 120 · splits: {"culprit_dimensions": {"lore_density": 27, "pacing": 35, "trope_misunderstanding": 32, "female_agency": 26}}
Evaluation protocol
Rank culprit steps from the full trace, compare top prediction and reciprocal rank to gold, then estimate counterfactual repair gain and confidence calibration.
Metric definitions
| Metric | Definition |
|---|---|
culprit_step_accuracy | Exact-match accuracy for the delayed-feedback culprit step. |
culprit_dimension_accuracy | Accuracy for the causal preference/content dimension. |
credit_mrr | Mean reciprocal rank of the gold culprit step. |
repair_gain | Mean expected utility gain from the selected counterfactual repair. |
credit_calibration | One minus confidence error; higher means confidence tracks correctness. |
Metrics vs previous
| Metric | Value | vs previous |
|---|---|---|
| culprit_step_accuracy | 0.9583 | +0.0083 |
| culprit_dimension_accuracy | 0.9583 | +0.0083 |
| credit_mrr | 0.9792 | +0.0042 |
| repair_gain | 0.2153 | +0.0137 |
| counterfactual_repair_gain | 0.2153 | +0.0137 |
| credit_calibration | 0.9400 | +0.0080 |
Concrete problems found
- trace_0008
step_miss:gold=4:pred=2, dimension_miss:gold=trope_misunderstanding:pred=female_agency那个误会梗埋得太早，后面解释让我出戏。
- trace_0017
step_miss:gold=6:pred=4, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0034
step_miss:gold=4:pred=2, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0094
step_miss:gold=5:pred=3, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0101
step_miss:gold=6:pred=5, dimension_miss:gold=pacing:pred=female_agency读到后面才发现前面铺垫太拖，剧情推进慢了。
Top failure examples
- trace_0008 — step_miss:gold=4:pred=2, dimension_miss:gold=trope_misunderstanding:pred=female_agency
那个误会梗埋得太早，后面解释让我出戏。 - trace_0017 — step_miss:gold=6:pred=4, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0034 — step_miss:gold=4:pred=2, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0094 — step_miss:gold=5:pred=3, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0101 — step_miss:gold=6:pred=5, dimension_miss:gold=pacing:pred=female_agency
读到后面才发现前面铺垫太拖，剧情推进慢了。
Why zero-valued metrics appear
本轮没有需要特别解释的 0 指标。
Research-state update
本轮将失败模式写回下一轮方法假设：Keep CREDIT-TRACE verifier; next expansion should add multi-cause labels from real reader logs, not AHEAD.