Long-Horizon / Traceable Credit Assignment
Round 3/10
vs previous: Round 2
patch source: main-agent autodiagnosed
Round 3: temporal utility window
Autoresearch loop: 主 agent 读取数据和指标 → 诊断具体问题 → 决定 patch → 改代码/评测 → 生成本轮可视化。当前小说场景 AHEAD intentionally disabled。
Idea tested
Combine dimension evidence with utility drops, fast swipes, and delayed-feedback timing.
Diagnosed issues
Long-horizon round 3 dominant issue: culprit_dimension_evidence_gap.
credit_calibration changed +0.2803 vs previous round.
Selected next patch
Fuse delayed-feedback dimension tokens with per-turn feature tags, severity, and repairability.
Data used this round
TRACE-USER-Bench synthetic delayed-feedback traces
One row per multi-turn episode: turn trace, utility/implicit feedback, delayed comment, gold culprit step/dimension, counterfactual repairs.
sample_count: 120 · splits: {"culprit_dimensions": {"lore_density": 27, "pacing": 35, "trope_misunderstanding": 32, "female_agency": 26}}
Evaluation protocol
Rank culprit steps from the full trace, compare top prediction and reciprocal rank to gold, then estimate counterfactual repair gain and confidence calibration.
Metric definitions
| Metric | Definition |
|---|---|
culprit_step_accuracy | Exact-match accuracy for the delayed-feedback culprit step. |
culprit_dimension_accuracy | Accuracy for the causal preference/content dimension. |
credit_mrr | Mean reciprocal rank of the gold culprit step. |
repair_gain | Mean expected utility gain from the selected counterfactual repair. |
credit_calibration | One minus confidence error; higher means confidence tracks correctness. |
Metrics vs previous
| Metric | Value | vs previous |
|---|---|---|
| culprit_step_accuracy | 0.9250 | +0.2750 |
| culprit_dimension_accuracy | 0.9333 | +0.0416 |
| credit_mrr | 0.9625 | +0.1458 |
| repair_gain | 0.1846 | +0.0385 |
| counterfactual_repair_gain | 0.1846 | +0.0385 |
| credit_calibration | 0.9080 | +0.2803 |
Concrete problems found
- trace_0008
step_miss:gold=4:pred=2, dimension_miss:gold=trope_misunderstanding:pred=female_agency那个误会梗埋得太早，后面解释让我出戏。
- trace_0017
step_miss:gold=6:pred=4, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0034
step_miss:gold=4:pred=2, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0050
step_miss:gold=2:pred=1, dimension_miss:gold=trope_misunderstanding:pred=pacing那个误会梗埋得太早，后面解释让我出戏。
- trace_0094
step_miss:gold=5:pred=3, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
Top failure examples
- trace_0008 — step_miss:gold=4:pred=2, dimension_miss:gold=trope_misunderstanding:pred=female_agency
那个误会梗埋得太早，后面解释让我出戏。 - trace_0017 — step_miss:gold=6:pred=4, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0034 — step_miss:gold=4:pred=2, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0050 — step_miss:gold=2:pred=1, dimension_miss:gold=trope_misunderstanding:pred=pacing
那个误会梗埋得太早，后面解释让我出戏。 - trace_0094 — step_miss:gold=5:pred=3, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。
Why zero-valued metrics appear
本轮没有需要特别解释的 0 指标。
Research-state update
本轮将失败模式写回下一轮方法假设：Fuse delayed-feedback dimension tokens with per-turn feature tags, severity, and repairability.