Long-Horizon / Traceable Credit Assignment
Round 4/10
vs previous: Round 3
patch source: main-agent autodiagnosed
Round 4: causal candidate scorer
Autoresearch loop: 主 agent 读取数据和指标 → 诊断具体问题 → 决定 patch → 改代码/评测 → 生成本轮可视化。当前小说场景 AHEAD intentionally disabled。
Idea tested
Rank steps by dimension match, severity, persistence, and repairability.
Diagnosed issues
Long-horizon round 4 dominant issue: multi_cause_trace_ambiguity.
culprit_step_accuracy changed +0.0250 vs previous round.
Selected next patch
Keep CREDIT-TRACE verifier; next expansion should add multi-cause labels from real reader logs, not AHEAD.
Data used this round
TRACE-USER-Bench synthetic delayed-feedback traces
One row per multi-turn episode: turn trace, utility/implicit feedback, delayed comment, gold culprit step/dimension, counterfactual repairs.
sample_count: 120 · splits: {"culprit_dimensions": {"lore_density": 27, "pacing": 35, "trope_misunderstanding": 32, "female_agency": 26}}
Evaluation protocol
Rank culprit steps from the full trace, compare top prediction and reciprocal rank to gold, then estimate counterfactual repair gain and confidence calibration.
Metric definitions
| Metric | Definition |
|---|---|
culprit_step_accuracy | Exact-match accuracy for the delayed-feedback culprit step. |
culprit_dimension_accuracy | Accuracy for the causal preference/content dimension. |
credit_mrr | Mean reciprocal rank of the gold culprit step. |
repair_gain | Mean expected utility gain from the selected counterfactual repair. |
credit_calibration | One minus confidence error; higher means confidence tracks correctness. |
Metrics vs previous
| Metric | Value | vs previous |
|---|---|---|
| culprit_step_accuracy | 0.9500 | +0.0250 |
| culprit_dimension_accuracy | 0.9500 | +0.0167 |
| credit_mrr | 0.9750 | +0.0125 |
| repair_gain | 0.2016 | +0.0170 |
| counterfactual_repair_gain | 0.2016 | +0.0170 |
| credit_calibration | 0.9320 | +0.0240 |
Concrete problems found
- trace_0017
step_miss:gold=6:pred=4, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0034
step_miss:gold=4:pred=2, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0094
step_miss:gold=5:pred=3, dimension_miss:gold=lore_density:pred=female_agency前面设定解释太密，后面节奏被拖住了。
- trace_0101
step_miss:gold=6:pred=5, dimension_miss:gold=pacing:pred=female_agency读到后面才发现前面铺垫太拖，剧情推进慢了。
- trace_0114
step_miss:gold=3:pred=2, dimension_miss:gold=lore_density:pred=trope_misunderstanding前面设定解释太密，后面节奏被拖住了。
Top failure examples
- trace_0017 — step_miss:gold=6:pred=4, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0034 — step_miss:gold=4:pred=2, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0094 — step_miss:gold=5:pred=3, dimension_miss:gold=lore_density:pred=female_agency
前面设定解释太密，后面节奏被拖住了。 - trace_0101 — step_miss:gold=6:pred=5, dimension_miss:gold=pacing:pred=female_agency
读到后面才发现前面铺垫太拖，剧情推进慢了。 - trace_0114 — step_miss:gold=3:pred=2, dimension_miss:gold=lore_density:pred=trope_misunderstanding
前面设定解释太密，后面节奏被拖住了。
Why zero-valued metrics appear
本轮没有需要特别解释的 0 指标。
Research-state update
本轮将失败模式写回下一轮方法假设：Keep CREDIT-TRACE verifier; next expansion should add multi-cause labels from real reader logs, not AHEAD.