Self-Evolve / Feedback-to-State
Round 6/10
vs previous: Round 5
patch source: main-agent autodiagnosed
Round 6: Round 6 neutral-control evaluator
Autoresearch loop: 主 agent 读取数据和指标 → 诊断具体问题 → 决定 patch → 改代码/评测 → 生成本轮可视化。当前小说场景 AHEAD intentionally disabled。
Idea tested
Add neutral/no-update examples and rename no_update_precision to neutral_no_update_accuracy.
Diagnosed issues
Self-evolve round 6 dominant issue: residual_future_probe_gap.
future_probe_win_rate changed +0.0083 vs previous round.
Selected next patch
Use future-probe-aware reranking and hard anti-overgeneralization probes; keep AHEAD disabled.
Data used this round
PIF-Bench synthetic novel feedback trajectories
One row per user-turn: hidden profile, content features, explicit feedback, implicit feedback, gold preference update, gold agent-state diff, future probe.
sample_count: 466 · splits: {"feedback_rows": 400, "neutral_no_update_rows": 66}
Evaluation protocol
Run the round-specific updater on every feedback event, compare predicted dimensions/targets/scope with gold state diffs, then estimate future-probe success.
Metric definitions
| Metric | Definition |
|---|---|
dimension_f1 | Macro F1 between predicted preference dimensions and gold dimensions. |
target_f1 | Macro F1 between predicted agent-state targets and gold targets. |
future_probe_win_rate | Mean synthetic win probability on near/far/anti-overgeneralization future probes. |
overgeneralization_rate | Fraction of predicted updates that incorrectly globalize local/story feedback; zero is good. |
neutral_no_update_accuracy | Accuracy on neutral/no-gold-update rows; replaces the confusing no_update_precision metric. |
Metrics vs previous
| Metric | Value | vs previous |
|---|---|---|
| dimension_f1 | 0.7165 | +0.0000 |
| target_f1 | 0.7317 | +0.0000 |
| update_target_f1 | 0.7317 | +0.0000 |
| future_probe_win_rate | 0.8294 | +0.0083 |
| overgeneralization_rate | 0.0000 | +0.0000 |
| feedback_incorporation_rate | 0.7775 | +0.0000 |
| neutral_no_update_accuracy | 1.0000 | +0.0000 |
| update_rate | 0.6674 | +0.0000 |
Concrete problems found
- u_016_turn_04
missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy不要再用误会梗了
- u_020_turn_02
missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy不要再用误会梗了
- u_022_turn_03
missed_dimensions=female_agency,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy这段不错，继续这个张力
- u_022_turn_07
missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy不要再用误会梗了
- u_032_turn_03
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,retriever.policy这段不错，继续这个张力
Top failure examples
- u_016_turn_04 — missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy
不要再用误会梗了 - u_020_turn_02 — missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy
不要再用误会梗了 - u_022_turn_03 — missed_dimensions=female_agency,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy
这段不错，继续这个张力 - u_022_turn_07 — missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy
不要再用误会梗了 - u_032_turn_03 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,retriever.policy
这段不错，继续这个张力
Why zero-valued metrics appear
- overgeneralization_rate = 0 是好信号，表示本轮没有把局部反馈错误泛化为全局偏好。
Research-state update
本轮将失败模式写回下一轮方法假设：Use future-probe-aware reranking and hard anti-overgeneralization probes; keep AHEAD disabled.