Self-Evolve / Feedback-to-State
Round 3/10
vs previous: Round 2
patch source: main-agent autodiagnosed
Round 3: state-diff target router
Autoresearch loop: 主 agent 读取数据和指标 → 诊断具体问题 → 决定 patch → 改代码/评测 → 生成本轮可视化。当前小说场景 AHEAD intentionally disabled。
Idea tested
Map inferred dimensions to concrete agent-state targets: critic, planner, retriever, reranker, memory.
Diagnosed issues
Self-evolve round 3 dominant issue: scope_overgeneralization.
target_f1 changed +0.3679 vs previous round.
Selected next patch
Add verifier to distinguish current-story/current-arc updates from durable global user preferences.
Data used this round
PIF-Bench synthetic novel feedback trajectories
One row per user-turn: hidden profile, content features, explicit feedback, implicit feedback, gold preference update, gold agent-state diff, future probe.
sample_count: 466 · splits: {"feedback_rows": 400, "neutral_no_update_rows": 66}
Evaluation protocol
Run the round-specific updater on every feedback event, compare predicted dimensions/targets/scope with gold state diffs, then estimate future-probe success.
Metric definitions
| Metric | Definition |
|---|---|
dimension_f1 | Macro F1 between predicted preference dimensions and gold dimensions. |
target_f1 | Macro F1 between predicted agent-state targets and gold targets. |
future_probe_win_rate | Mean synthetic win probability on near/far/anti-overgeneralization future probes. |
overgeneralization_rate | Fraction of predicted updates that incorrectly globalize local/story feedback; zero is good. |
neutral_no_update_accuracy | Accuracy on neutral/no-gold-update rows; replaces the confusing no_update_precision metric. |
Metrics vs previous
| Metric | Value | vs previous |
|---|---|---|
| dimension_f1 | 0.7165 | -0.1115 |
| target_f1 | 0.6211 | +0.3679 |
| update_target_f1 | 0.6211 | +0.3679 |
| future_probe_win_rate | 0.7061 | +0.0426 |
| overgeneralization_rate | 0.8553 | +0.1903 |
| feedback_incorporation_rate | 0.7775 | +0.6475 |
| neutral_no_update_accuracy | 1.0000 | +0.0000 |
| update_rate | 0.6674 | -0.1910 |
Concrete problems found
- u_022_turn_03
missed_dimensions=female_agency,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy, overgeneralized_scope这段不错，继续这个张力
- u_032_turn_03
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,retriever.policy, overgeneralized_scope这段不错，继续这个张力
- u_033_turn_10
missed_dimensions=female_agency,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy, overgeneralized_scope这段不错，继续这个张力
- u_034_turn_02
missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy, overgeneralized_scope这段不错，继续这个张力
- u_004_turn_09
missed_dimensions=lore_density,pacing, missed_targets=planner.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
Top failure examples
- u_022_turn_03 — missed_dimensions=female_agency,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy, overgeneralized_scope
这段不错，继续这个张力 - u_032_turn_03 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,retriever.policy, overgeneralized_scope
这段不错，继续这个张力 - u_033_turn_10 — missed_dimensions=female_agency,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy, overgeneralized_scope
这段不错，继续这个张力 - u_034_turn_02 — missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy, overgeneralized_scope
这段不错，继续这个张力 - u_004_turn_09 — missed_dimensions=lore_density,pacing, missed_targets=planner.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定
Why zero-valued metrics appear
本轮没有需要特别解释的 0 指标。
Research-state update
本轮将失败模式写回下一轮方法假设：Add verifier to distinguish current-story/current-arc updates from durable global user preferences.