Self-Evolve / Feedback-to-State
Round 2/10
vs previous: Round 1
patch source: main-agent autodiagnosed
Round 2: implicit calibrated updater
Autoresearch loop: 主 agent 读取数据和指标 → 诊断具体问题 → 决定 patch → 改代码/评测 → 生成本轮可视化。当前小说场景 AHEAD intentionally disabled。
Idea tested
Fuse comments with fast-swipe, dwell z-score, and continue-next signals.
Diagnosed issues
Self-evolve round 2 dominant issue: scope_overgeneralization.
future_probe_win_rate changed +0.0119 vs previous round.
Selected next patch
Add verifier to distinguish current-story/current-arc updates from durable global user preferences.
Data used this round
PIF-Bench synthetic novel feedback trajectories
One row per user-turn: hidden profile, content features, explicit feedback, implicit feedback, gold preference update, gold agent-state diff, future probe.
sample_count: 466 · splits: {"feedback_rows": 400, "neutral_no_update_rows": 66}
Evaluation protocol
Run the round-specific updater on every feedback event, compare predicted dimensions/targets/scope with gold state diffs, then estimate future-probe success.
Metric definitions
| Metric | Definition |
|---|---|
dimension_f1 | Macro F1 between predicted preference dimensions and gold dimensions. |
target_f1 | Macro F1 between predicted agent-state targets and gold targets. |
future_probe_win_rate | Mean synthetic win probability on near/far/anti-overgeneralization future probes. |
overgeneralization_rate | Fraction of predicted updates that incorrectly globalize local/story feedback; zero is good. |
neutral_no_update_accuracy | Accuracy on neutral/no-gold-update rows; replaces the confusing no_update_precision metric. |
Metrics vs previous
| Metric | Value | vs previous |
|---|---|---|
| dimension_f1 | 0.8280 | +0.0284 |
| target_f1 | 0.2532 | +0.0000 |
| update_target_f1 | 0.2532 | +0.0000 |
| future_probe_win_rate | 0.6635 | +0.0119 |
| overgeneralization_rate | 0.6650 | +0.0000 |
| feedback_incorporation_rate | 0.1300 | +0.0000 |
| neutral_no_update_accuracy | 1.0000 | +0.0000 |
| update_rate | 0.8584 | +0.0000 |
Concrete problems found
- u_004_turn_09
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
- u_015_turn_09
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
- u_016_turn_07
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
- u_026_turn_07
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
- u_028_turn_01
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
Top failure examples
- u_004_turn_09 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定 - u_015_turn_09 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定 - u_016_turn_07 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定 - u_026_turn_07 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定 - u_028_turn_01 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定
Why zero-valued metrics appear
本轮没有需要特别解释的 0 指标。
Research-state update
本轮将失败模式写回下一轮方法假设：Add verifier to distinguish current-story/current-arc updates from durable global user preferences.