Self-Evolve / Feedback-to-State
Round 4/10
vs previous: Round 3
patch source: main-agent autodiagnosed
Round 4: scope verifier
Autoresearch loop: 主 agent 读取数据和指标 → 诊断具体问题 → 决定 patch → 改代码/评测 → 生成本轮可视化。当前小说场景 AHEAD intentionally disabled。
Idea tested
Separate current-story/current-arc updates from durable global-user preferences.
Diagnosed issues
Self-evolve round 4 dominant issue: framework_target_gap_after_scope_fix.
overgeneralization_rate changed -0.8553 vs previous round.
Selected next patch
Run a verifier pass that re-routes each surviving dimension to all primary framework targets.
Data used this round
PIF-Bench synthetic novel feedback trajectories
One row per user-turn: hidden profile, content features, explicit feedback, implicit feedback, gold preference update, gold agent-state diff, future probe.
sample_count: 466 · splits: {"feedback_rows": 400, "neutral_no_update_rows": 66}
Evaluation protocol
Run the round-specific updater on every feedback event, compare predicted dimensions/targets/scope with gold state diffs, then estimate future-probe success.
Metric definitions
| Metric | Definition |
|---|---|
dimension_f1 | Macro F1 between predicted preference dimensions and gold dimensions. |
target_f1 | Macro F1 between predicted agent-state targets and gold targets. |
future_probe_win_rate | Mean synthetic win probability on near/far/anti-overgeneralization future probes. |
overgeneralization_rate | Fraction of predicted updates that incorrectly globalize local/story feedback; zero is good. |
neutral_no_update_accuracy | Accuracy on neutral/no-gold-update rows; replaces the confusing no_update_precision metric. |
Metrics vs previous
| Metric | Value | vs previous |
|---|---|---|
| dimension_f1 | 0.7165 | +0.0000 |
| target_f1 | 0.6211 | +0.0000 |
| update_target_f1 | 0.6211 | +0.0000 |
| future_probe_win_rate | 0.7977 | +0.0916 |
| overgeneralization_rate | 0.0000 | -0.8553 |
| feedback_incorporation_rate | 0.7775 | +0.0000 |
| neutral_no_update_accuracy | 1.0000 | +0.0000 |
| update_rate | 0.6674 | +0.0000 |
Concrete problems found
- u_016_turn_04
missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy, extra_targets=memory.user不要再用误会梗了
- u_020_turn_02
missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy, extra_targets=memory.user不要再用误会梗了
- u_022_turn_07
missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy, extra_targets=memory.user不要再用误会梗了
- u_022_turn_03
missed_dimensions=female_agency,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy这段不错，继续这个张力
- u_032_turn_03
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,retriever.policy这段不错，继续这个张力
Top failure examples
- u_016_turn_04 — missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy, extra_targets=memory.user
不要再用误会梗了 - u_020_turn_02 — missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy, extra_targets=memory.user
不要再用误会梗了 - u_022_turn_07 — missed_dimensions=female_agency,lore_density, missed_targets=critic.checklist,reranker.policy,retriever.policy, extra_targets=memory.user
不要再用误会梗了 - u_022_turn_03 — missed_dimensions=female_agency,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy
这段不错，继续这个张力 - u_032_turn_03 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,retriever.policy
这段不错，继续这个张力
Why zero-valued metrics appear
- overgeneralization_rate = 0 是好信号，表示本轮没有把局部反馈错误泛化为全局偏好。
Research-state update
本轮将失败模式写回下一轮方法假设：Run a verifier pass that re-routes each surviving dimension to all primary framework targets.