Self-Evolve / Feedback-to-State
Round 1/10
vs previous: baseline round
patch source: main-agent autodiagnosed
Round 1: explicit-only baseline
Autoresearch loop: 主 agent 读取数据和指标 → 诊断具体问题 → 决定 patch → 改代码/评测 → 生成本轮可视化。当前小说场景 AHEAD intentionally disabled。
Idea tested
Read only comments/ratings and write coarse memory updates.
Diagnosed issues
Self-evolve round 1 dominant issue: implicit_feedback_blind_spots.
Baseline round; no previous round exists yet.
Selected next patch
Add dwell/fast-swipe/continue calibration and content-feature evidence for silent reader dissatisfaction.
Data used this round
PIF-Bench synthetic novel feedback trajectories
One row per user-turn: hidden profile, content features, explicit feedback, implicit feedback, gold preference update, gold agent-state diff, future probe.
sample_count: 466 · splits: {"feedback_rows": 400, "neutral_no_update_rows": 66}
Evaluation protocol
Run the round-specific updater on every feedback event, compare predicted dimensions/targets/scope with gold state diffs, then estimate future-probe success.
Metric definitions
| Metric | Definition |
|---|---|
dimension_f1 | Macro F1 between predicted preference dimensions and gold dimensions. |
target_f1 | Macro F1 between predicted agent-state targets and gold targets. |
future_probe_win_rate | Mean synthetic win probability on near/far/anti-overgeneralization future probes. |
overgeneralization_rate | Fraction of predicted updates that incorrectly globalize local/story feedback; zero is good. |
neutral_no_update_accuracy | Accuracy on neutral/no-gold-update rows; replaces the confusing no_update_precision metric. |
Metrics vs previous
| Metric | Value | vs previous |
|---|---|---|
| dimension_f1 | 0.7996 | baseline |
| target_f1 | 0.2532 | baseline |
| update_target_f1 | 0.2532 | baseline |
| future_probe_win_rate | 0.6516 | baseline |
| overgeneralization_rate | 0.6650 | baseline |
| feedback_incorporation_rate | 0.1300 | baseline |
| neutral_no_update_accuracy | 1.0000 | baseline |
| update_rate | 0.8584 | baseline |
Concrete problems found
- u_000_turn_09
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
- u_004_turn_06
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
- u_004_turn_09
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
- u_005_turn_06
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
- u_006_turn_01
missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope女主有点被动，想看她自己做决定
Top failure examples
- u_000_turn_09 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定 - u_004_turn_06 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定 - u_004_turn_09 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定 - u_005_turn_06 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定 - u_006_turn_01 — missed_dimensions=lore_density,pacing, missed_targets=critic.checklist,planner.policy,reranker.policy,retriever.policy, extra_targets=memory.user, overgeneralized_scope
女主有点被动，想看她自己做决定
Why zero-valued metrics appear
本轮没有需要特别解释的 0 指标。
Research-state update
本轮将失败模式写回下一轮方法假设：Add dwell/fast-swipe/continue calibration and content-feature evidence for silent reader dissatisfaction.