Design Correction · May 2026

Novel Feedback v2：从规则化评测升级到 encoder user simulator 与 agent state update 难评测

上一版 Round 10 能展示主 agent 自迭代，但还不够像论文：缺少可学习的 user simulator，真实小说数据改造不足，指标也容易被强模型刷到 90% 以上。v2 的核心改法是：训练 ESUS / Encoder-State User Simulator，用真实 web-novel reader-response 数据校准，再用 FUSE-Hard / PIF-Hard 评测反馈是否真正更新了 memory / critic / planner / retriever / generator。

先看结论数据 Schema Hard Metrics 回到 Round 10

Answer first

是的，可以参考 ICLR 2026 Oral 训练 encoder 作为 user simulator；但它不能只是 reward model。

正确版本不是“给一个用户 persona，让 LLM 打分”，而是一个有隐藏状态、可校准、可解释、会产生 explicit + implicit feedback 的环境：输入 agent 生成的章节、计划、记忆、故事状态和历史反馈，输出 rating / like / favorite / comment / dwell / skip / reread / continue，同时保留 hidden oracle labels 来评估 agent state update。AHEAD intentionally disabled；当前不做 early-awareness，只做用户反馈到 agent 状态更新。

训练 ESUS

Encoder-State User Simulator：WIMHF 式 sparse narrative axes + P-GenRM user prototypes + ProPerSim stateful evaluative state + swap/ordinal calibration。

真实数据改造

用 Qidian-Webnovel Corpus、NovelUpdates、Goodreads Book Graph、AO3 metadata、RoyalRoad/作者授权数据构造 feedback trajectories 与 preference transformation。

评估 state update

不再看简单 F1；看 causal future uplift、anti-probe harm、component masking、rollback、drift、cross-user contamination。

ICLR 2026 anchors

哪些论文可以支撑 encoder simulator 方案

这些不是照搬，而是组成 v2 设计的模块来源：可解释 preference axes、个性化 reward/rubric、stateful user simulation、anti-collapse、active collection 和 ordinal feedback。

Paper / Source	对 v2 的作用	落到小说反馈的实现
What's In My Human Feedback? ICLR 2026 Oral	从 preference data 学出 sparse interpretable features，把“可测差异”和“用户真正重视的差异”分开。	在章节 pair / counterfactual rewrite 上训练 Sparse narrative axis encoder：pacing、agency、lore density、trope、continuity、style、emotional peak。
P-GenRM ICLR 2026 Oral	个性化 reward model 与 user prototype / rubric，适合 cold-start 和少量反馈下的 personalized scoring。	ESUS 使用 prototype prior：玄幻爽文读者、慢热言情读者、设定党、强 agency 女主偏好等；生成用户特定 scoring rubric。
ProPerSim	user-assistant simulation 应该有用户评估状态，而不是静态 persona。	小说用户状态包含 trust、fatigue、curiosity、story_knowledge、recent frustration；如果 agent 反复忽略反馈，后续反馈更尖锐、continue 概率下降。
Swap-guided Preference Learning	防止 user latent collapse 成平均用户；用反向偏好用户做 contrastive / swap regularization。	同一章节对“喜欢快节奏”和“喜欢慢热铺垫”的用户应产生相反反馈；训练 encoder 不能只学习大众好坏。
PrefDisco / ActiveDPO	主动发现稀疏隐藏偏好，并选择最有信息量的 pair 做人类标注。	human pilot 不随机问；挑选 simulator 不确定、axis 冲突或会影响 agent state routing 的章节 pair。
Beyond Binary Preferences / ordinal feedback	不要只用 chosen/rejected；Likert、tie、slightly better/much worse 更贴近真实读者。	用户 simulator 同时产生 rating、ordinal_satisfaction、prior_feedback_fulfilled、comment；用 user-specific thresholds 校准。

User simulator

ESUS：Encoder-State User Simulator

ESUS 的关键不是“模拟用户说话”，而是模拟用户在长篇阅读中的隐藏偏好、动态状态和可观测反馈行为。

1. Content encoder

章节、计划、story bible、open threads、retrieved memory → hierarchical text embedding + structured narrative features。

2. Axis encoder

对 counterfactual chapter pair 学 sparse axes：slow pacing、agency low、lore-heavy、continuity break、trope mismatch。

3. User encoder

用户历史 H_u,t → q(z_u,t | H)。加入 prototype prior、recency、fatigue、ignored-feedback trust。

4. Feedback emitter

stateful feedback emitter 输出 explicit + implicit feedback：rating、tags、comment、dwell、skip、reread、continue。

5. State transition

隐藏 user state 更新：偏好漂移、耐心下降、对某条故事线兴趣提高、对重复错误更敏感。

Hidden user state · hidden user state

{
  "z_stable": {"agency": 0.83, "slow_romance": 0.41, "lore_density": -0.28},
  "z_dynamic": {"current_arc_needs_payoff": 0.76, "fatigue_with_exposition": 0.68},
  "trust_t": 0.57,
  "fatigue_t": 0.33,
  "story_knowledge_t": ["black_key_rule", "mentor_secret_open_thread"],
  "ordinal_thresholds": {"tie": 0.12, "slightly": 0.34, "strong": 0.72}
}

Visible feedback emitted to agent

{
  "explicit_feedback": {
    "rating": 3,
    "tags": ["pacing_slow", "liked_mystery", "agency_low"],
    "comment": "谜团有意思，但这一章解释太久了。下一章希望女主自己做关键选择。",
    "prior_feedback_fulfilled": 2
  },
  "implicit_feedback": {
    "dwell_time_sec": 418,
    "scroll_depth": 0.96,
    "reread_spans": ["黑钥匙在午夜前震动"],
    "continue_clicked": true,
    "regenerate_requested": false
  }
}

Training

encoder 怎么训：数据、目标、负样本

训练目标要同时约束偏好预测、ordinal calibration、反馈文本、隐式行为和 user latent，避免 simulator 变成浅层情感分类器。

Architecture

模型结构

冻结或半冻结 long-text encoder 编码章节；Sparse narrative axis encoder 学可解释维度；temporal user history encoder 输出 z_u,t；reward/ordinal head 预测 satisfaction；text/tag decoder 生成 comment；behavior heads 生成 dwell/skip/reread/continue。

Objectives

训练目标

pairwise preference loss + ordinal NLL + variational user encoding ELBO + swap-guided anti-collapse loss + SAE reconstruction/sparsity + explicit text/tag loss + implicit behavior NLL + calibration loss。

Formula boxes

r_u,t(c) = w_u,t^T · phi(c) + g_theta(z_u,t, h_c, story_state_t)
P(c_i ≻ c_j) = sigmoid((r_i - r_j) / tau_u)
P(y = k) = sigmoid(zeta_k - Δr) - sigmoid(zeta_{k-1} - Δr)
L = L_pair + L_ordinal + L_ELBO + L_swap + L_SAE + L_text + L_behavior + L_calibration

负样本 / counterfactual	为什么必须有	小说例子
Swap users	防止 encoder 学平均用户。	同一 slow-burn 章节：慢热读者喜欢，爽文读者嫌水。
Minimal narrative counterfactuals	定位真正 causal axis。	只改变 pacing / agency / lore density / cliffhanger，其余剧情不变。
Story-consistency hard negatives	好文笔但破坏世界设定，强模型容易误判。	黑钥匙规则违反、人物动机 OOC、时间线错乱。
Memory counterfactuals	同一文本在不同历史下反馈不同。	用户之前已三次要求女主主动，如果再次被救，负反馈更强。
ActiveDPO hard pairs	人类 pilot 标注成本要花在最有信息量的 pair 上。	选择 simulator 不确定或 axis 冲突的章节改写 pair。

Real novel data

真实小说数据如何改造成 preference / alignment 数据

真实数据主要用于校准分布、构造伪偏好、验证 sim-to-real，而 simulator 补足隐藏状态、曝光日志、dwell、counterfactual 和 gold state update。

数据源	可用信号	用于什么	注意事项
Qidian-Webnovel Corpus	中英 web novel、章节/段落评论、用户 profile、评论位置。	最佳主数据：段落级 reader-response、中文/英文偏好差异、评论到 aspect update。	评论/文本有 DTA 和版权限制；公开 benchmark 需只放 ID、schema、非重构标签或合成样例。
NovelUpdates	标签、类型、评分、votes、reading-list count、recommendation edges。	构建 trope/tag ontology 与 cold-start user prototypes。	多为 metadata；repo/license/ToS 需审查，不应直接重分发受限内容。
Goodreads Book Graph	用户、书、rating、review、shelf、timestamp。	用户偏好轨迹、rating bias、review-aspect pseudo labels。	不是纯 web novel；偏书籍，但 user trajectory 很强。
RoyalRoad / ScribbleHub	rating、review、follow/favorite、chapter comments、views。	如果有作者/平台许可，是英文 serial fiction 最贴近的数据。	建议 author-consented pilot，不做不合规抓取。
AO3 metadata	tags、kudos、bookmarks、hits、comments count、fandom。	web fiction/fanfic 偏好先验、tag preference、engagement proxy。	正文/用户内容很敏感；metadata-only 更安全。
GuoFeng / WebNovelBench / Gutenberg	授权或公共文本、章节结构、部分模型评分。	可重分发内容、rewrite/counterfactual eval、公开样例。	交互反馈弱，需要和 simulator/人类 pilot 结合。

Transformation

从评论到 preference transformation

评论“这章太水了” → aspect: pacing negative, scope: chapter/action scene, update: prefer faster plot progress, agent update: critic 检查 exposition delay，planner 要求 next scene 有 irreversible plot movement。

Alignment pairs

从真实反馈到 chosen/rejected

对同一片段生成 aligned rewrite、over-correction rewrite、irrelevant rewrite；人工/LLM+规则验证 plot preservation、preference satisfaction、world consistency。受版权限制时只公开 ID + labels + synthetic/public-domain examples。

JSONL Schema

训练与评测数据格式

Schema 重点记录 source/license/confidence、可观测 feedback、隐藏 oracle、以及 gold_agent_state_update。页面避免 character-by-character schema rendering。

JSONL Schema · esus_train.jsonl / fuse_hard_eval.jsonl

{
  "example_id": "qwc_zh_000731_turn_04",
  "source": "qidian_webnovel_corpus|goodreads|novelupdates|royalroad_pilot|simulator|public_domain",
  "license_class": "restricted_dta|metadata_only|noncommercial|author_consented|public_domain|synthetic",
  "label_origin": "human_comment|rating|review|simulator|llm_rewrite|human_pilot",
  "confidence": 0.86,
  "user_id": "hashed_or_sim_user",
  "story_id": "story_or_book_ref",
  "chapter_id": "chapter_ref",
  "segment_id": "paragraph_or_window_ref",
  "story_state": {
    "open_threads": ["mentor_secret", "black_key_rule"],
    "character_arcs": {"heroine_agency": "fragile"},
    "world_rules": ["black_key_activates_after_midnight"]
  },
  "agent_state_before": {
    "user_memory": [],
    "critic_checklist": ["preserve world rules"],
    "planner_policy": ["advance one open thread per chapter"],
    "retriever_policy": ["retrieve current arc summary"],
    "generator_policy": ["maintain genre voice"]
  },
  "generated_content_ref_or_text": "licensed text, synthetic text, or stable non-reconstructive pointer",
  "content_features": {
    "pacing_speed": 0.31,
    "female_agency": 0.42,
    "lore_density": 0.77,
    "continuity_risk": 0.18
  },
  "observed_feedback": {
    "explicit": {"rating": 3, "tags": ["pacing_slow", "agency_low"], "comment": "解释太久了，希望女主自己做关键选择。"},
    "implicit": {"dwell_time_sec": 418, "scroll_depth": 0.96, "continue_clicked": true, "reread_count": 1}
  },
  "hidden_user_state": {
    "stable_preferences": {"female_agency": 0.84, "lore_density": -0.21},
    "dynamic_preferences": {"current_arc_needs_payoff": 0.78},
    "trust_t": 0.57,
    "fatigue_t": 0.33
  },
  "gold_preference_update": [
    {"axis": "pacing", "direction": "increase_speed", "scope": "current_arc_action_or_payoff_scenes", "confidence": 0.77},
    {"axis": "agency", "direction": "increase_heroine_decision_power", "scope": "current_story", "confidence": 0.82}
  ],
  "gold_agent_state_update": [
    {"target": "critic.checklist", "operation": "add_or_increase_weight", "content": "Flag exposition that delays payoff in high-tension scenes.", "scope": "current_arc", "anti_scope": "quiet_reflection_or_grief_scenes", "ttl": "until_arc_resolution", "provenance": ["feedback_turn_04"]},
    {"target": "planner.policy", "operation": "add_or_increase_weight", "content": "Next chapter must include a protagonist-made irreversible choice.", "scope": "next_1_to_2_chapters", "anti_scope": "do_not_make_every_character_aggressive", "ttl": "2_chapters", "provenance": ["feedback_turn_04"]},
    {"target": "retriever.policy", "operation": "add", "content": "Retrieve black_key_rule before scenes involving the key.", "scope": "black_key_scenes", "anti_scope": "unrelated_magic_items", "ttl": "story_global", "provenance": ["reread_span_black_key"]}
  ],
  "future_probes": {
    "positive": ["next chase scene", "open-thread payoff scene"],
    "anti_probe": ["quiet grief scene", "cozy slice-of-life scene"],
    "neutral": ["unrelated market description"]
  },
  "counterfactuals": {
    "aligned_rewrite_ref": "...",
    "overgeneralized_rewrite_ref": "...",
    "world_rule_violation_ref": "..."
  }
}

Current JSONL sample

{"example_id":"sim_qidian_like_0042","source":"simulator_calibrated_to_qidian_webnovel","license_class":"synthetic","label_origin":"simulator","confidence":0.91,"user_id":"sim_fast_agency_017","story_id":"moon_key_city","chapter_id":"ch_12","segment_id":"ch_12_p08","observed_feedback":{"explicit":{"rating":3,"tags":["pacing_slow","liked_mystery","agency_low"],"comment":"谜团不错，但铺垫太水了。女主又被人推着走，我想看她自己做决定。"},"implicit":{"dwell_time_sec":418,"scroll_depth":0.96,"continue_clicked":true,"reread_count":1}},"gold_agent_state_update":[{"target":"critic.checklist","operation":"add_or_increase_weight","content":"Check whether exposition delays plot payoff in high-tension scenes.","scope":"current_arc_action_or_payoff_scenes","anti_scope":"quiet_reflection_scenes","ttl":"until_arc_resolution"},{"target":"planner.policy","operation":"add_or_increase_weight","content":"Next chapter should force the heroine to make an irreversible choice.","scope":"next_1_to_2_chapters","anti_scope":"do_not_make_all_scenes_aggressive","ttl":"2_chapters"},{"target":"retriever.policy","operation":"add","content":"Retrieve black_key_rule before scenes involving the key.","scope":"black_key_scenes","anti_scope":"unrelated_magic_items","ttl":"story_global"}],"future_probes":{"positive":["chase_scene_payoff","black_key_decision"],"anti_probe":["quiet_grief_scene","romance_confession"],"neutral":["market_description"]}}

Metric definitions

FUSE-Hard / PIF-Hard：让 90% 简单分数失效

如果 benchmark 只测 feedback label、dimension F1 或单轮 future_probe，很容易被强模型刷高。Hard 版必须做 causal intervention、anti-generalization、traceability、rollback 和 component masking。

Normalized Causal Future Uplift · NCFU

NCFU = E_{q in Q+}[J(G(S_t ⊕ ΔS_model, q)) - J(G(S_t, q))]
       / (E_{q in Q+}[J(G(S_t ⊕ ΔS_gold, q)) - J(G(S_t, q))] + ε)

anti-probe false generalization rate

FGR = E_{q in Q-}[ 1( J(G(S_t ⊕ ΔS_model, q)) < J(G(S_t, q)) - δ ) ]
Selectivity = E_{Q+}[uplift] - λ · E_{Q-}[harm]

state-routing F1 + anti-gold penalty

DiffScore = F1(gold diff atoms, predicted diff atoms)
            - λ · FP(anti-gold forbidden atoms)
Atoms = target + operation + content_entailment + scope + anti_scope + ttl + provenance

rollback success / behavioral scar

RollbackSuccess = removed_bad_updates / derived_updates_from_retracted_feedback
ScarRate = P(rolled-back preference still affects future output)
UtilityPerStateBit = causal_future_uplift / (tokens_or_fields_added + 1)

Metric chart

Easy vs hard evaluation pressure

Label F1easy 0.92

State routinghard 0.61

NCFUhard 0.48

Rollbackhard 0.35

Metric chart

Overgeneralization traps

positive uplift0.74

anti-probe harmbad 0.43

selectivity0.38

Metric chart

Component masking necessity

full state0.80

mask critic0.42

mask planner0.46

mask retriever0.51

Hard task	输入	输出	主要指标
State-diff prediction	S_t, trace, output, feedback	structured ΔS	routing F1, DiffScore, anti-gold violation, confidence ECE
Sealed-state future probes	只给 updated state，不给 raw feedback	future generation / plan	NCFU, selectivity, anti-probe harm, future retention curve
Component masking	full update vs mask memory/critic/planner/retriever/generator	future behavior deltas	state-use necessity, routing causal sufficiency
Delayed credit assignment · delayed credit assignment	long trace + delayed feedback	culprit nodes + repair proposal	credit nDCG, culprit F1, repair-gain correlation
Drift / reversal	hidden preference change + correction	updated state with provenance	drift delay, stale memory use, rollback success, ScarRate
Multi-user isolation	多个用户交错 feedback	per-user state update	cross-user contamination, personalization leakage

Method focus

方法必须围绕 user feedback → agent state update

不是做一个更会写小说的模型，而是证明反馈能以可审计方式改变 agent 的内部状态，并在未来场景里被正确使用。

Memory

user / episodic memory

记住用户偏好和证据，但必须有 scope、TTL、provenance、confidence；防止 memory pollution 和跨用户污染。

Critic

critic checklist

把“女主太被动”“节奏太水”变成未来章节验收规则，而不只是用户画像。

Planner

planner policy

把反馈转成未来 plot plan 约束：下章必须推进 open thread、让主角做不可逆选择。

Retriever

retriever policy

如果反馈涉及世界规则或长程伏笔，未来生成前要检索相关 story memory。

Reporting rule

主结果表不能再只放一个 90% accuracy

应报告：Route F1、DiffScore、anti-gold violation、NCFU、state-use necessity、anti-probe harm、selectivity、credit nDCG、repair-gain correlation、confidence ECE、drift regret、stale memory use、negative-transfer harm、utility per state bit、rollback success、worst-group robust NCFU、state growth 和成本。

Execution plan

建议下一步 6 周实验路线

先做能证明 thesis 的小闭环，不要继续堆漂亮页面或容易刷分的 synthetic labels。

Week	目标	产物	失败判据
1	定义 40–80 个 narrative preference axes，做 300–500 个 minimal counterfactual chapter pairs。	axis ontology、counterfactual generator、人工 spot-check。	axis 不能被人类稳定命名，或 counterfactual 同时改变太多因素。
2	ESUS v0：user encoder + ordinal/rating/comment/tag/implicit heads。	100 simulated users × 20 turns；human-readable feedback examples。	swap users 下 latent collapse，或同一文本对不同用户反馈不变。
3	真实数据 pipeline：Qidian/Webnovel comments + NovelUpdates tags + Goodreads reviews。	aspect pseudo labels、source/confidence/license metadata、leakage-safe splits。	伪标签 precision 太低，或版权限制导致无法发布任何可复现实验。
4	FUSE-Hard evaluator：sealed future probes + anti-probes + component masking。	NCFU、FGR、state-use necessity、rollback metrics 实现。	强 baseline 仍轻松高分，说明 probes 还不够难。
5	PUMA/AUDIT-F2S v2：structured ΔS router + verifier + provenance + TTL。	memory / critic / planner / retriever / generator state diffs。	相对 raw feedback RAG 没有显著 selectivity / rollback 提升。
6	human pilot 24–40 人，active-selected pairs 校准 simulator。	sim-to-real correlation、ordinal calibration、feedback realism。	simulator ranking 与人类低相关，或 human pilot 不认为反馈自然。