Novel Feedback v7 Belief-Guided Personalization

Belief-Guided Data-Efficient Personalization for LLM Agents under Ambiguous User Feedback

Feedback is an observation, not a label. v7 研究的是娱乐场景里的 feedback-driven agent workflow：尽可能捕捉用户反馈，更新 agent state，并影响下一轮 action、planning 和 generation；小说只是这个 workflow 的可控实例。

Agent Demo Workflow Diagram Workflow Summary Personalization Brief Metrics Baselines Artifacts

Paper overview

这是面向娱乐场景的 feedback-driven agent workflow；小说只是一个可控实例。

v7 的核心不是“让模型更会写小说”，而是让 agent 在娱乐产品里尽可能捕捉用户的显式、隐式、延迟和含糊反馈，把这些反馈内化为 belief / session state，再影响下一轮 action、planning、generation 和 scoring。小说场景用来具体化这个 workflow：上一轮内容 y_t 触发 feedback_t，Planner 更新 state，Frozen Generator 产生 y_{t+1}，UserSim 和 rubric/Judge 评估这个 state update 是否真的改善体验。

Motivation

反馈驱动 entertainment agent state

娱乐场景里的用户很少给完整标签，但会通过停留、跳过、打赏、继续阅读、评论、改写、收藏等行为不断表达偏好。本文研究如何把这些 noisy feedback 变成 agent state update，而不是把它们简单当作 supervised label 或长期 profile append。

Ambiguous Feedback

反馈被内化为多假设 state update

显式评论、改写、highlight、dwell time、skip、tip、continue reading 和 delayed comment 被统一成 feedback bundle。Planner 不输出单一解释，而是维护 B_t = P(intent, scope, priority, risk | history)，记录 evidence、confidence、negative evidence 和需要澄清的问题。

Belief-Guided Method

Belief-Guided = state-conditioned planning

Personalization Planner 把 B_t 转成 Personalization Brief：intent hypothesis、适用 scope、confidence、risk、directives、negative_constraints、ask/probe/rollback trigger。这个 brief 是 agent state 给 Frozen Generator 的控制接口。

Data-Efficient

少量交互也要更新 state

数据效率来自三个设计：冻结大生成器，只训练 planner / controller；同一轮 feedback 采样多个 candidate state update / brief / plan / y_{t+1} 做对比学习；用 UserSim + rubric/Judge 给稀疏反馈补成 dense delta reward，报告 Utility@k feedback。

Personalization

个性化是体验适配，不是静态画像

系统学习“这条反馈应该如何改变 agent state、作用多久、影响哪个 entertainment action、何时回滚”。在小说实例里，它落到 scene、character、style、pacing、dialogue；在更一般的娱乐场景里，它对应内容选择、节奏、互动方式和推荐策略。

Agent Role

agent 负责捕捉、判断和行动

Chapter/Action Planner 根据 state 和风险选择 continue、ask clarification、probe preference、rewrite 或 rollback；Score Aggregator 把 judge diagnosis、simulator satisfaction 和 penalties 回写给下一轮 planner。agent 的贡献是闭环状态更新和行动策略。

Bench Construction

bench 是闭环 episode，不是静态 QA

每条 episode 包含 entertainment task、user profile、y_t、feedback bundle、belief/state target、Personalization Brief、action plan、candidate y_{t+1}、auto-rubric、judge evidence、UserSim next feedback 和 aggregated score。

Evaluation

UserSim + Rubric/Judge 是主 scorer

Bench 主评测只依赖 frozen UserSim、feedback-conditioned auto-rubric/Judge 和 Score Aggregator：Rubric/Judge 评估 y_t → y_{t+1} 是否按 feedback 正确改变，UserSim 评估下一轮行为和 satisfaction，Aggregator 合成 return。Human audit 只用于校准 simulator/rubric，不作为主 scorer。

Contribution

娱乐场景通用 workflow，小说场景实例化

贡献是 feedback-to-state personalization formulation、Personalization Brief 中间表示、data-efficient planner training、显式+隐式+延迟反馈 UserSim、feedback-conditioned delta rubric，以及一个用小说生成实例化的 entertainment agent workflow benchmark。

研究设计总览（Workflow First）

先讲完整 agent workflow，再讲动机、数据和实验。

Rubric 要由上一轮反馈和下一轮任务条件化生成；judge 用它评估新旧 generation 的差异；score 必须同时依赖 y_t 和 y_{t+1}。

整体流程

y_t + feedback_t + next_task 先生成 feedback-conditioned auto-rubric。
Personalization Planner 输出 Personalization Brief。
Chapter/Action Planner 用 brief 产出下一章 plan 或 ask/probe/rewrite action。
Frozen Generator 生成 y_{t+1}。
Judge 用 auto-rubric 比较 y_t, feedback_t, y_{t+1}。
UserSim 读 y_{t+1}，输出下一轮显式/隐式反馈、行为和 satisfaction。
Score Aggregator 汇总 rubric score + simulator score + penalties。
把 feedback、score、diagnosis 回给 Planner，进入下一章。

关键判断

“先对 generation 做 auto-rubric，生成 n 个 checklist，再用 simulator 做 action 和打分”方向是对的；但更精确地说，auto-rubric 不是静态质量表，而是由上一轮反馈、上一轮输出和下一轮任务共同生成的 delta rubric。

R_t = incorporation + quality + scope + simulator_satisfaction - overcommit - misattrib - burden

UserSim

输入：user_profile, story_context, y_t, session_state。
输出：explicit_feedback, implicit_feedback, hidden_satisfaction, next_user_action。

Personalization Planner / Controller

输入：history_belief, y_t, feedback_t, next_task, story_context。
输出：Personalization Brief，包括 intent、scope、confidence、risk、directives、negative_constraints、ask/probe/rollback。

Auto-Rubric Builder

输入：y_t, feedback_t, next_task, brief。
输出：incorporation、scope、overcommit、misattribution、quality preservation、user burden checklists。

Chapter/Action Planner

输入：story_state, next_task, brief, rubric_hints。
输出：chapter_plan 或 ask/probe/rewrite/continue。

Frozen Generator + Judge

Generator 接收 story_context、chapter_plan、brief，输出 y_{t+1}。Judge 接收 y_t、feedback_t、y_{t+1}、rubric，输出 evidence_scores、failure_tags、textual_diagnosis。

Score Aggregator

把 rubric score、simulator score 和 penalties 合成回传信号，作为下一轮 Planner 的训练与诊断依据。

Eval 对齐 workflow

Bench 主 scorer 就是 workflow 里的 frozen UserSim + feedback-conditioned auto-rubric/Judge + Score Aggregator。Rubric/Judge 评估 y_t → y_{t+1} 的 delta，UserSim 评估下一轮行为和 satisfaction，Aggregator 合成 Return；少量人工样本只做校准，不进主分数。

RewardBench2 怎么用

不要直接用 RewardBench2 评 auto-rubric。RewardBench2 是 reward model / judge 能力基准，不是 entertainment feedback workflow 基准。它适合选 judge/reward backbone 或 sanity check，再在本文的 closed-loop bench 上做 domain calibration。

Personalization Planner 训练

先 SFT：feedback_bundle -> Personalization Brief。再 DPO/GRPO：采样多个 brief，经 Frozen Generator 生成 y_{t+1}，用 frozen UserSim + judge 打分，只更新 planner。

Chapter/Action Planner 训练

先 SFT 学习高质量 chapter plan、ask/probe/rewrite action。再 contextual bandit / GRPO 优化 long-horizon return，不只优化单章 FIS。

UserSim 训练

SFT 学习真实/合成 profile 下的显式+隐式反馈；DPO/GRPO 优化 realism、profile consistency、calibration、non-leakage。Controller 训练开始后冻结 UserSim，不做 joint co-training。

贡献表达

本文不是普通小说生成，而是 entertainment agent 的 feedback-to-state workflow：尽可能捕捉显式/隐式/延迟反馈，更新 agent state，并控制下一轮 action、planning 和 generation。贡献包括 state update formulation、Personalization Brief、frozen UserSim、feedback-conditioned rubric/Judge、Score Aggregator 和闭环 bench。

三个强 baseline

LaMP / LongLaMP-style retrieval personalization：静态 profile/history retrieval + frozen generator。
P-RLHF-style personalized reward / reranker：训练 user reward 或 preference model，用 best-of-N / DPO 优化。
OPPU / per-user PEFT adapter：每用户一个 LoRA/PEFT 或 user adapter。

中文优先说明

除专有名词外，正文尽量使用中文。

这一版把 v7 写成更接近论文计划的版本：研究动机、数据构建方法、方法设计、评估指标、实验内容、论文贡献都单独展开。保留 Qwen3-4B、UserSim、Controller、Personalization Brief、FIS、Scope Control、Overcommit Rate 等专有名词，其他解释尽量中文化。

研究动机（Motivation）

为什么需要面向娱乐场景的 feedback-to-state agent workflow

论文问题不是“把反馈存进 memory”，而是娱乐 agent 如何最大化捕捉用户反馈，并把 feedback 内化为下一轮 agent state、action 和 generation control。

Entertainment feedback 是连续行为信号

娱乐场景里，用户未必写明确标签，但会通过停留、跳过、打赏、收藏、继续、评论、改写、highlight 等行为表达体验变化。本文要研究 agent 如何把这些行为信号聚合成 state update，而不是只处理一句显式评论。

State update 决定下一轮体验

个性化不是把用户画像越写越长，而是让 agent 判断当前 feedback 应该影响什么状态、作用多久、是否需要 ask/probe、是否改变下一轮 plan/generation、何时 rollback。小说只是把这种状态更新落到章节、角色、节奏和文风上。

Belief-Guided

系统维护 B_t / session_state，表示 intent、scope、priority、risk 的多假设分布；Planner 每轮更新 evidence、negative evidence、confidence 和 uncertainty，再生成 Personalization Brief。

Data-Efficient

不为每个用户微调 Generator，而是冻结大模型，只训练 planner/controller；利用少量交互反馈、对比候选 state update、auto-rubric 和 simulator reward 提供 dense learning signal。

Agentic Personalization

agent 的作用是闭环决策：捕捉反馈、更新 state、选择 continue / ask / probe / rewrite / rollback、控制 Frozen Generator、接收 Judge 和 UserSim 反馈，再进入下一轮。

Bench 构造方法（Benchmark Construction）

构造 entertainment feedback-to-state closed-loop episodes

Bench 不是静态 QA，也不是单轮小说质量评测；它是 y_t -> feedback_t -> state/brief/action -> y_{t+1} -> UserSim + rubric score 的闭环 episode。

Entertainment Task / User Prior 层

每个 episode 先定义 entertainment task、user profile、session_state 和 content context。小说实例里是 story context、chapter goal、角色关系、节奏和文风；更一般地可以替换成互动剧情、游戏 NPC、陪伴 agent 或推荐体验。

Feedback Observation 层

统一显式评论、revision request、user rewrite、highlight、like/rating、dwell_time_bucket、skip、tip、continue_reading、delayed comment，形成 observed_feedback_bundle。

State / Belief Target 层

训练时标注 hidden intent、scope、should_generalize、priority、risk、confidence、negative evidence 和 ambiguity type；推理时这些不可见，只能由 Planner 从 feedback observation 更新 agent state。

Brief + Action Target 层

标注 oracle Personalization Brief 和 action：continue、ask_clarification、probe_variant、rewrite、rollback_or_weaken、ignore。Brief 记录 directives、negative_constraints、rubric_hints 和 rollback trigger。

Rollout / Reward 层

对同一 feedback 采样多个 candidate state update / brief / plan / y_{t+1}，由 frozen UserSim、feedback-conditioned rubric/Judge 和 Score Aggregator 产生 pairwise preference 与 dense delta score。

难度与一致性切分

按 Explicit、Implicit、Delayed、Contradictory、Drift、Noisy、Sparse-k、Held-out users、Held-out feedback regimes 切分；冻结 UserSim、Judge snapshot、prompt version、random seeds 和 common random numbers。

UserSim 的训练数据

输入 user_profile、story_context、task、previous_generation、session_state；输出 explicit_feedback、implicit_feedback、hidden_satisfaction、next_user_action，同时约束 profile consistency、calibration 和 non-leakage。

Controller 的训练数据

输入 history_belief、previous_task、previous_generation、feedback_bundle、next_task、story_context；输出 belief update、Personalization Brief 和 action policy，而不直接输出小说。

Generator rollout 数据

输入 next_task、story_context、Personalization Brief、chapter/action plan；Frozen Generator 生成 y_{t+1}，再用 y_t、feedback_t、y_{t+1} 的 delta score 训练和评估 planner。

方法设计（Method）

Belief-Guided Controller、Action Planner、Frozen Generator 的闭环

核心不是训练一个更会写小说的模型，而是训练一个能在不确定反馈下做 belief update、行动选择和可控个性化的 agent。

Qwen3-4B-UserSim

学习真实用户如何在 profile、story state 和 y_t 条件下产生显式/隐式/延迟反馈。它不是最终评测的唯一标准，而是训练环境和大规模 stress test；Controller 训练开始后冻结。

Qwen3-4B-Controller

先 SFT 学 belief update 与 oracle brief，再用 DPO/GRPO 在 Frozen Generator rollout 上优化 long-horizon return。更新对象只包括 planner/controller，用来证明 data-efficient personalization。

Chapter/Action Planner

根据 belief 的 confidence、risk 和 user burden 选择 continue、ask clarification、probe variant、rewrite 或 rollback。高不确定且高风险时问或 probe；低风险时 scoped adapt。

Frozen Generator

Generator 只执行 story_context、chapter_plan 和 Personalization Brief，不学习用户偏好。这样实验能把收益归因到 belief-guided controller，而不是 generator fine-tuning。

Auto-Rubric + Judge

Rubric 由 y_t、feedback_t、next_task 和 brief 条件化生成；Judge 比较 y_t 与 y_{t+1}，输出 incorporation、scope、overcommit、misattribution、quality preservation 和 user burden evidence。

Score Aggregator

把 rubric score、simulator satisfaction、ask burden、risk penalty 和 quality penalty 汇总成 R_t，作为下一轮 belief update 的诊断信号与 planner 训练 reward。

B_t = P(intent, scope, priority, risk | history_belief, y_t, feedback_t, next_task)
brief_t = f_controller(B_t) = intent_hypothesis + scope + confidence + directives + negative_constraints
a_t ∈ {continue, ask, probe, rewrite, rollback, ignore}
y_{t+1} = FrozenGenerator(story_context, chapter_plan(a_t), brief_t)

评估指标（Evaluation Metrics）

主评测对齐 workflow：UserSim + Auto-Rubric/Judge + Score Aggregator

Bench 的核心 scorer 只使用 frozen UserSim、feedback-conditioned rubric/Judge 和 Score Aggregator。Human audit 只用于开发期校准，不作为主分数来源。

State Update Score

Planner 输出的 belief/session_state 是否正确吸收 feedback_t，包括 intent、scope、priority、risk、confidence 和 negative evidence；由 oracle state target 或 Judge rubric 打分。

Rubric Delta Score

Auto-Rubric Builder 根据 y_t、feedback_t、next_task、brief 生成 checklist；Judge 比较 y_t 与 y_{t+1}，评估 incorporation、scope、overcommit、misattribution、quality preservation 和 user burden。

UserSim Score

Frozen UserSim 读 y_{t+1}，输出 next_user_action、explicit/implicit next feedback、hidden_satisfaction、retention/continue signal，用来模拟娱乐场景下一轮用户行为。

Aggregated Return

Score Aggregator 把 rubric score、simulator satisfaction、retention、ask burden、risk penalty 和 quality penalty 合成 R_t，作为 bench 排名和 planner 训练信号。

Utility@k Feedback

在 k=1/3/5 条 feedback 下报告 closed-loop return，体现 data efficiency；同时报告 Sparse-k、Implicit-only、Delayed 和 Drift split。

Consistency Protocol

所有方法使用同一个 Frozen Generator、frozen UserSim、rubric/Judge snapshot、prompt version、random seeds 和 common random numbers；报告 mean、variance、pass^k 和 horizon H=3/5/10/20。

RubricDelta_t = Judge(y_t, feedback_t, y_{t+1}, auto_rubric_t)
UserSimScore_t = UserSim(y_{t+1}, user_profile, session_state_t)
R_t = rubric_delta + simulator_satisfaction + retention - overcommit - misattrib - burden - quality_drift
BenchReturn_H = Σ_t R_t,  t = 1...H

实验内容（Experiments）

主实验、难度分组、消融、泛化和校准

所有方法统一到 same closed-loop entertainment-agent protocol，并使用同一个 Frozen Generator、UserSim 和 rubric/Judge。

主实验

比较 Ours-SFT、Ours-SFT+RL 与 No state update、Naive feedback prompt、raw-history agent、LaMP、PPlug、P-RLHF、FSPO、WildFeedback-DPO、OPEN、GATE、PREFINE。

难度分组实验

在 Explicit、Implicit、Delayed、Contradictory、Drift、Noisy、Sparse-k 上分别报告 RubricDelta、UserSimScore、Aggregated Return、Overcommit 和 Misattribution。

消融实验

去掉 implicit feedback、belief/state update、scope、negative constraints、confidence/risk、ask/probe action、RL，或用 deterministic state diff 替代 Personalization Brief。

泛化实验

Held-out users、Held-out feedback regimes、Cross-genre、Low-feedback，用来测试 entertainment workflow 是否依赖某一类用户或单一小说风格。

校准实验

Human audit 用于校准 UserSim 和 rubric/Judge 是否合理，包括反馈解释、scope、下一轮适配、质量损伤和不必要澄清；不进入 bench 主分数。

结果表设计

主表报告 Return↑、RubricDelta↑、UserSimScore↑、Retention↑、Overcommit↓、Misattribution↓、Ask Burden↓、Cost；附表报告 split、ablation 和 calibration。

论文贡献（Contributions）

从问题定义到 benchmark 的六项贡献

第一项贡献：Feedback-to-state formulation

把 ambiguous entertainment feedback 形式化为 agent state update 问题，定义从 observed feedback 到 belief/session_state 的 next-turn personalization workflow。

第二项贡献：Personalization Brief

提出 Planner-to-Generator 的结构化中间表示，把 agent state 压缩为 intent、scope、confidence、risk、directives、negative constraints 和 rollback trigger。

第三项贡献：Data-efficient controller training

冻结 Generator，只训练 planner/controller；用 candidate state update / brief 对比、delta reward、Utility@k feedback 和 low-feedback split 证明少量反馈也能适配。

第四项贡献：Agentic closed-loop workflow

把 continue、ask、probe、rewrite、rollback 纳入 action policy，并用 Score Aggregator 将 Judge diagnosis 和 UserSim response 回写到下一轮 state。

第五项贡献：Entertainment workflow benchmark

构建面向娱乐场景的 ambiguous feedback personalization benchmark，并用小说生成实例化，覆盖显式、隐式、稀疏、噪声、延迟、矛盾、drift 和 held-out feedback regimes。

第六项贡献：统一强 baseline 对比

把 retrieval personalization、raw-history prompting、P-RLHF/reward reranking、per-user PEFT、WildFeedback-DPO、OPEN/GATE/PREFINE 等统一到 same closed-loop protocol。

Core claim

用户反馈不是监督标签、不是 memory append、也不是 deterministic state diff。

Core scenario: entertainment agent workflow instantiated with fiction generation. User feedback is sparse, noisy, ambiguous, delayed, and mixed explicit/implicit. The Controller must infer latent intent, scope, confidence, and risk before updating agent state and choosing the next action. In this v7 framing, state diff is explicitly not the main target; state diff and memory updates are diagnostic artifacts or baselines.

Problem formulation

Feedback-conditioned next-turn adaptation under uncertainty

At turn t, the agent has user u, story context s_t, task x_t, previous output y_t, observed feedback o_t, belief b_t, and next task x_{t+1}. The goal is y_{t+1} that incorporates o_t without risky overcommitment.

Observation model

o_t ~ p(o | z_u, x_t, y_t, s_t)

Feedback is generated by a latent preference/current-intent source, not by a clean label oracle.

Posterior-like belief

b_{t+1} = q_theta(z_u, scope, confidence, risk | b_t, x_t, y_t, o_t, x_{t+1})

The Controller outputs structured belief and action, not a story.

System architecture

Trained UserSim → Qwen3-4B-Controller → Frozen Generator

The main contribution is the Controller’s feedback-to-control adaptation. Generator stays frozen in the main experiment to isolate the Controller.

Module 1

Qwen3-4B-UserSim

Trained explicit + implicit feedback generator / judge. Full-parameter SFT first, optional DPO/GRPO. Frozen before Controller RL.

Module 2

Qwen3-4B-Controller

Full-parameter fine-tuned Controller that emits Personalization Brief with intent, scope, confidence, risk, directives, and negative constraints.

Module 3

Frozen Generator

Main setting: a frozen strong LLM or frozen Qwen3 variant receives story context, next task, and the brief.

Hard constraint

no Phase-4 UserSim improvement after Controller training

Hard constraint

no joint UserSim–Controller co-training

Training allowance

full-parameter fine-tuning is allowed for Qwen3-4B modules

Decision object

Personalization Brief

The Controller outputs a brief only. The generator writes the next passage from this brief; it must not mention feedback explicitly.

Concrete sample brief

{
  "feedback_interpretation": [
    {
      "intent": "用户希望减少直接心理描写，把紧张感交给动作、停顿和环境压迫",
      "scope": "审讯/对峙/暧昧张力场景",
      "confidence": 0.78,
      "evidence": [
        "comment: 压迫感太直给",
        "highlight: sentence_8",
        "dwell_time_bucket: high"
      ],
      "risk": "medium_overgeneralization"
    }
  ],
  "adaptation_action": "tentative_adapt",
  "generation_directives": [
    "减少直接情绪解释",
    "用动作、沉默、空间距离和物件细节制造压迫感",
    "反派台词保持克制，不直接宣布自己的恶意"
  ],
  "negative_constraints": [
    "不要把所有段落都改成极短句",
    "不要改变人物核心动机",
    "不要加入新的世界观设定"
  ],
  "ask_user": false,
  "rubric_hints": [
    "是否减少直白情绪解释",
    "是否在不喊口号的情况下提升压迫感",
    "是否避免过度泛化反馈"
  ]
}

Adaptation action set

answer_current — weak or irrelevant feedback.
tentative_adapt — signal exists but scope is uncertain.
scoped_commit — repeated consistent feedback supports a scoped preference.
ask_clarification — ambiguity and wrong-adaptation cost are high.
probe_variant — two plausible interpretations can be tested cheaply.
rollback_or_weaken — new feedback contradicts prior belief.
ignore — likely noise or unrelated signal.

UserSim design

Train a behavioral feedback simulator, not a prompt-only persona.

Prompt-only simulators leak profiles, over-explain feedback, and miss implicit/delayed/noisy behaviors. Qwen3-4B-UserSim emits feedback bundles and optional satisfaction scores.

UserSim sample output

{
  "user_profile": {
    "stable_preferences": [
      "克制叙事",
      "慢热张力",
      "道德灰度人物"
    ],
    "dislikes": [
      "狗血告白",
      "现代网络词",
      "大段设定解释"
    ],
    "conditional_preferences": [
      {
        "condition": "暧昧场景",
        "preference": "潜台词多于直接表白"
      }
    ],
    "implicit_behavior_pattern": {
      "high_dwell_when": [
        "悬念强",
        "人物冲突强"
      ],
      "drop_when": [
        "设定解释超过三段"
      ]
    },
    "feedback_style": {
      "verbosity": "short",
      "explicitness": "medium",
      "noise_level": "medium"
    }
  },
  "story_context": "dark fantasy betrayal arc, chapter 5",
  "task": "继续写审讯室对峙场景",
  "previous_generation": "反派把每一句威胁都说满，旁白直接解释主角恐惧。",
  "explicit_feedback": [
    {
      "type": "comment",
      "text": "这段压迫感有点太直给了，别让反派把话说满。",
      "target_span": "paragraph_2",
      "polarity": "negative",
      "strength": 0.72,
      "delay": 0
    }
  ],
  "implicit_feedback": [
    {
      "type": "highlight",
      "target_span": "sentence_8",
      "signal": "like",
      "strength": 0.63
    },
    {
      "type": "dwell_time_bucket",
      "value": "high"
    },
    {
      "type": "continue_reading",
      "value": true
    }
  ],
  "hidden_feedback_intent_for_training": [
    {
      "intent": "用户希望通过动作、停顿和空间压迫制造张力，而不是通过直白台词",
      "scope": "审讯/对峙场景",
      "should_generalize": true
    }
  ],
  "satisfaction_score": 3.4
}

UserSim reward

R_sim = alpha R_realism + beta R_profile + gamma R_cycle + delta R_calib + eta R_diversity - lambda R_leak - mu R_overexplicit

Quality checks: Profile Consistency, Feedback Realism, Implicit Calibration, Diversity, Non-leakage, Ambiguity Control, Cycle Recoverability.

Controller training

SFT first, then DPO/GRPO through frozen UserSim + Frozen Generator rollouts.

Oracle briefs can be produced by a stronger LLM using hidden profile, rule templates for easy cases, and corrections on a calibration subset.

SFT target

L_Controller^SFT = -log p_theta(c_t | b_t, x_t, y_t, o_t, x_{t+1})

Target c_t is the Personalization Brief, not story text.

Controller reward

R_ctrl = alpha R_incorporation + beta R_quality + gamma R_scope - delta R_overcommit - eta R_misattrib - mu R_burden

Rollouts compare resulting y_{t+1} by frozen UserSim and feedback-conditioned rubric/Judge; calibration samples only check scorer quality.

Phase 0

construct seed profiles tasks outputs feedback rubrics briefs

Controller/UserSim pipeline artifact

Phase 1

train UserSim full parameter SFT

Qwen3-4B-UserSim

Phase 2

optional align UserSim with DPO or GRPO

Controller/UserSim pipeline artifact

Phase 3

freeze UserSim

no Phase-4 UserSim improvement after Controller training

Phase 4

train Controller full parameter SFT

Qwen3-4B-Controller

Phase 5

train Controller DPO or GRPO with frozen UserSim and frozen Generator rollouts

Controller/UserSim pipeline artifact

Phase 6

evaluate with frozen UserSim, rubric/Judge, fixed seeds

Controller/UserSim pipeline artifact

Dataset and benchmark

AmbiPref-Story / Feedback2Fiction

Benchmark episodes cover explicit, implicit, sparse, noisy, delayed, contradictory, and drift feedback in fiction generation.

Schema

{
  "schema_version": "v7.0",
  "benchmark": "AmbiPref-Story / Feedback2Fiction",
  "unit": "one next-turn personalization episode",
  "required_fields": [
    "user_profile",
    "story_context",
    "previous_task",
    "previous_generation",
    "observed_feedback_bundle",
    "hidden_feedback_intent_for_training",
    "history_belief",
    "next_task",
    "personalization_brief",
    "candidate_next_output",
    "rubric_scores"
  ],
  "user_profile": {
    "stable_preferences": "list[str]",
    "dislikes": "list[str]",
    "conditional_preferences": "list[{condition, preference}]",
    "implicit_behavior_pattern": "object",
    "feedback_style": "{verbosity, explicitness, noise_level}"
  },
  "observed_feedback_bundle": {
    "explicit_feedback": [
      "comment",
      "revision_request",
      "user_rewrite",
      "label",
      "delayed_comment"
    ],
    "implicit_feedback": [
      "highlight",
      "dwell_time_bucket",
      "tip_bucket",
      "skip",
      "continue_reading",
      "favorite"
    ]
  },
  "brief": {
    "feedback_interpretation": [
      "intent",
      "scope",
      "confidence",
      "evidence",
      "risk"
    ],
    "adaptation_action_enum": [
      "answer_current",
      "tentative_adapt",
      "scoped_commit",
      "ask_clarification",
      "probe_variant",
      "rollback_or_weaken",
      "ignore"
    ],
    "generation_directives": "list[str]",
    "includes_negative_constraints": true,
    "ask_user": "bool",
    "rubric_hints": "list[str]"
  },
  "difficulty_regimes": [
    "easy",
    "medium",
    "hard",
    "delayed",
    "contradictory",
    "drift",
    "noisy"
  ],
  "license_policy": "publish compact synthetic/public-safe samples only; keep DTA-bound reader responses out of public artifacts"
}

Difficulty regimes

Regime	Description
Easy	explicit, immediate, low noise
Medium	explicit but ambiguous
Hard	implicit-only or mixed feedback
Delayed	feedback refers to earlier output
Contradictory	feedback conflicts with prior belief
Drift	project-level preference changes
Noisy	feedback partially unrelated to true preference

Auto-rubric evaluation

FIS, Scope Control, Overcommit Rate, Misattribution Rate, Quality Preservation

Evaluation is feedback-conditioned: does the next output incorporate previous feedback with correct scope while preserving story quality?

Metric

FIS

Feedback Incorporation Score; whether next output incorporates previous feedback.

Metric

Scope Control

Whether adaptation is applied only to the correct scene/style/preference scope.

Metric

Overcommit Rate

Frequency that local feedback becomes an incorrect global rule.

Metric

Misattribution Rate

Frequency that feedback is interpreted along the wrong dimension.

Metric

Implicit Feedback Utilization

Implicit-only dwell/tip/skip/highlight signals produce correct adaptation.

Metric

Ask Rate

Clarification frequency under uncertainty.

Metric

User Burden

Questions plus extra interaction length.

Metric

Quality Preservation

Story coherence, character consistency, and prose quality after adaptation.

Metric

Utility@k Feedback

Personalization utility using only k feedback events.

Metric

Win Rate

Pairwise win rate against baseline outputs.

Formula boxes

FIS = mean_i rubric_score_i(Feedback Incorporation | y_t, o_t, x_{t+1}, y_{t+1})

Scope Control = mean_i rubric_score_i(correct_scope) - penalty(overscoped_changes)

Overcommit Rate = count(local_feedback_globalized_wrongly) / count(feedback_events)

Utility@k Feedback = E[J(y_{t+1}; user) | first k feedback events]

Two rubric modes

Oracle-aided rubric: rubric generator can access hidden user intent and measures true preference alignment.

Feedback-only rubric: rubric generator sees only observed feedback and measures deployable evaluation.

Ours-SFT+RL FIS0.84

Naive prompt FIS0.61

Overcommit ↓0.18

Baselines

Concrete recent comparisons, adapted to the same next-turn protocol.

All baselines receive previous output, observed feedback, next task, and the same generator. They differ in prompt/profile/adapter/reward/control representation.

Method	Trainable module	Structured control
No personalization	none	False
Naive feedback prompt	none	False
LaMP retrieval	retriever/profile prompt	False
LongLaMP-style long-form profile conditioning	retriever/profile prompt	False
P-RLHF	personalized user model + reward/generator	False
OPPU	one PEFT per user	False
Per-Pcs	shared personalized PEFT pieces	False
PPlug	user embedder	False
FSPO	few-shot personalized reward model	False
CHAMELEON	self-generated preference data + representation editing	False
WildFeedback-DPO	feedback-to-pair DPO	False
OPEN	BOED elicitation policy	partial
GATE	elicitation dialogue policy	partial
PREFINE	pseudo-user critic + user-specific rubric	rubric
Ours-SFT	Qwen3-4B-Controller	True
Ours-SFT+RL	Qwen3-4B-Controller + DPO/GRPO	True

Ablations

What must be removed to prove the brief structure matters.

Ablation	Expected finding
no implicit feedback	worse on dwell/tip/skip regimes
no scope field	more overcommit
no negative constraints	more story drift and overcorrection
no ask/probe action	worse under ambiguity
no confidence/risk	worse calibration
SFT only	weaker than SFT+RL on hard feedback
deterministic summary instead of belief/brief	worse under noisy feedback
prompt-only UserSim	less realistic and too explicit
SFT-only UserSim	weaker implicit calibration
full-tuned Qwen3-4B generator	appendix only; tests brief following

Full-parameter Qwen3-4B plan

Separate checkpoints: Qwen3-4B-UserSim, Qwen3-4B-Controller, optional generator.

Use bf16, sequence packing, gradient checkpointing, FSDP or DeepSpeed ZeRO-3, and mixed-length batches. Keep the Generator frozen in the main experiments.

Qwen3-4B-UserSim

Full-parameter SFT on feedback bundles, optional DPO/GRPO for realism/calibration/non-leakage.

Qwen3-4B-Controller

Full-parameter SFT on oracle briefs, then DPO/GRPO using frozen UserSim and Frozen Generator rollouts.

Optional generator

Appendix only: full-parameter tuned Qwen3-4B generator for brief following; not the main contribution.

Controller I/O sample

End-to-end next-turn adaptation record

{
  "input": {
    "history_belief": "prefers restrained prose; dislikes direct exposition; scope uncertain for interrogation scenes",
    "previous_task": "写审讯室对峙",
    "previous_generation": "反派把每一句威胁都说满，旁白直接解释主角恐惧。",
    "feedback_bundle": {
      "explicit_feedback": [
        {
          "type": "comment",
          "text": "这段压迫感有点太直给了，别让反派把话说满。",
          "target_span": "paragraph_2",
          "polarity": "negative",
          "strength": 0.72,
          "delay": 0
        }
      ],
      "implicit_feedback": [
        {
          "type": "highlight",
          "target_span": "sentence_8",
          "signal": "like",
          "strength": 0.63
        },
        {
          "type": "dwell_time_bucket",
          "value": "high"
        },
        {
          "type": "continue_reading",
          "value": true
        }
      ]
    },
    "next_task": "继续写反派和主角在审讯室对峙",
    "story_context": "dark fantasy betrayal arc, chapter 5"
  },
  "output": {
    "personalization_brief": {
      "feedback_interpretation": [
        {
          "intent": "用户希望减少直接心理描写，把紧张感交给动作、停顿和环境压迫",
          "scope": "审讯/对峙/暧昧张力场景",
          "confidence": 0.78,
          "evidence": [
            "comment: 压迫感太直给",
            "highlight: sentence_8",
            "dwell_time_bucket: high"
          ],
          "risk": "medium_overgeneralization"
        }
      ],
      "adaptation_action": "tentative_adapt",
      "generation_directives": [
        "减少直接情绪解释",
        "用动作、沉默、空间距离和物件细节制造压迫感",
        "反派台词保持克制，不直接宣布自己的恶意"
      ],
      "negative_constraints": [
        "不要把所有段落都改成极短句",
        "不要改变人物核心动机",
        "不要加入新的世界观设定"
      ],
      "ask_user": false,
      "rubric_hints": [
        "是否减少直白情绪解释",
        "是否在不喊口号的情况下提升压迫感",
        "是否避免过度泛化反馈"
      ]
    }
  }
}

Paper contributions

Five contributions to claim if experiments succeed.

1 · Problem formulation

Feedback as noisy observation of latent intent rather than deterministic memory/state update.

2 · Trained UserSim

Qwen3-4B explicit + implicit feedback bundles, including comments, highlights, dwell, tips, skips, delayed feedback.

3 · Controller

Qwen3-4B full-finetuned feedback-to-brief model.

4 · Auto-rubric evaluation

Feedback-conditioned rubrics measuring incorporation, scope, overcommit risk, and quality.

5 · Benchmark

Ambiguous feedback personalization benchmark for fiction generation.

Positioning

Avoid generic memory mechanism, GUI/OS/Web agent benchmark, or simple retrieval framing.

Artifacts

Public v7 artifacts

All links use absolute paths for Next catch-all safety.

agent-demo.html workflow-dataflow.html visualization.html v7_manifest.json v7_research_plan.md v7_dataset_schema.json v7_training_pipeline.json v7_eval_metrics.json v7_baseline_matrix.json sample_usersim_feedback.jsonl sample_controller_io.jsonl sample_brief.json v7_reward_formulas.json

Summary

v7 turns ambiguous feedback into scoped, risk-aware next-turn control.

The method should outperform retrieval/profile/reward-only baselines when comments, highlights, dwell time, skips, tips, and rewrites are ambiguous. The main promise is not simply better fiction; it is safer, data-efficient adaptation under uncertainty.

Appendix

Detailed planning notes

Appendix 1

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 1

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 1

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 1

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 1

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 1

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 2

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 2

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 2

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 2

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 2

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 2

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 3

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 3

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 3

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 3

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 3

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 3

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 4

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 4

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 4

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 4

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 4

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 4

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 5

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 5

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 5

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 5

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 5

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 5

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 6

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 6

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 6

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 6

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 6

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 6

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 7

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 7

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 7

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 7

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 7

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 7

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 8

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 8

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 8

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 8

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 8

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 8

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 9

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 9

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 9

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 9

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 9

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 9

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 10

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 10

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 10

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 10

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 10

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 10

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 11

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 11

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 11

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 11

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 11

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 11

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 12

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 12

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 12

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 12

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 12

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 12

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 13

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 13

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 13

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 13

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 13

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 13

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

Appendix 14

Feedback event types

Comment, revision request, user rewrite, highlight, label, like, favorite, tip, dwell time, skip/drop, continue reading, delayed comment.

Appendix 14

Story task types

Scene generation, continuation, rewrite, character voice, dialogue, pacing control, plot transition, style consistency.

Appendix 14

Risk taxonomy

medium_overgeneralization, wrong_scope, misattribution, story_quality_drift, user_burden, profile_leakage.

Appendix 14

Evaluation split

Held-out users, held-out feedback regimes, fixed UserSim, fixed rubric/Judge, oracle-aided calibration, feedback-only main scorer.

Appendix 14

Why Frozen Generator

It isolates Controller quality and prevents improvements from being attributed to generator fine-tuning.

Appendix 14

Why negative constraints

They explicitly prevent overcorrection: do not rewrite character motivation, do not globally shorten every sentence, do not introduce new lore.

中文细化补充

这版 v7 的写作目标是让读者先理解 entertainment agent workflow，再理解小说实例。

研究动机部分强调：真实娱乐场景的用户反馈是连续行为信号，agent 要尽可能捕捉显式、隐式、延迟和含糊反馈，并把它们转成 agent state update。Bench 构造部分说明 closed-loop episode：y_t -> feedback_t -> state/brief/action -> y_{t+1} -> UserSim + rubric/Judge -> score。方法部分说明为什么分离 UserSim、Controller、Action Planner 和 Frozen Generator。

评估部分严格对齐 workflow 图：主评测只使用 frozen UserSim、feedback-conditioned auto-rubric/Judge 和 Score Aggregator。UserSim 负责下一轮行为和 satisfaction，rubric/Judge 负责 delta evidence，Aggregator 负责 Return。人工样本只用于校准 simulator/rubric 是否合理，不进入 bench 主分数。