# Corrected Direction: Personalized User-Feedback Agent Evolution

> 日期：2026-05-08 UTC
> 项目：`novel-demo`
> 纠偏后核心：**捕捉显式/隐式用户反馈 → 推断用户偏好状态 → 更新 agent memory / critic / planner / generation policy → 改善后续生成内容**。
> 明确排除：暂不主打 CUA / GUI / 多模态；LoCoMo、Reflexion、ALFWorld 只作为历史背景，不作为核心 baseline 或 benchmark。

---

## 0. 一句话结论

你现在的 topic 更像 **personalized interactive generation / recommender-style user feedback learning**，不是传统 agent benchmark。

最推荐的论文定位是：

> **Learning User Preference State from Explicit and Implicit Feedback for Self-Evolving Generative Agents**

具体做成：

1. **Benchmark**：`PIF-Bench` — Personalized Interactive Feedback Benchmark for Generative Agents。
2. **Method**：`PUMA` — Preference Update for Memory-and-Agent Framework Adaptation。

核心闭环：

```text
content generated at turn t
  -> user explicit feedback: like / favorite / comment / rating / rewrite request
  -> user implicit feedback: dwell time / fast swipe / reread / skip / continue / stop
  -> infer user preference state z_u,t
  -> update memory / critic / planner / generation policy
  -> generate next content y_{t+1}
  -> evaluate future engagement + preference fit + feedback incorporation
```

---

## 1. 纠偏：之前哪些东西不应该做核心

| 之前我提到的方向 | 当前处理方式 | 原因 |
|---|---|---|
| Reflexion / ALFWorld | 只当历史背景，不作为核心 | 太老，且不是个性化用户反馈；feedback 是任务失败，不是用户偏好/行为信号 |
| LoCoMo | 不作为核心 | 它是长期对话记忆 QA，不是停留/点赞/收藏/划走这种 user feedback |
| CUA / GUI / multimodal agent | 暂时不碰 | 你的重点不是操作电脑，而是从用户反馈中提取偏好并更新后续生成 |
| Long-horizon credit assignment | 可作为后续 extension | 但第一篇应聚焦 user feedback -> preference state -> memory/framework update |
| 普通 personalization / static profile | 作为 baseline | 你的 novelty 是动态 feedback-driven personalization，不是静态 profile |

---

## 2. 相关 ICLR 2026 高相关 paper

我用 OpenReview 查了 ICLR 2026 accepted papers，并下载了相关 PDF 到：

```text
/root/code/vepfs/hermes/novel-demo/personalized-user-feedback-agent-evolution/papers/iclr2026
```

这些比 Reflexion/LoCoMo 更接近你的方向。

### 2.1 最相关：human feedback / preference interpretation / personalization

| Paper | Venue | 为什么重要 |
|---|---|---|
| **What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data** | ICLR 2026 Oral | 非常关键：它研究 human feedback 里面到底编码了什么偏好。你的隐式/显式 feedback 也需要从低级行为信号中解释出偏好特征。 |
| **P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling** | ICLR 2026 Oral | 直接相关：personalized reward model，根据用户历史和偏好做 user-specific scoring；可作为 reward/reranker baseline。 |
| **Swap-guided Preference Learning for Personalized RLHF** | ICLR 2026 Poster | 个性化 RLHF，建 user-specific latent preference，适合作为“用户潜变量偏好建模”baseline。 |
| **ActiveDPO** | ICLR 2026 Poster | 主动选择最有价值的 preference feedback；可以用于决定什么时候向用户要显式反馈。 |
| **Beyond Binary Preferences: Reward Modeling with Ordinal Feedback** | ICLR 2026 Poster | 你的反馈不只是 like/dislike，还有停留时长、收藏、快速划走等强弱不同的 ordinal / continuous signal。 |
| **Causally Robust Reward Learning from Reason-Augmented Preference Feedback** | ICLR 2026 Poster | 用 rationale 减少 preference learning 的 spurious correlation；对“停留长不一定等于喜欢”很重要。 |
| **What’s In My Human Feedback?** meta review 还明确说：它能用于 data curation 和 personalization。这个方向和你的“从反馈中抽偏好并更新 agent state”非常贴。 |

### 2.2 最相关：user simulator / proactive personalized assistant

| Paper | Venue | 为什么重要 |
|---|---|---|
| **ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation** | ICLR 2026 Poster | 直接回答 user simulator 是否值得做：他们用 simulated user + assistant 环境评估 personalization/proactivity。你的 benchmark 可以借鉴，但要换成内容消费反馈。 |
| **PrefDisco: Benchmarking Proactive Personalized Reasoning** | ICLR 2026 Poster | 把静态任务变成 interactive personalization，要求模型主动发现缺失用户偏好。你可以借鉴 preference discovery 和问询机制。 |
| **Flipping the Dialogue: Training and Evaluating User Language Models** | ICLR 2026 Poster | 用户模拟本身是一个研究方向。可用于构建 benchmark 中的 user simulator component。 |
| **Supporting High-Stakes Decision Making Through Interactive Preference Elicitation in the Latent Space** | ICLR 2026 Poster | 用 pairwise comparisons 做 interactive preference elicitation；适合作为显式反馈收集方式参考。 |

### 2.3 推荐系统 / 隐式行为反馈相关

| Paper | Venue | 为什么重要 |
|---|---|---|
| **Beyond Match Maximization and Fairness: Retention-Optimized Two-Sided Matching** | ICLR 2026 Poster | 把目标从 immediate match 转向 retention；你的内容生成也不应只优化当前点赞，而要优化继续阅读/长期满意。 |
| **Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenders** | ICLR 2026 Poster | 处理超长 user interaction history 的工业推荐模型；说明长用户历史建模是核心问题。 |
| **PerFit: Exploring Personalization Shifts in Representation Space of LLMs** | ICLR 2026 Poster | LaMP-style personalization baseline，但 reviewer 也指出它缺少 continuously evolving user scenarios；这正好是你的机会。 |

### 2.4 自动评价 / 用户反馈指标化

| Paper | Venue | 为什么重要 |
|---|---|---|
| **AutoLibra: Agent Metric Induction from Open-Ended Human Feedback** | ICLR 2026 Poster | 从开放式 human feedback 中诱导 agent evaluation metrics；你的 feedback-to-preference parser 可以借鉴。 |
| **AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators** | ICLR 2026 Poster | 关注 thumbs up/down、behavioral signals、retention 这类 user-centered feedback，和你的显式/隐式反馈非常贴近。 |
| **RewardBench 2** | ICLR 2026 Poster | reward model evaluation 的近期 benchmark，可作为 reward/reranker 部分参考。 |

---

## 3. 你的 benchmark 应该怎么做：PIF-Bench

### 3.1 Benchmark 名字

**PIF-Bench: Personalized Interactive Feedback Benchmark for Generative Agents**

### 3.2 核心任务

不是让 agent “写一章小说”，而是：

> 给定一段内容生成历史、用户显式/隐式反馈日志，agent 需要推断用户当前偏好状态，并更新 memory/framework，使下一轮生成更符合用户偏好并提升长期 engagement。

形式化：

```text
At turn t:
Input:
  - generated content y_t
  - content metadata/features c_t
  - explicit feedback e_t: like, dislike, favorite, comment, rating, rewrite request
  - implicit feedback b_t: dwell time, fast swipe, reread, skip, continue, stop
  - previous user state z_{u,t-1}
  - previous agent memory/framework S_{t-1}

Output:
  - updated user preference state z_{u,t}
  - memory/framework update ΔS_t
  - next generation plan/content y_{t+1}

Evaluation:
  - future feedback prediction
  - next content preference fit
  - long-term engagement/retention
  - update correctness / overgeneralization / stability
```

### 3.3 数据应该包含什么

每个样本不是单 turn，而是一条 user-content interaction trajectory：

```json
{
  "user_id": "u_013",
  "hidden_preference_profile": {
    "stable": {
      "genre": "dark romance + political intrigue",
      "pacing": "fast plot progress with emotional beats",
      "character": "strong female agency",
      "taboo": ["misunderstanding trope", "passive rescue resolution"]
    },
    "contextual": {
      "romance_scene": "subtext > direct confession",
      "battle_scene": "decisive action > exposition"
    },
    "drift": {
      "after_5_chapters": "wants more plot progress"
    }
  },
  "trajectory": [
    {
      "turn": 1,
      "generated_content": "...",
      "content_features": {
        "pacing": 0.32,
        "romance_tension": 0.76,
        "agency": 0.41,
        "lore_density": 0.58
      },
      "explicit_feedback": {
        "like": false,
        "favorite": false,
        "comment": "有点慢",
        "rating": 2
      },
      "implicit_feedback": {
        "dwell_time_sec": 8.2,
        "normalized_dwell": -0.7,
        "fast_swipe": true,
        "reread": false,
        "continue_next": false
      },
      "gold_preference_update": [
        {
          "dimension": "pacing",
          "direction": "increase_speed",
          "scope": "current_story",
          "confidence": 0.74,
          "evidence": ["comment: 有点慢", "fast_swipe"]
        }
      ],
      "gold_agent_update": [
        {
          "target": "critic.checklist",
          "content": "Before generating next scene, check whether exposition delays plot progress.",
          "scope": "current_story"
        },
        {
          "target": "planner.policy",
          "content": "Prefer next scene plans that introduce irreversible plot movement within 500 words.",
          "scope": "current_arc"
        }
      ]
    }
  ]
}
```

### 3.4 显式/隐式反馈如何建模

#### 显式反馈

| Signal | 含义 | 置信度 |
|---|---|---:|
| like/dislike | 二元偏好 | 中 |
| favorite/bookmark | 强正反馈 | 高 |
| rating | ordinal feedback | 中高 |
| comment | 最有信息量，但需解析 | 高 |
| rewrite request | 明确负反馈 + 修改方向 | 高 |
| A/B choice | 最适合训练 preference model | 高 |

#### 隐式反馈

| Signal | 可能含义 | 风险 |
|---|---|---|
| dwell time 长 | 喜欢/沉浸/难懂/走神 | ambiguous |
| fast swipe | 不喜欢/看过/不感兴趣 | noisy |
| reread | 喜欢/困惑/信息密度高 | ambiguous |
| continue next | 短期 engagement | 不等于长期满意 |
| favorite after dwell | 强正反馈 | 高 |
| stop reading | 负反馈/外部干扰 | noisy |

所以 benchmark 要评估一个关键能力：

> **agent 能不能区分隐式反馈的多义性，并结合上下文和显式反馈推断偏好，而不是把 dwell time 简单等同于喜欢。**

---

## 4. User simulator 是否值得放进 benchmark？

### 4.1 结论

**值得，但不能作为唯一评估。**

最好的设计是：

```text
PIF-Bench = logged/pilot human feedback subset + controlled user simulator subset + human audit subset
```

原因：

1. 真实用户反馈难收集，尤其是隐式反馈和长期轨迹。
2. simulator 可以提供 hidden preference ground truth，方便评估 preference-state inference 是否正确。
3. simulator 可以系统生成 edge cases：偏好漂移、噪声反馈、显式/隐式冲突、短期喜欢但长期厌倦。
4. 但 simulator 可能不真实，所以必须用少量真实用户日志校准，并做人类 audit。

### 4.2 User simulator 应该怎么做

不要做一个只会聊天的 LLM user simulator。你需要的是 **behavioral feedback simulator**。

它应该由 5 个模块组成：

```text
1. Hidden Preference Profile
2. Content Perception Model
3. Feedback Emission Model
4. Preference Drift / Fatigue Model
5. Calibration / Noise Model
```

#### Module 1: Hidden Preference Profile

每个用户有稳定偏好、上下文偏好、禁忌和控制偏好：

```json
{
  "stable_preferences": {
    "pacing": "fast_with_emotional_beats",
    "agency": "strong_female_lead",
    "romance": "slow_burn_high_tension",
    "lore_density": "medium"
  },
  "contextual_preferences": {
    "romance_scene": "subtext_not_direct_confession",
    "battle_scene": "decisive_action",
    "political_scene": "high_strategy_density"
  },
  "taboos": ["misunderstanding trope", "passive rescue resolution"],
  "control_preference": "wants choices before major plot branch"
}
```

#### Module 2: Content Perception Model

把生成内容转成 feature vector：

```json
{
  "pacing_speed": 0.31,
  "plot_progress": 0.42,
  "romance_tension": 0.78,
  "female_agency": 0.35,
  "lore_density": 0.66,
  "trope_misunderstanding": 0.81,
  "novelty": 0.57,
  "coherence": 0.72
}
```

这个可以由：

- LLM judge；
- small classifier；
- rule-based feature extractor；
- human labels；

组合得到。

#### Module 3: Feedback Emission Model

用户是否点赞/收藏/停留/划走，由 utility 决定：

```text
utility_t = preference_match(content_t, user_profile)
          + novelty_bonus
          + coherence_bonus
          - taboo_penalty
          - repetition_fatigue
          - question_burden
          + noise
```

然后生成不同反馈：

```text
P(like)       = sigmoid(a1 * utility_t)
P(favorite)   = sigmoid(a2 * utility_t + a3 * emotional_peak)
P(fast_swipe) = sigmoid(-b1 * utility_t + b2 * early_boredom)
P(continue)   = sigmoid(c1 * utility_t + c2 * unresolved_hook)
dwell_time    = lognormal(mu = d1 * utility_t + d2 * complexity)
comment       = generated from top mismatch dimensions
```

关键：dwell time 不能简单等于喜欢。

例如：

```text
long dwell = high engagement OR high confusion
reread = emotional favorite OR confusing exposition
fast swipe = dislike OR user already saw this trope
```

#### Module 4: Drift / Fatigue Model

用户偏好会变：

```text
after repeated romance tension with no plot progress -> wants faster plot
after too many action scenes -> wants emotional resolution
after feedback ignored twice -> trust decreases
```

这正好对应你的 “agent 要不断更新后续生成内容”。

#### Module 5: Calibration

用真实 pilot 用户数据校准：

```text
- dwell time distribution
- like/favorite probability
- fast swipe threshold
- comment frequency
- feedback ignored -> churn probability
```

哪怕只有 20–50 个真实用户，也能让 simulator 更可信。

### 4.3 Simulator 在 benchmark 中的位置

建议有三个 split：

| Split | 来源 | 作用 |
|---|---|---|
| `sim-controlled` | simulator | 有 hidden preference gold，可评估偏好推断准确率 |
| `sim-stress` | simulator edge cases | 测噪声、漂移、显式/隐式冲突、overgeneralization |
| `human-pilot` | 真实用户日志 | 检查 simulator 是否有效，最终 human preference / engagement 验证 |

论文里要明确：

> The simulator is not the final proof of human satisfaction; it is a controlled instrument for evaluating preference-state inference and update mechanisms under known hidden preferences.

---

## 5. Method 应该怎么改：PUMA

### 5.1 Method 名字

**PUMA: Preference Update for Memory-and-Agent Framework Adaptation**

### 5.2 核心思想

把用户反馈转成两层更新：

```text
1. User preference state update
2. Agent framework update
```

不是所有反馈都直接写进 memory。

例如：

```text
feedback: 快速划走 + 评论“太拖了”

Preference update:
  用户当前不喜欢 exposition-heavy pacing

Memory update:
  user prefers faster plot movement in current story

Critic update:
  add pacing-progress check

Planner update:
  next scene must include irreversible event

Generation policy update:
  sample candidates with higher plot_progress score
```

### 5.3 Architecture

```text
Feedback Event Encoder
  - explicit encoder: like, favorite, rating, comment, rewrite request
  - implicit encoder: dwell, fast swipe, reread, continue, stop
  - context encoder: content features, story state, previous feedback
        ↓
Preference Belief Updater
  - Bayesian / recurrent / transformer state update
  - outputs preference dimensions + uncertainty + scope
        ↓
Memory & Framework Update Router
  - decides whether to update memory, critic, planner, retriever, generation policy
        ↓
Verifier / Anti-overgeneralization Checker
  - prevents learning wrong global rule from local/noisy signal
        ↓
Personalized Generator / Reranker
  - generates multiple next-content candidates
  - reranks by predicted engagement + preference fit + diversity + story consistency
```

### 5.4 Baselines

| Baseline | 意义 |
|---|---|
| No personalization | 不使用用户反馈 |
| Static profile prompting | 只用初始用户画像 |
| Full history prompting | 把所有交互塞上下文 |
| Raw feedback RAG | 检索历史反馈原文 |
| LaMP-style personalization | 静态用户 profile / history baseline |
| Sequential recommender baseline | SASRec/BERT4Rec/HSTU-style 用户行为序列建模 |
| Reward/reranker baseline | P-GenRM / personalized reward model |
| DPO/ActiveDPO baseline | 用显式 preference pair 更新模型/reranker |
| Ours memory-only | 只更新 memory |
| Ours full PUMA | preference belief + memory/framework update + verifier |

### 5.5 Metrics

| Metric | 说明 |
|---|---|
| Feedback prediction AUC/NDCG | 能否预测 like/favorite/continue/skip |
| Dwell calibration | 预测停留时长分布是否准 |
| Preference state accuracy | 在 simulator split 上和 hidden preference 比较 |
| Next-content win rate | 后续生成是否更符合用户偏好 |
| Long-term engagement | 多轮 continue / favorite / retention |
| Feedback incorporation | 用户负反馈后，后续同类错误是否下降 |
| Overgeneralization rate | 是否把局部反馈错误变成全局规则 |
| Drift adaptation speed | 用户偏好变化后几轮能适应 |
| Memory/framework update accuracy | 更新到了 memory/critic/planner/generator 哪一层是否正确 |

---

## 6. 实验主线怎么设计

### Experiment 1: Can models infer preferences from mixed explicit/implicit feedback?

Input:

```text
content + feedback signals
```

Output:

```text
preference state dimensions
```

Compare:

- explicit only；
- implicit only；
- explicit + implicit；
- no context；
- with content features；
- with previous memory。

### Experiment 2: Does preference-state update improve future generation?

Compare:

```text
static profile vs full history vs raw feedback RAG vs PUMA
```

Evaluate:

- next content win rate；
- future like/favorite/continue；
- long-term retention。

### Experiment 3: Memory-only vs framework update

Ablation:

```text
PUMA-memory-only
PUMA-memory+critic
PUMA-memory+planner
PUMA-memory+critic+planner+reranker
```

This directly answers your core question:

> 捕捉到用户偏好后，到底只更新 memory 够不够？还是需要更新 agent framework？

### Experiment 4: Simulator validity

Compare simulator feedback with human pilot feedback:

- dwell distribution；
- like/favorite correlation；
- feedback text categories；
- preference inference agreement；
- generated content human A/B。

---

## 7. 最推荐的第一篇 paper framing

### Title

**PIF-Bench: Learning User Preference State from Explicit and Implicit Feedback for Self-Evolving Generative Agents**

或者 method-focused：

**PUMA: Preference-State Memory and Framework Updates from Implicit User Feedback**

### Core contributions

1. Define the problem of **feedback-driven agent evolution for personalized generation**.
2. Build `PIF-Bench`, with explicit feedback, implicit behavior signals, hidden preferences, drift, and future generation probes.
3. Propose `PUMA`, which updates both user memory and agent framework from inferred preference state.
4. Show that implicit feedback is useful but ambiguous; combining it with content/context and explicit feedback improves personalization.
5. Show memory-only update is insufficient; framework update improves repeated-error reduction and long-term engagement.

---

## 8. 现在应该怎么做下一步

第一步不要再下载 CUA/LoCoMo/ALFWorld 了。接下来应该做：

1. 定义 20–30 个内容偏好维度：pacing、agency、romance tension、lore density、humor、darkness、plot progress、trope tolerance 等。
2. 写 user simulator v0：hidden profile -> feedback signal emission。
3. 构造 100 个 simulated users × 20 turns。
4. 做 4 个 baseline：static profile、full history、raw feedback RAG、PUMA-lite。
5. 指标先做：feedback prediction、preference-state accuracy、next-content win rate、overgeneralization。
6. 再加 20 个真实 pilot users 校准 simulator。

---

## 9. 对 user simulator 的最终判断

**值得放进 benchmark，但要定位清楚：**

- 它不是为了证明“用户真的会这样”；
- 它是为了提供 hidden preference ground truth，系统评估 agent 是否能从 noisy explicit/implicit feedback 中恢复偏好状态；
- 最终仍需要 human-pilot split 验证 simulator 和真实行为的相关性。

所以 benchmark 结构应该是：

```text
PIF-Bench
  ├── simulated-controlled split  # 有 hidden preference gold
  ├── simulated-stress split      # 噪声/漂移/冲突/稀疏反馈
  └── human-pilot split           # 小规模真实用户验证
```

这会比纯 simulator 或纯静态 personalization benchmark 都更 solid。