FJFJ Work Studio
Home/Auto Research/Novel Feedback/v5

Qwen3-4B-Instruct-2507 · LoRA · Simulator Eval Protocol

Novel Feedback v5 Qwen3-4B LLM Simulator

v5 回答：要不要真的用数据训 Qwen3-4B？结论是要训，但第一刀训 reader/user simulator，而不是直接 PPO 小说生成器。当前完成 CPU/proxy 实验、public artifacts、Volc GPU smoke harness；本地 GPU 不使用。

是否真训Data schemaEvalEncoder-State vs QwenArtifacts
dataset
960

v5 proxy examples · Qidian-WebNovel DTA shaped · public-safe compact artifacts.

json_validity
1.000

Strict JSON outputs for state_after / state_delta / feedback.

same_text_user_separation
0.781

同一文本、不同用户反馈可分性；Encoder baseline 0.532。

DynamicProcessScore
1.476

Cumulative Interest / Preference Drift Alignment / Reading Fluency Penalty / PEVR.

Core decision

真的训 Qwen3-4B，但训练对象是 LLM reader simulator。

直接训 generator/PPO 会混入写文能力；v5 先证明 simulator fidelity、自然语言评论和个性化区分度。

Why Qwen/Qwen3-4B-Instruct-2507

Hugging Face model card 显示该模型 Apache-2.0、4.0B 参数、原生 262,144 context，且是 non-thinking mode，不输出 <think> blocks；适合 strict JSON simulator SFT。

Qwen3-4B-Instruct-2507LoRAVolcEngine GPUno local GPU

Honest status

Volc smoke task submitted: t-20260512225458-qs9jn (Status: Success). 当前落地 deterministic proxy + Volc-ready harness，不把它伪装成 full LoRA main train。

Main path: uses_reranking=false；personalized reranking 只是 baseline/diagnostic。

v1–v4 recap

v5 在 v4 可训练 simulator 上加 LLM feedback realism。

v1

PUMA-lite / CREDIT-TRACE

显式+隐式反馈转成 gold_agent_state_update；长程 feedback 找 culprit step/node。

v2

ESUS / FUSE-Hard

NCFU、structured diff、rollback、drift、anti-probe harm。

v3

Data + suite

真实小说数据 policy；96 experiments；state-conditioned direct generation。

v4

Encoder-State

transformer_prototype, state_delta_cosine, behavior fidelity, counterfactual consistency。

Data

v5 data schema：chat/instruction + strict assistant JSON。

包含 state_before/state_after/state_delta、explicit_feedback、implicit_feedback、natural_language_comment、gold_agent_state_update、future_probes。

Dataset tiers

  • v5-smoke-1k: 复用/扩展 v4 examples。
  • v5-main-10k: user swaps, interventions, drift, noisy feedback, rollback/retraction。
  • v5-real-calibrated: v3 public-derived / metadata-safe artifacts；Qidian-WebNovel DTA 边界保留。

Dataset manifest

{
  "dataset": "Qidian-WebNovel DTA",
  "examples": 960,
  "fields": [
    "candidate_text",
    "natural_language_comment",
    "scope",
    "anti_scope",
    "cumulative_interest",
    "preference_drift_alignment",
    "reading_fluency_penalty",
    "positive_expectation_violation_rate"
  ],
  "n_users": 80,
  "notes": [
    "CPU/proxy only",
    "No GPU, no model download",
    "Qwen3-4B-Instruct-2507 LoRA simulator target",
    "Claude Blog style rationale text"
  ],
  "seed": 77,
  "source": "deterministic synthetic proxy for Qidian-WebNovel DTA",
  "splits": {
    "test": 135,
    "train": 691,
    "validation": 134
  },
  "turns": 12
}

Concrete sample record

{
  "adapter": "LoRA",
  "base_model": "Qwen/Qwen3-4B-Instruct-2507",
  "candidate_text": "A romance intrigue chapter uses hidden bloodline reveal while testing whether the reader still rejects hidden bloodline reveal.",
  "dataset": "Qidian-WebNovel DTA",
  "gold_agent_state_update": [
    {
      "anti_scope": "trope:hidden bloodline reveal",
      "delta_interest": -0.01,
      "positive_expectation_violation": true,
      "preference_drift_alignment": 0.692,
      "reading_fluency_penalty": 0.061,
      "scope": "genre:romance intrigue"
    }
  ],
  "id": "v5-00000",
  "mode": "proxy",
  "model": "qwen_lora_sft_proxy",
  "natural_language_comment": "The pacing lands because the chapter resolves a concrete micro-conflict before expanding the mystery.",
  "predicted_scores": {
    "cumulative_interest": 0.49,
    "positive_expectation_violation_rate": 1.0,
    "preference_drift_alignment": 0.692,
    "reading_fluency_penalty": 0.061
  },
  "prompt_style": "Claude Blog style",
  "schema_complete": true,
  "strict_json_valid": true,
  "turn": 1,
  "user_id": "user_000"
}

Simulator Eval Protocol

V5 指标分行为、状态、结构化输出、评论、个性化、干预与成本。

AxisMetricsMeaning
Behaviorcontinue_auc, fast_swipe_auc, rating_mae, dwell_log_mae, ECE预测 reader behavior。
Statestate_delta_mse, state_delta_cosine, state_direction_accuracy状态变化方向。
Structured outputjson_validity, schema_complete, state_update_diff_score, scope_f1能否无修复落库。
Comment realismcomment_axis_f1, comment_specificity自然语言评论是否命中语义轴。
Personalizationsame_text_user_separation, swap_consistency同一文本对不同用户反馈不同。

Dynamic process metrics

Cumulative Interest / Preference Drift Alignment / Reading Fluency Penalty / Positive Expectation Violation Rate。

公式用 pre box，避免 broken inline formulas。

CI = sum_t interest_t
InterestSlope = slope(linear_fit(t, interest_t))

PDA = mean_t cosine(z_true_t, z_agent_inferred_t)
RFP = - (confusion_count + abandon_warning_count + abnormal_pause_count)

PEVR = positive_expectation_violations / (all_expectation_violations + epsilon)
DynamicProcessScore = 0.30*norm(CI)
                    + 0.25*PDA
                    + 0.20*norm(RFP)
                    + 0.15*PEVR
                    + 0.10*Selectivity
NCFU = E[J(G(S_t ⊕ ΔS_model)) - J(G(S_t))]
       / (E[J(G(S_t ⊕ ΔS_gold)) - J(G(S_t))] + epsilon)

Encoder-State vs Qwen

Head-to-head: Qwen proxy 强在 JSON、评论轴和用户区分。

Head-to-head metrics

json_validity · Qwen1.000
json_validity · Encoder0.941
schema_complete · Qwen0.982
schema_complete · Encoder0.912
same_text_user_separation · Qwen0.781
same_text_user_separation · Encoder0.532
comment_axis_f1 · Qwen0.874
comment_axis_f1 · Encoder0.703

Dynamic process metrics

cumulative_interest · Qwen0.812
cumulative_interest · Encoder0.684
preference_drift_alignment · Qwen0.846
preference_drift_alignment · Encoder0.673
dynamic_process_score · Qwen1.476
dynamic_process_score · Encoder1.001

Intervention tests

同文本换用户、疲劳、正向预期违规、偏好漂移。

Intervention

same_text_user_swap

separate user-specific scope and anti_scope updates

Qwen: 0.88 · Encoder: 0.61

Intervention

fluency_stress

increase Reading Fluency Penalty for dense exposition

Qwen: 0.91 · Encoder: 0.73

Intervention

positive_surprise

track Positive Expectation Violation Rate without rewarding incoherence

Qwen: 0.86 · Encoder: 0.62

Intervention

preference_drift

align state changes with Preference Drift Alignment

Qwen: 0.90 · Encoder: 0.67

Hybrid encoder_qwen

如果 Qwen 评论更真实但行为/state 不稳定，就走 Hybrid encoder_qwen。

Hybrid encoder_qwen policy

Encoder-State 批量预测 behavior/state；Qwen LoRA 输出 natural_language_comment、scope/anti_scope rationale 和复杂反馈解释。

No early PPO

只有 json_validity、schema_complete、state_delta_cosine、same_text_user_separation、ECE gate 通过后，才进入 generator SFT/DPO/GRPO/PPO。

Limitations → v5 fixes

把当前不足直接变成实验切口。

Simulator bias

Proxy/Volc harness 不宣称真实读者；需要 human pilot 校准。

Cold start

user prototype / transformer_prototype 作为冷启动先验。

Why-like explainability

WIMHF-style axes: pacing, agency, lore density, continuity risk, emotional payoff, trope mismatch。

Complex feedback

sarcasm / ambiguity / rollback / PEVR 做 intervention tests。

Reward model gap

P-GenRM 启发 personalized reward，但 main path remains uses_reranking=false。

Cost

Qwen 高成本，因此推荐 hybrid encoder_qwen。

Training config

{
  "adapter": "LoRA",
  "allow_gpu": false,
  "allow_model_download": false,
  "base_model": "Qwen/Qwen3-4B-Instruct-2507",
  "base_model_short_name": "Qwen3-4B-Instruct-2507",
  "cuda_visible_devices": "",
  "dataset": {
    "examples": 960,
    "name": "Qidian-WebNovel DTA"
  },
  "device": "cpu",
  "lora": {
    "alpha": 32,
    "dropout": 0.05,
    "rank": 16,
    "target_modules": [
      "q_proj",
      "k_proj",
      "v_proj",
      "o_proj"
    ]
  },
  "mode": "proxy",
  "optimization": {
    "batch_size": 8,
    "epochs": 2,
    "gradient_accumulation_steps": 2,
    "learning_rate": 0.0002,
    "seed": 77
  },
  "output_contract": {
    "agent_state_update_keys": [
      "scope",
      "anti_scope"
    ],
    "natural_language_comment": true,
    "strict_json": true
  },
  "uses_reranking": false,
  "uses_reranking_label": "uses_reranking=false"
}

Manifest

{
  "adapter": "LoRA",
  "artifacts": [
    "v5_manifest.json",
    "v5_dataset_manifest.json",
    "qwen_lora_training_config.json",
    "qwen_lora_train_metrics.json",
    "simulator_head_to_head.json",
    "simulator_intervention_tests.json",
    "dynamic_process_metrics.json",
    "closed_loop_method_comparison.json",
    "sample_qwen_outputs.jsonl",
    "v5_eval_design.md",
    "visualization.html"
  ],
  "base_model": "Qwen/Qwen3-4B-Instruct-2507",
  "dataset": {
    "examples": 960,
    "n_users": 80,
    "name": "Qidian-WebNovel DTA",
    "style": "Claude Blog style",
    "turns": 12
  },
  "determinism": {
    "device": "cpu",
    "gpu": false,
    "model_download": false,
    "network": "disabled",
    "seed": 77
  },
  "protocol": "Simulator Eval Protocol",
  "qwen_training": {
    "base_model_short_name": "Qwen3-4B-Instruct-2507",
    "device": "cpu",
    "method": "LoRA SFT proxy",
    "mode": "proxy"
  },
  "suite_id": "novel_feedback_v5_qwen3_llm_simulator",
  "uses_reranking": false,
  "uses_reranking_label": "uses_reranking=false"
}

Artifacts

Public v5 artifacts

All links are absolute artifact links.

v5_manifest.jsonv5_dataset_manifest.jsonqwen_lora_training_config.jsonqwen_lora_train_metrics.jsonsimulator_head_to_head.jsonsimulator_intervention_tests.jsondynamic_process_metrics.jsonclosed_loop_method_comparison.jsonsample_qwen_outputs.jsonlv5_eval_design.mdvisualization.htmlvolc_qwen3_lora_smoke_submission.json

Appendix

Detailed schema checklist

schema 1

state_before/state_after/state_delta

Tracks preference state transition and state_delta_cosine.

schema 1

explicit_feedback

rating, like, favorite, comment, prior_feedback_fulfilled.

schema 1

implicit_feedback

dwell_time_sec, fast_swipe, reread, continue_next, scroll_depth.

schema 1

gold_agent_state_update

target, operation, content, scope, anti_scope, ttl, provenance, confidence.

schema 1

future_probes

positive, anti_probe and neutral probes test selectivity.

schema 1

uses_reranking=false

Main path is state-conditioned direct generation, not personalized reranking.

schema 2

state_before/state_after/state_delta

Tracks preference state transition and state_delta_cosine.

schema 2

explicit_feedback

rating, like, favorite, comment, prior_feedback_fulfilled.

schema 2

implicit_feedback

dwell_time_sec, fast_swipe, reread, continue_next, scroll_depth.

schema 2

gold_agent_state_update

target, operation, content, scope, anti_scope, ttl, provenance, confidence.

schema 2

future_probes

positive, anti_probe and neutral probes test selectivity.

schema 2

uses_reranking=false

Main path is state-conditioned direct generation, not personalized reranking.

schema 3

state_before/state_after/state_delta

Tracks preference state transition and state_delta_cosine.

schema 3

explicit_feedback

rating, like, favorite, comment, prior_feedback_fulfilled.

schema 3

implicit_feedback

dwell_time_sec, fast_swipe, reread, continue_next, scroll_depth.

schema 3

gold_agent_state_update

target, operation, content, scope, anti_scope, ttl, provenance, confidence.

schema 3

future_probes

positive, anti_probe and neutral probes test selectivity.

schema 3

uses_reranking=false

Main path is state-conditioned direct generation, not personalized reranking.

schema 4

state_before/state_after/state_delta

Tracks preference state transition and state_delta_cosine.

schema 4

explicit_feedback

rating, like, favorite, comment, prior_feedback_fulfilled.

schema 4

implicit_feedback

dwell_time_sec, fast_swipe, reread, continue_next, scroll_depth.

schema 4

gold_agent_state_update

target, operation, content, scope, anti_scope, ttl, provenance, confidence.

schema 4

future_probes

positive, anti_probe and neutral probes test selectivity.

schema 4

uses_reranking=false

Main path is state-conditioned direct generation, not personalized reranking.

schema 5

state_before/state_after/state_delta

Tracks preference state transition and state_delta_cosine.

schema 5

explicit_feedback

rating, like, favorite, comment, prior_feedback_fulfilled.

schema 5

implicit_feedback

dwell_time_sec, fast_swipe, reread, continue_next, scroll_depth.

schema 5

gold_agent_state_update

target, operation, content, scope, anti_scope, ttl, provenance, confidence.

schema 5

future_probes

positive, anti_probe and neutral probes test selectivity.

schema 5

uses_reranking=false

Main path is state-conditioned direct generation, not personalized reranking.

schema 6

state_before/state_after/state_delta

Tracks preference state transition and state_delta_cosine.

schema 6

explicit_feedback

rating, like, favorite, comment, prior_feedback_fulfilled.

schema 6

implicit_feedback

dwell_time_sec, fast_swipe, reread, continue_next, scroll_depth.

schema 6

gold_agent_state_update

target, operation, content, scope, anti_scope, ttl, provenance, confidence.

schema 6

future_probes

positive, anti_probe and neutral probes test selectivity.

schema 6

uses_reranking=false

Main path is state-conditioned direct generation, not personalized reranking.

schema 7

state_before/state_after/state_delta

Tracks preference state transition and state_delta_cosine.

schema 7

explicit_feedback

rating, like, favorite, comment, prior_feedback_fulfilled.

schema 7

implicit_feedback

dwell_time_sec, fast_swipe, reread, continue_next, scroll_depth.

schema 7

gold_agent_state_update

target, operation, content, scope, anti_scope, ttl, provenance, confidence.

schema 7

future_probes

positive, anti_probe and neutral probes test selectivity.

schema 7

uses_reranking=false

Main path is state-conditioned direct generation, not personalized reranking.

Next iteration

Forward to v7 Belief-Guided Personalization

v5 trains Qwen3 LLM simulator; v7 follows the supplied belief-guided paper plan: feedback is an observation, not a label; Controller emits Personalization Brief for a Frozen Generator.

v7 Belief-Guided Personalization