# Novel Feedback v4 evaluation design

## Behavior fidelity

Measure whether a simulator predicts reader behavior, not just whether a method produces nice-looking text: continue AUC, comment AUPRC, log-dwell MAE, fast-swipe AUC, Brier score and calibration ECE. Segment results by cold-start, drift, noisy feedback and implicit-only scenarios.

## State fidelity

State is the object the agent must update. Evaluate state-delta MSE, **state-delta cosine**, directional accuracy per state field, boundedness violations and adaptation half-life after a preference change.

## Story / canon consistency

Maintain a story bible with entities, relationships, facts, open loops, plot promises and user constraints. Score generated later content with claim extraction + retrieval + NLI/QA: contradiction rate, unsupported major fact rate, invalid retcon rate, entity-status accuracy, character voice consistency and outline adherence.

## Long-horizon credit assignment

Delayed feedback should update the right event and not the most recent chapter by default. Report culprit event F1, credit nDCG, repair-gain correlation, feedback-to-future trace and rollback scar rate.

## Intervention tests

Run same-story/different-user swaps, same-user/different-content swaps, fatigue up/down interventions, hidden preference probes, rollback/retraction probes, and cross-user contamination checks. These determine whether the simulator is truly personalized rather than a generic sentiment classifier.

## Bridge Synthetic-to-Real

A practical human split is a **Crowdsourced browser-plugin reading experiment**: opt-in readers consume public-domain or author-consented serialized fiction, choose among continuation directions, rate chapters, optionally comment, and permit local logging of dwell/continue/scroll/reread. The plugin exports `UserTrajectory` records without protected platform scraping.

Qidian-Webnovel reader comments/replies remain a DTA path only. No protected Qidian reader-response scraping is part of v4.