# Novel Feedback v3 Experiment Suite

## Summary

- Suite ID: `novel_feedback_v3_suite`
- Experiments: 96 = 12 scenarios × 8 methods.
- Core method direction: state-conditioned direct generation; **no personalized reranking**.
- Qidian DTA adapter: Qidian/WebNovel metadata is retained as a DTA-ready path; reader comments/replies require a Data Transfer Agreement, and this suite performs no bypass or protected scraping.
- Best non-oracle average dynamic_return: `uncertainty_aware_planning = 2.5509`.
- State-conditioned planning uplift over memory-only: `0.2748`.

## What was optimized

This suite turns the single v1 smoke experiment into a matrix that can support many future experiments: data-condition ablations, simulator variants, closed-loop update methods, stress tests, counterfactual validation, rollback recovery, contamination checks, direct-generation ablation, and GPU-scale sequence simulator replacement.

## Data conditions and legal status

- `public_text_augmented_feedback` — WebNovelBench/NovelUpdates public snapshots plus calibrated feedback trajectories. License/status: `public_or_metadata_review_required`.
- `qidian_metadata_dta_ready` — Qidian/WebNovel metadata mapping and DTA-ready adapter path; no protected comments are copied or scraped. License/status: `metadata_only_dta_required_for_reader_response`.
- `implicit_action_augmented` — Implicit dwell/fast-swipe/reread/continue perturbations for ambiguity stress tests. License/status: `derived_from_public_snapshot`.
- `counterfactual_user_swap` — Synthetic counterfactual swaps of user archetype and content axes for simulator validity tests. License/status: `synthetic_from_public_features`.

## GPU path

Status: `ready_not_submitted`. CPU suite is sufficient for deterministic artifact generation; GPU should be used for the next sequence/encoder simulator training run, not for this synthetic matrix expansion.

Recommended next GPU task: `novel_feedback_v3_sequence_simulator`, using the VolcEngine task-specific YAML workflow and keeping all credentials/logs inside `/root/code/vepfs/hermes/gpu`.

## Metric definitions

### `dynamic_return`

Mean simulated future engagement/reward over a multi-chapter rollout after applying the method update.

```text
DynamicReturn = E_t[ rating_t + 0.6*continue_t + 0.3*reread_t - 0.4*fast_swipe_t - 0.2*fatigue_t ]
```

Higher is better; not bounded and should be compared across methods within the same scenario.

### `expectation_alignment`

How well the next direct-generation plan satisfies expectations created by earlier chapters and feedback.

```text
ExpectationAlign = |matched_created_expectations| / (|created_expectations| + epsilon)
```

Bounded [0,1]; higher means the planner used dynamic reader state rather than generic profile text.

### `ncfu`

Normalized causal future uplift against no-update and oracle-update controls.

```text
NCFU = E[J(G(S_t ⊕ ΔS_model)) - J(G(S_t))] / (E[J(G(S_t ⊕ ΔS_gold)) - J(G(S_t))] + epsilon)
```

Bounded [0,1] in this suite; higher means the update causes future improvement rather than just correlating with it.

### `selectivity`

Positive-probe uplift minus anti-probe harm/overgeneralization.

```text
Selectivity = Uplift(Q_positive) - Harm(Q_anti_probe)
```

Higher is better; a method can have high return but low selectivity if it overgeneralizes local feedback.

### `regret_to_oracle`

Regret-to-oracle: dynamic-return gap between method and hidden oracle state update.

```text
RegretToOracle = DynamicReturn(oracle_update) - DynamicReturn(method)
```

Lower is better; reports how much room remains before hidden-state upper bound.

### `calibration_ece`

Expected calibration error of simulator/user-state confidence bins.

```text
ECE = Σ_b |B_b|/n · | accuracy(B_b) - confidence(B_b) |
```

Lower is better; needed because implicit feedback and sarcasm are ambiguous.

### `counterfactual_separation`

Distance between outcomes under same text/different users and same user/different content axes.

```text
CFS = E[ | f(u_i, x_a) - f(u_j, x_a) | + | f(u_i, x_a) - f(u_i, x_b) | ] / 2
```

Higher means the simulator is not a static sentiment classifier.

### `rollback_success`

Fraction of corrected/retracted preference updates that are undone without harming later valid preferences.

```text
RollbackSuccess = valid_retractions_repaired / (all_retractions + epsilon)
```

Higher is better; requires TTL/provenance/scope in state diffs.

### `behavioral_scar_rate`

Rate at which an invalidated feedback update continues to affect future generation after rollback.

```text
ScarRate = lingering_invalid_effects / (rollback_events + epsilon)
```

Lower is better.

### `contamination_leakage`

Fraction of private user-preference updates that affect another user or global story state.

```text
Leakage = cross_user_private_effects / (private_updates + epsilon)
```

Lower is better; must be near zero for multi-user personalization.

### `direct_generation_diversity`

Diversity of direct chapter plans/excerpts produced under different reader states without reranking candidate sets.

```text
Diversity = mean_pairwise_distance(plan_axes | same_story, different_user_state)
```

Higher means the generator actually changes outputs with state; too high may hurt story coherence.

### `no_reranking_compliance`

Binary compliance flag that the method produced a direct plan/generation prompt rather than ranking K candidate chapters.

```text
NoRerankingCompliance = 1[ no candidate_set and no reranked_candidates and uses_reranking=false ]
```

Must be 1.0 for Lucian’s direct-generation research direction.

## Direct-generation ablation / no reranking

The suite requires every matrix row, scenario result, sample and public artifact to set `uses_reranking=false`. Samples contain `direct_generation_prompt`, `chapter_plan`, `generated_excerpt`, and `state_diff_applied`; they deliberately omit `candidate_set` and `reranked_candidates`.