OPeRA-EA：先把“提前感知最终 intent”问题做清楚，再证明方法超过 BM25

Research pain point

电商/生活 Agent 的失败经常不是不会推荐，而是太晚感知用户真实目标，或者看到早期噪声后过早锁错 top-1。EA 把问题收敛为 closed-domain final intent distribution：在每个 prefix 上输出 top-k + uncertainty，让 Planner 决定 ASK / ACT / DEFER。

和 BM25 的边界

BM25 能利用搜索词、商品标题、ASIN/title overlap；在 OPeRA 这种 title-heavy 数据上非常强。你的方法要成立，必须在 BM25 不容易解决的场景中赢：目标晚暴露、早期是 use-case/constraint 而非商品名、存在 hard negative drift、需要 step aggregation 和 noise gate。

Trajectory schema

{
  "traj_id": "op_s_000123",
  "source_session_id": "original OPeRA session id",
  "user_id": "workspace-local anonymized user id",
  "split": "train|valid|test (user-disjoint)",
  "domain": "amazon_shopping",
  "final_intent_id": "op_g_042",
  "final_intent_name": "product-title cluster name",
  "final_intent_source": "purchase_click_direct_asin|cart_delta_unique|last_engaged_product",
  "label_confidence": "strong|medium|weak",
  "T_raw": 74,
  "T": 30,
  "t_star": 21,
  "t_star_source": "asin_match|product_title_match|cart_asin_match|anchor_fallback",
  "noise_ratio": 0.2,
  "steps": [
    {
      "t": 1,
      "s": "state text",
      "a": "action text",
      "action_type": "search|click|scroll|navigation",
      "explicit_final_intent": false,
      "is_noise": false,
      "noise_type": null
    }
  ]
}

Prefix schema

{
  "prefix_id": "op_s_000123_p021",
  "traj_id": "op_s_000123",
  "user_id": "...",
  "split": "test",
  "prefix_t": 21,
  "prefix_ratio": 0.5,
  "is_pre_t_star": true,
  "t_star": 29,
  "T": 42,
  "final_intent_id": "op_g_042",
  "candidate_intents": [
    "op_g_000",
    "op_g_001",
    "..."
  ],
  "text_prefix": "[STATE] ... [ACTION] ... [TYPE] ...",
  "steps": "prefix steps only; labels not used at inference",
  "noise_ratio_in_prefix": 0.19,
  "contains_explicit_final_intent": false,
  "label_confidence": "strong"
}

Intent catalog schema

{
  "intent_id": "op_g_042",
  "intent_type": "product_title_cluster",
  "name": "lightweight sleep tracker band / product phrase",
  "aliases": [
    "title alias"
  ],
  "representative_asins": [
    "ASIN..."
  ],
  "representative_titles": [
    "raw product title"
  ],
  "support": {
    "sessions": 10,
    "purchase_clicks": 8,
    "cart_delta": 1,
    "last_engaged": 1
  },
  "neighbor_intents": [
    "op_g_018"
  ],
  "hard_negative_intents": [
    "op_g_019"
  ]
}

EA prediction schema

{
  "prefix_id": "op_s_000123_p021",
  "gold": {
    "intent_id": "op_g_042",
    "name": "..."
  },
  "top_k_intents": [
    {
      "intent_id": "op_g_042",
      "name": "...",
      "prob": 0.38
    }
  ],
  "uncertainty": "high|medium|low",
  "suggested_policy": "ASK_DISAMBIGUATING_QUESTION|ACT_WITH_TOP_K_GUARDRAILS",
  "text_prefix_excerpt": "first 600 chars"
}

Dataset stats

{
  "dataset": "OPeRA-EA",
  "source_copy": "/root/code/vepfs/hermes/autoresearch/life_agent/1_EA/dataset/OPeRA_EA",
  "version": "full",
  "sessions_raw": 692,
  "actions_raw": 28904,
  "unified_actions": 28904,
  "users": 54,
  "intents": 50,
  "trajectories": 80,
  "prefixes": 803,
  "pre_t_star_prefixes": 249,
  "mean_T": 18.825,
  "mean_noise_ratio": 0.40297374337903796,
  "label_confidence_counts": {
    "strong": 56,
    "medium": 5,
    "weak": 19
  },
  "dropped_sessions": {
    "not_in_top_intents_or_unresolved": 570,
    "too_short": 42
  }
}

Leakage checks

{
  "user_overlap_train_valid": 0,
  "user_overlap_train_test": 0,
  "user_overlap_valid_test": 0,
  "pre_t_star_prefix_contains_explicit_final_intent": 0,
  "prefix_text_contains_final_intent_id": 0
}

Metric definitions

{
  "Hit@5 = 1[rank(g*) ≤ 5]": "gold final intent appears in the top-5 candidate distribution.",
  "NDCG@5 = 1/log2(rank(g*)+1) if rank≤5 else 0": "rank-sensitive top-5 quality; higher if the true intent is ranked earlier.",
  "MRR@10 = 1/rank(g*) if rank≤10 else 0": "reciprocal-rank metric for top-10 retrieval.",
  "Pre-t* AUC_NDCG@5": "mean NDCG@5 over prefixes with prefix_t < t*, before explicit final-intent exposure.",
  "LeadTime@5 = t* - min{t<t*: g* in Top5(prefix_t)}": "how many steps before explicit exposure the model already includes the final intent in top-5.",
  "NormalizedLead@5 = LeadTime@5 / T": "lead time normalized by trajectory length.",
  "Coverage@5": "fraction of trajectories for which any pre-t* prefix hits top-5.",
  "ECE": "expected calibration error over top-1 confidence bins.",
  "Brier": "mean squared error between predicted distribution and one-hot intent label.",
  "NLL": "negative log likelihood of the gold final intent.",
  "FCR@0.5": "False Commitment Rate: pre-t* wrong top-1 predictions whose confidence is at least 0.5."
}

Model table

可信结论

• OPeRA → OPeRA-EA 的 schema、split、prefix、metrics pipeline 已经可复查。
• TF-IDF candidate scorer 是强 baseline，在 title/search heavy OPeRA 上 Hit@5 = 0.8380。
• 当前 EA-Predictor 的主要收益是校准：ECE/Brier/NLL 改善，而非排序改善。

不能声称的结论

• 不能说 EA 方法已经超过 BM25。
• 不能把 TF-IDF+scale 写成完整 Step Encoder + Noise Gate + Transformer。
• 不能用 LLM utility proxy 代替真实 frozen Planner 评测。

数据侧

1. 控制 t*/T：减少 0-20% easy bucket，提高 40-90% late exposure。
2. 构造 pre-t* 语义线索：use-case、constraint、budget、compatibility、场景路径，而不是提前出现商品 title。
3. 加入 hard negative drift：相似 intent 长时间干扰，测试 FCR。
4. 平衡每个 intent 的 train/valid/test support，减少 long-tail ceiling。
5. 增加 Synerise 主数据或合成 EA-hard 数据；OPeRA 只作为 case study。

模型侧

1. 实现真正 Step Encoder（BGE/GTE/MiniLM）+ step-level Noise Gate。
2. 用 GRU/Lightweight Transformer/Segment Aggregator 聚合时序证据。
3. Candidate scorer 使用 prefix_emb 与 intent_emb 的 dot/bilinear，而不是直接 TF-IDF cosine。
4. 加入 ambiguity soft target 与 calibration loss，降低早期错误高置信。
5. 做 ablation：no gate、no sequence、BM25 lexical、embedding retrieval、SBR baseline。

Intent examples

[
  {
    "intent_id": "op_g_000",
    "intent_type": "product_title_cluster",
    "name": "scott comfortplus toilet paper double rolls sheets",
    "aliases": [],
    "representative_asins": [
      "B07BGLT25K",
      "/promo/A3J5TZR6FTI151",
      "/promo/A29SFZNGDLW6NN"
    ],
    "representative_titles": [
      "Scott ComfortPlus Toilet Paper, 12 Double Rolls, 231 Sheets per Roll, Septic-Safe, 1-Ply Toilet Tissue"
    ],
    "support": {
      "sessions": 10,
      "purchase_clicks": 8,
      "cart_delta": 0,
      "last_engaged": 2
    },
    "neighbor_intents": [
      "op_g_009",
      "op_g_028"
    ],
    "hard_negative_intents": []
  },
  {
    "intent_id": "op_g_001",
    "intent_type": "product_title_cluster",
    "name": "tea tree oil foot soak epsom salt",
    "aliases": [],
    "representative_asins": [
      "B0829DGWD5"
    ],
    "representative_titles": [
      "Tea Tree Oil Foot Soak with Epsom Salt - For Toenail Repair, Athletes Foot, Softens Calluses, Soothes Sore & Tired Feet, Nail Discoloration, odor Scent, Spa Pedicure Care - Made in USA 16 oz"
    ],
    "support": {
      "sessions": 4,
      "purchase_clicks": 0,
      "cart_delta": 0,
      "last_engaged": 4
    },
    "neighbor_intents": [
      "op_g_026",
      "op_g_021",
      "op_g_015"
    ],
    "hard_negative_intents": []
  },
  {
    "intent_id": "op_g_002",
    "intent_type": "product_title_cluster",
    "name": "cat spring toys packs colorful plastic spring",
    "aliases": [],
    "representative_asins": [
      "B0DG25N8P6"
    ],
    "representative_titles": [
      "Cat Spring Toys 40 Packs Colorful Plastic Spring Coils Attract Cats to Swat, Bite, Cat Toys for Indoor Cats for All Breeds"
    ],
    "support": {
      "sessions": 4,
      "purchase_clicks": 4,
      "cart_delta": 0,
      "last_engaged": 0
    },
    "neighbor_intents": [],
    "hard_negative_intents": []
  },
  {
    "intent_id": "op_g_003",
    "intent_type": "product_title_cluster",
    "name": "zeagoo sleeveless button down shirts blouses solid",
    "aliases": [],
    "representative_asins": [
      "B0BZDKCQQ2",
      "B07334PKW6"
    ],
    "representative_titles": [
      "Zeagoo Women's Sleeveless Button Down Shirts Blouses Solid Casual Loose V Neck Tank Tops for Work S-XXL"
    ],
    "support": {
      "sessions": 3,
      "purchase_clicks": 3,
      "cart_delta": 0,
      "last_engaged": 0
    },
    "neighbor_intents": [
      "op_g_023",
      "op_g_044",
      "op_g_007"
    ],
    "hard_negative_intents": []
  },
  {
    "intent_id": "op_g_004",
    "intent_type": "product_title_cluster",
    "name": "lodge cast iron pre seasoned skillet signature",
    "aliases": [
      "Lodge 9 Inch Cast Iron Pre-Seasoned Skillet – Signature Teardrop Handle - Use in the Oven, on the Stove, on the Grill, or Over a Campfire, Black"
    ],
    "representative_asins": [
      "B00006JSUA",
      "B00063RWTS"
    ],
    "representative_titles": [
      "Lodge 10.25 Inch Cast Iron Pre-Seasoned Skillet – Signature Teardrop Handle - Use in the Oven, on the Stove, on the Grill, or Over a Campfire, Black",
      "Lodge 9 Inch Cast Iron Pre-Seasoned Skillet – Signature Teardrop Handle - Use in the Oven, on the Stove, on the Grill, or Over a Campfire, Black"
    ],
    "support": {
      "sessions": 3,
      "purchase_clicks": 2,
      "cart_delta": 1,
      "last_engaged": 0
    },
    "neighbor_intents": [],
    "hard_negative_intents": []
  },
  {
    "intent_id": "op_g_005",
    "intent_type": "product_title_cluster",
    "name": "kitchenaid hard anodized ceramic nonstick pots pans",
    "aliases": [],
    "representative_asins": [
      "B0DM6ZLKF8"
    ],
    "representative_titles": [
      "KitchenAid Hard Anodized Ceramic Nonstick Pots and Pans Set with Saucepans, Frying Pans, Stockpot, and Lids, 10 Piece Non Stick Cookware Set with Stainless Steel Handles - Porcelain White"
    ],
    "support": {
      "sessions": 3,
      "purchase_clicks": 2,
      "cart_delta": 0,
      "last_engaged": 1
    },
    "neighbor_intents": [],
    "hard_negative_intents": []
  },
  {
    "intent_id": "op_g_006",
    "intent_type": "product_title_cluster",
    "name": "instinct grain recipe variety natural wet canned",
    "aliases": [],
    "representative_asins": [
      "B07NM328YZ",
      "B01FV0F8H8"
    ],
    "representative_titles": [
      "Instinct Original Grain Free Recipe Variety Pack Natural Wet Canned Cat Food by Nature's Variety, 3 Ounce (Pack of 12)"
    ],
    "support": {
      "sessions": 3,
      "purchase_clicks": 3,
      "cart_delta": 0,
      "last_engaged": 0
    },
    "neighbor_intents": [
      "op_g_030",
      "op_g_019"
    ],
    "hard_negative_intents": []
  },
  {
    "intent_id": "op_g_007",
    "intent_type": "product_title_cluster",
    "name": "comfneat lightweight shirts cotton crew neck regular",
    "aliases": [],
    "representative_asins": [
      "B09NBP8811"
    ],
    "representative_titles": [
      "Comfneat Men's 3-Pack Lightweight T-Shirts Cotton Crew Neck Regular Fit Solid Tee"
    ],
    "support": {
      "sessions": 3,
      "purchase_clicks": 3,
      "cart_delta": 0,
      "last_engaged": 0
    },
    "neighbor_intents": [
      "op_g_044",
      "op_g_023",
      "op_g_009"
    ],
    "hard_negative_intents": []
  }
]

EA output sample

[
  {
    "prefix_id": "op_s_000019_p001",
    "traj_id": "op_s_000019",
    "prefix_t": 1,
    "t_star": 2,
    "gold": {
      "intent_id": "op_g_032",
      "name": "oral io series electric toothbrush brush head"
    },
    "top_k_intents": [
      {
        "intent_id": "op_g_013",
        "name": "all kindle paperwhite gb our fastest kindle",
        "prob": 0.02286420576274395
      },
      {
        "intent_id": "op_g_000",
        "name": "scott comfortplus toilet paper double rolls sheets",
        "prob": 0.019941547885537148
      },
      {
        "intent_id": "op_g_002",
        "name": "cat spring toys packs colorful plastic spring",
        "prob": 0.019941547885537148
      },
      {
        "intent_id": "op_g_001",
        "name": "tea tree oil foot soak epsom salt",
        "prob": 0.019941547885537148
      },
      {
        "intent_id": "op_g_004",
        "name": "lodge cast iron pre seasoned skillet signature",
        "prob": 0.019941547885537148
      }
    ],
    "uncertainty": "high",
    "suggested_policy": "ASK_DISAMBIGUATING_QUESTION",
    "text_prefix_excerpt": "[STATE] 当前页面为 Amazon 购物交互页面。 [ACTION] 用户打开或切换到新的 Amazon 页面。 [TYPE] navigation"
  }
]