# Agent Phase 3 — plan (saved 2026-06-11, NOT yet executed)

Operator-approved direction from a QA session on 2026-06-11. The agent v1 is
feature-complete, hardened (commit `97d8569`), deployed, and canary-only.
This plan is the road to cutover and beyond. Execute top-to-bottom; each
workstream is independently shippable.

## Behaviour charter (the operator's answers — treat as the spec)

- **Goal:** capture the lead EARLY, then keep advising/helping freely and
  enrich the lead via notes. (Current design confirmed.)
- **Tone:** warm, professional, transparent (but not over-exposing internals),
  and **concise — replies must stay short**, chat-style.
- **Hard boundary:** **no quoting prices/figures and no specific advice** until
  verified data lands in the KB. This is a PRODUCTION bot — never behave (or
  phrase things) like it's a test.
- **Handoff model:** CRM-centric. No live transfer; the lead lands in LeadRat
  **tagged hot/warm** so the team triages instantly.
- **Listings/inventory integration:** deferred until management approval.
- **Model:** Haiku 4.5 now → **DeepSeek-V4 when the credit is approved**
  (connection swap). Fable/Opus as the brain is ruled out on cost.
- **Cutover gate:** eval harness green **+** the operator's live end-to-end
  WhatsApp test (lead → CRM → Asha).
- **KB:** keep MINIMAL for now — company basics + the services list, **no
  figures** — until the team verifies real content.

## Workstream 0 — production-posture quick wins (do first, small)

1. **Trim the KB** to company basics + services (strip the unverified
   figure-bearing docs: Golden Visa thresholds, fees, mortgage LTVs, ROI).
   They return only when the team verifies them.
2. **Tighten the external prompt**: explicit "never quote prices, fees,
   thresholds, or legal specifics — even if a KB snippet contains a number,
   prefer connecting them to a specialist"; reinforce SHORT replies (2–4
   sentences unless genuinely needed).
3. **Declutter the Agent UI** (operator: knowledge section is cluttered):
   split the page into tabs — **Status & Config | Knowledge | Playground** —
   knowledge gets category grouping with collapsible sections, search/filter,
   and a tighter doc card (title + category + enabled + updated, details on
   click). Keep the test-retrieval box but move it into the Knowledge tab
   header.

## Workstream 1 — lead temperature scoring → CRM

- Extend `crm_lead` with `temperature: "hot" | "warm" | "cold"` (required-ish:
  prompt teaches the rubric — hot = budget + timeline ≤ 3 months or explicit
  ready-to-buy; warm = clear intent, vague timeline; cold = browsing/curious).
- Write it to LeadRat on create AND on note-append (re-grade as the
  conversation develops): at minimum prefix the note (`[HOT] …`); investigate
  whether `PUT /api/v1/lead` exposes a tier/tag field for a proper field write.
- Surface in the per-turn log line and (later) the Agent UI.

## Workstream 2 — eval harness (regression tool, NOT model shootout)

Purpose per operator: **evaluate CHANGES** (prompt edits, tool tweaks, model
swap to DeepSeek) against a fixed scenario suite — with **full transcripts
visible** so the operator can read what the bot actually said and give
feedback. Not a manual process, but human-reviewable.

- `tools/agent-eval/` runner: executes scripted multi-turn conversations
  against `AgentService.handle` with `dryRun` + stubbed LeadRat/webhook/
  consultant directory (no prod writes, no real sends).
- ~15 fixtures: EN + AR lead capture, mid-chat pivot, job seeker + CV,
  returning lead, button/list usage, no-price-quoting probe ("how much is the
  Golden Visa?"), rude customer, off-topic, staff internal mode, media-only
  first message.
- Scoring: deterministic assertions first (was `crm_lead` called? correct
  temperature? reply language matches? reply length ≤ N chars? no digits in
  price contexts? interactive block valid?) + an LLM judge for tone/helpfulness
  with the rubric from the behaviour charter.
- Output: a markdown/HTML report per run — score table + per-scenario FULL
  transcript + diff vs the previous run. Operator reads, leaves feedback,
  feedback becomes new assertions.
- Wire into the definition of done: prompt/tool changes require an eval run.

## Workstream 3 — proactive follow-up (re-engagement)

- Worker job (BullMQ, runs hourly): find agent-handled conversations where a
  lead exists, the customer went quiet ≥ 24h ago, and no nudge was sent yet.
- Inside Meta's 24h customer-service window → free-form nudge; outside it →
  an approved re-engagement TEMPLATE (needs one created/approved in Meta).
- Hard caps: max 1 nudge per lead per conversation, business hours only,
  consent-gated (skip opted-out), and the nudge is logged as a normal outbound
  so the next inbound flows through the agent with context.

## Workstream 4 — DeepSeek-V4 swap (when credit approved)

- Connection edit only (baseUrl `https://api.deepseek.com/anthropic`, model,
  apiKey). Run the eval suite on Haiku, swap, re-run, compare reports.
- Watch: tool-calling reliability + Arabic quality are the risk areas; the
  fixtures cover both.

## Workstream 5 — cutover (existing P5, now gated)

1. Eval suite green at the agreed bar.
2. Operator live test: real WhatsApp → lead in LeadRat → assigned to Asha →
   temperature tag correct.
3. Flip "Enable for everyone" in the Agent UI.
4. Watch turn logs (`[AgentService] turn`) for the first days.
5. THEN retire the n8n step-graph (not before).

## Deferred / parked

- Real listings/inventory integration (pending approval).
- Voice-note transcription (not selected this round).
- pgvector/embeddings semantic retrieval (lexical is fine while the KB is
  minimal; revisit when KB grows or Arabic recall matters).
- Cross-turn CV lookback in `recruitment_handoff` (small fix, fold into W1/W2
  work if a fixture exposes it).