Paul
AI customer-service agent for Homebase. RAG-grounded, confidence-scored, human-escalated.
A support agent that answers from synced docs, scores its own confidence, and hands off to a human the moment it is unsure.
Paul handles inbound chat and email for Homebase across the marketing site and product. Every message is classified by intent, answered from a Notion-synced knowledge base retrieved through Cloudflare Vectorize, and scored for confidence before anything is sent. PII, loop, and domain guardrails gate every response, and low-confidence cases escalate to a human in Slack with full context instead of guessing. The whole agent runs on Cloudflare Workers, and a daily eval loop scores answer groundedness and retrieval quality in production, handing findings off to the dev workflow automatically.
Every answer is grounded in synced docs and scored for confidence. The moment it is not sure, it hands off to a human instead of guessing.
- 01Every reply grounded in a Notion-synced knowledge base, retrieved via Vectorize
- 02Confidence gate on every answer: low-scoring replies escalate to a human in Slack
- 03Daily production evals score groundedness and retrieval quality per category
- 04PII, loop, and domain guardrails on every response; HMAC + idempotency on every webhook
01. Conversational AI
- Intent classification routes product questions, pricing, support, and partnerships to the right response strategy
- RAG-powered responses grounded in retrieved knowledge chunks, not model memory
- Confidence scoring on every reply; low-confidence answers escalate instead of guessing
- Context retention across message threads for coherent multi-turn conversations
02. Knowledge base
- Automatic sync from internal Notion documentation
- Documents chunked and embedded via Voyage for semantic retrieval
- Cloudflare Vectorize vector search at inference time
- Dev endpoints for triggering KB sync and testing retrieval quality
03. Safety and quality
- PII guardrails detect and redact personal information before processing
- Loop detection prevents repetitive or circular conversation patterns
- Domain boundaries: refuses questions outside its trained domain
- Human escalation to Slack when confidence thresholds are not met
04. Evaluation loop
- Daily quality eval posted to Slack: auto-send, escalation, and groundedness rates per run
- Retrieval eval scoring top-1 and top-3 accuracy across labeled question categories
- Findings handed off automatically to a Linear issue-creator agent
- Built on the Howdah Eval Harness for faithfulness scoring
05. Integrations
- HubSpot Conversations as the primary channel: webhooks in, replies out via the Conversations API
- Slack interactive flows for team notifications, escalation handling, and approvals
- HMAC signature verification on all inbound webhooks
- Idempotency layer prevents double-processing of webhook events
+-----------------------------------------------------------+
| Inbound channels |
| HubSpot Conversations Slack interactive |
| (webhook -> HMAC verify) (actions + views) |
+-----------------------------------------------------------+
|
v
+-----------------------------------------------------------+
| Paul Worker (Hono) |
| |
| Webhook handler: |
| HMAC verify -> idempotency (KV) -> queue |
| | |
| v |
| Orchestrator: |
| classify -> retrieve -> generate -> score |
| |
| +----------+ +------------+ +--------------------+ |
| | KB sync | | Vectorize | | Guardrails | |
| | (Notion) | | (search) | | PII, loops, domain | |
| +----------+ +------------+ +--------------------+ |
| |
| D1 (Drizzle): conversations, messages, KB chunks, audit |
+-----------------------------------------------------------+
| | |
v v v
+------------+ +------------+ +------------+
| HubSpot | | Slack | | Claude |
| (reply) | | (escalate) | | (LLM) |
+------------+ +------------+ +------------+Webhooks verified and deduped, answers grounded and scored; anything uncertain escalates to a human in Slack.
