Build and Evaluate a Personalized Shopping Assistant
Before you start: Agent fundamentals and search evaluation.
A shopping assistant should turn “I need comfortable waterproof shoes for a weekend trip” into useful, available options. The system needs intent understanding, catalog retrieval, personalization, and a response grounded in product facts. Multiple agents can separate responsibilities, but the design earns its complexity only if those responsibilities are explicit and measurable.
Define the shared decision state
state = {
"query": "waterproof shoes for a weekend trip",
"constraints": {"size": "8", "max_price": 5000},
"preferences": {"style": "casual"},
"candidate_ids": [],
"catalog_version": "catalog-v4",
"evidence": {},
"unresolved": ["preferred activity: city walking or hiking?"],
}The search component returns item IDs, scores, and factual attributes. Personalization can rerank those candidates using prior preferences, but it must not override stock, size, access, or other hard constraints. The response component reads the same item records and cites or links the corresponding catalog entries. Each handoff has a schema and validation rules.
Separate responsibilities where it helps
An intent component can resolve ambiguous requests, a search component can use lexical and dense retrieval tools, and a personalization component can apply user context. A coordinator decides whether to ask a focused clarification or proceed with available constraints. Use deterministic code for filters and arithmetic that do not require language reasoning.
Parallel calls can reduce latency when independent, but merging their results needs a clear policy. If one component returns “waterproof” and another only infers it from an image, the catalog fact should control the final claim. Bound tool retries and total work, and preserve a trace of which data each component used.
Connect the assistant to an evaluated search stack
Measure relevance under the user’s full context, not only the raw query string. Build test cases for strict budgets, out-of-stock items, cold-start users, conflicting preferences, and requests requiring clarification. The retrieval-stage diagnostics reveal whether failures come from candidates, filters, or ranking.
Personalization needs a baseline without history. Compare the same queries with and without permitted preference features. Include new users and users whose current intent conflicts with past behavior. A user who usually buys sneakers can still be shopping for formal shoes today; current constraints must remain visible.
Evaluate complete tasks and intermediate decisions
| Layer | Test |
|---|---|
| Intent | Were hard constraints extracted correctly? |
| Retrieval | Did eligible relevant items enter the candidate set? |
| Personalization | Did ranking improve for the intended user slices? |
| Grounding | Are factual claims supported by catalog records? |
| Interaction | Was clarification useful and was the task completed? |
| Operations | Latency, tool errors, cost, and fallback success |
Use deterministic checks for size, price, and availability; human or audited model judgments for softer usefulness; and paired comparisons on a frozen scenario set. Save traces to explain regressions. Test failures by replacing tool outputs with empty, stale, malformed, or conflicting records. A graceful fallback should be evaluated as part of the product.
Tie the design to user outcomes
Run a controlled experiment with a stable assignment unit. Track successful shopping sessions, repeat use, or another declared objective, alongside latency and error guardrails. Engagement alone can increase because the assistant forces extra turns. Inspect whether users complete tasks with appropriate effort and whether recommendations remain useful after the conversation.
Ablate each component: a single coordinator with tools, separate search and personalization roles, and the complete system. If an extra agent adds cost without measurable benefit, simplify the architecture. Keep the existing multi-agent foundations as the conceptual reference rather than repeating framework setup here.
Check your understanding: Should personalization be allowed to rerank an out-of-stock item to the top because the user likes its brand?
No. Hard eligibility constraints should be enforced consistently outside subjective ranking. Personalization operates over the eligible set.
Sources and further reading
- ReAct: synergizing reasoning and acting in language models — Tool use as an observable interaction loop.
- Covington et al.: YouTube recommendations — Candidate generation and ranking separation.