1. Define the job before choosing a model
“Support customers” is too broad. Define which requests it can resolve, what data it needs, which actions it may execute, and what confirms successful completion. A first agent should master one narrow journey—for example, answering order questions and creating a case when it cannot resolve them—before taking on more responsibility.
2. Separate knowledge, reasoning, and actions
Knowledge should come from identifiable, maintained sources. The model interprets the request and proposes the next step. Tools query systems or perform actions. Separating these layers lets teams change a source, policy, or model without rebuilding the whole product and makes it easier to investigate a response.
3. Grant the least possible permission
Reading an order and issuing a refund carry different risks. Every tool should validate identity, parameters, and limits outside the model. Irreversible or costly actions should require explicit confirmation or human approval. The agent should never be able to expand its own permissions.
4. Design the handoff to a person
Escalation is not failure; it is part of the product. Define signals such as low confidence, out-of-scope requests, frustrated customers, contradictory data, or high financial impact. The person should receive the context, consulted sources, and actions already taken—not an opaque conversation they must reconstruct.
5. Evaluate tasks, not impressions
A convincing demo does not prove reliability. Build a set of normal, ambiguous, adversarial, and incomplete-data cases. Measure correct resolution, proper tool use, valid citations, timely escalation, latency, and cost. Every instruction, model, or source change should pass that evaluation again.
- One clearly defined job and user
- Knowledge sources with owners and dates
- Tools with minimum permissions and validation
- Confirmation and escalation rules
- Repeatable evaluations and traceability
Primary references
Turn the guide into a concrete project