Operational Excellence - Agentic Patterns

Learn more about Well-Architected Operational Excellence → Agentic Enterprise Operational Excellence

Patterns

Where to look	What good looks like
Agentforce \| Session Tracing	✅ Use the native Session Tracing Data Model (STDM) capturing agent interactions in a unified data model on Data 360 rather than building a parallel telemetry system — recording each conversation as a trace hierarchy of sessions, interactions (turns), and steps (UserInputStep, LLMExecutionStep, FunctionStep)
Agentforce \| Session Tracing	✅ Rely on the standard STDM objects (AIAgentSession, AIAgentSessionParticipant, AIAgentInteraction, AIAgentInteractionStep, AIAgentInteractionMessage), which capture actions natively as interaction steps rather than requiring a separate action log
Agentforce \| Session Tracing	✅ Retrieve token usage, model, and prompt details by joining interaction steps to their LLM-call records (via GenerationId and GenAiGatewayRequest/Response references) rather than storing them as custom session fields
Agentforce \| Session Tracing	✅ Capture reasoning traces from the Atlas Reasoning Engine — Session Tracing records the planner’s steps as part of each interaction — to reveal agent decision processes and improve debuggability
Agentforce \| Agent Configuration	✅ Log turn-level details including user input, agent response, reasoning explanation, confidence scores, retrieved context, actions attempted, and errors encountered
Einstein \| Trust Layer	✅ Turn on the Einstein Trust Layer generative AI audit trail (prompt and response text, PII-masked versions, grounding data, safety and toxicity scores, model details) streamed to Data 360, plus Einstein Generative AI and Feedback Data Collection and Storage for long-term retention
Platform \| Custom Objects	✅ Reserve custom logging objects for deliberate exceptions — such as a lightweight object correlating agent behavior with business outcomes — rather than making them the primary logging substrate
Platform \| Dashboard	✅ Correlate agent behavior with business outcomes, identify improvement opportunities through pattern analysis, and feed continuous learning pipelines with quality-labeled examples

Anti-Patterns

Where to look	What bad looks like
Platform \| Custom Objects	⚠️ Hand-building custom transactional objects as the primary conversation-logging substrate instead of the native Session Tracing Data Model, running into scale problems storing multi-turn conversations, large JSON payloads, and rapid data growth
Agentforce \| Session Tracing	⚠️ Building a parallel telemetry system when Agentforce Session Tracing already captures interactions in a unified data model on Data 360
Agentforce \| Agent Configuration	⚠️ Logging only agent responses without capturing inputs, reasoning traces, retrieved context, and actions taken
Einstein \| Trust Layer	⚠️ Disabling the Einstein Trust Layer audit trail in production or not turning on Feedback Data Collection and Storage for long-term retention in Data 360
Platform \| Monitoring	⚠️ Relying solely on conversation success/failure without capturing token usage, model, and prompt details through the STDM LLM-call joins

Patterns

Where to look	What good looks like
Platform \| Monitoring	✅ Measure response latency at percentiles (p50, p95, p99) revealing typical performance and worst-case experiences
Platform \| Monitoring	✅ Instrument each latency phase separately (model inference, context retrieval, action execution, network) enabling targeted optimization
Platform \| Monitoring	✅ Alert when p95 latency exceeds thresholds indicating performance degradation
Platform \| Monitoring	✅ Track token distribution (prompt tokens, completion tokens, total tokens) per conversation, per agent, and per use case
Platform \| Monitoring	✅ Detect outlier conversations consuming excessive tokens suggesting prompt issues or misuse
Platform \| Monitoring	✅ Compare current throughput against known rate limits or capacity baselines alerting when approaching thresholds
Platform \| Monitoring	✅ Calculate error rates as percentage of total requests enabling comparison across agents and time periods
Platform \| Monitoring	✅ Distinguish between error categories (model errors, permission errors, integration errors) requiring different responses
Platform \| Monitoring	✅ Calculate availability as percentage of time agents successfully respond to requests and track against SLOs
Platform \| Proactive Monitoring	✅ Configure Proactive Monitoring thresholds based on baseline agent behavior patterns rather than accepting defaults
Platform \| Scale Center	✅ Review Scale Center data after deploying new agents or making significant prompt changes validating resource consumption matches expectations

Anti-Patterns

Where to look	What bad looks like
Platform \| Monitoring	⚠️ Measuring only average latency instead of percentiles, masking worst-case user experiences
Platform \| Monitoring	⚠️ Tracking total latency without instrumenting individual phases (inference, retrieval, action execution)
Platform \| Monitoring	⚠️ Treating all errors the same rather than categorizing by type (model, permission, integration) for appropriate response
Platform \| Proactive Monitoring	⚠️ Accepting default Proactive Monitoring thresholds without tuning to baseline agent behavior patterns
Platform \| Scale Center	⚠️ Deploying new agents or making prompt changes without reviewing Scale Center data to validate resource consumption
Platform \| Monitoring	⚠️ Ignoring token consumption patterns and failing to detect outlier conversations consuming 10x typical tokens

Patterns

Where to look	What good looks like
Agentforce \| Testing Center	✅ Start with native testing and evaluation tooling: maintain reusable test-case sets in Agentforce Testing Center and run agents against them in batch, checking whether the agent selected the expected topic and action and produced an acceptable response
Agentforce \| Testing Center	✅ Use native evaluators — out-of-the-box quality dimensions and the LLM-as-judge approach for scoring conversational quality — and author custom evaluators for organization-specific criteria, using representative conversations so evaluation reflects real usage
Agentforce \| Testing Center	✅ For factual questions with known answers, maintain known-answer assertions as repeatable test cases in Testing Center; for open-ended questions, add human review of sampled conversations
Data 360 \| Retrieval Quality	✅ For agents grounded in a knowledge base, use native retrieval-quality analysis to assess context relevance, answer relevance, and faithfulness (groundedness) of responses
Einstein \| Trust Layer	✅ Rely on the Einstein Trust Layer runtime toxicity detection as the first line of guardrail defense, then improve safety monitoring through content filtering, manual review sampling, user reporting, and fairness audits across demographic groups
Data 360 \| GenAIFeedback	✅ Capture user satisfaction through Agentforce feedback flowing into the GenAIFeedback / GenAIFeedbackDetail DMOs in Data 360 (thumbs-up/down, accept/reject, edits, verbatim comments) alongside gateway request and generation data
Platform \| Analytics	✅ Combine explicit feedback (asked before escalation, after conversation, follow-up surveys) with implicit signals (completion rates, escalation, abandonment, retries), correlating with conversation characteristics to guide improvement prioritization

Anti-Patterns

Where to look	What bad looks like
Agentforce \| Testing Center	⚠️ Building and running a custom test harness instead of starting with Agentforce Testing Center’s reusable test sets and native evaluators
Agentforce \| Testing Center	⚠️ Relying solely on automated evaluation without human review for quality assessment of open-ended responses
Platform \| Monitoring	⚠️ Monitoring only accuracy without tracking relevance, safety, and user satisfaction dimensions
Data 360 \| GenAIFeedback	⚠️ Failing to collect any user feedback (explicit or implicit) on agent response quality when Agentforce feedback already flows into the GenAIFeedback DMOs
Agentforce \| Agent Configuration	⚠️ Skipping fairness audits assessing outcome distributions across demographic groups

Patterns

Where to look	What good looks like
Agentforce \| Agent Configuration	✅ Establish baseline performance immediately after agent deployment measuring accuracy, relevance, safety, and satisfaction across representative test cases
Agentforce \| Agent Configuration	✅ Periodically re-evaluate against the same test cases revealing performance changes independent of usage pattern shifts
Platform \| Monitoring	✅ Alert when quality metrics decline beyond acceptable thresholds defined by business impact tolerance
Platform \| Analytics	✅ Monitor input characteristics (question length, topic distribution, language complexity, domain terminology) detecting significant distribution shifts
Platform \| Monitoring	✅ Monitor response characteristics (average response length, action invocation rates, escalation frequencies, error patterns) establishing behavioral baselines
Platform \| Monitoring	✅ Configure automated alerts firing when drift metrics exceed thresholds triggering investigation and remediation

Anti-Patterns

Where to look	What bad looks like
Agentforce \| Agent Configuration	⚠️ Failing to establish baseline performance after initial agent deployment, making drift undetectable
Platform \| Monitoring	⚠️ Ignoring distribution drift when user input patterns change significantly from what agents were designed to handle
Platform \| Monitoring	⚠️ Not monitoring behavioral changes (response length, action rates, escalation frequencies) against established baselines
Platform \| Monitoring	⚠️ Having no automated alerting for drift metrics, relying only on manual observation to detect degradation

Patterns

Where to look	What good looks like
Platform \| Monitoring	✅ Monitor token consumption distributions detecting outlier conversations and investigate those consuming 10x typical tokens
Platform \| Monitoring	✅ Detect sudden increases in failure rates, new error types not in baseline, or localized error concentrations affecting specific users or agents
Platform \| Analytics	✅ Track conversation abandonment by conversation stage identifying whether users abandon during initial interaction, mid-conversation, or after task completion
Platform \| Monitoring	✅ Route critical agent failures to on-call engineers immediately, quality degradation to product teams during business hours, and minor anomalies to daily digests

Anti-Patterns

Where to look	What bad looks like
Platform \| Monitoring	⚠️ Treating all alerts with equal urgency, causing alert fatigue and missed critical failures
Platform \| Monitoring	⚠️ Ignoring cost anomalies like token consumption spikes that may indicate prompt issues, abuse, or infrastructure problems
Platform \| Analytics	⚠️ Not tracking conversation abandonment rates or failing to analyze abandonment by stage
Platform \| Monitoring	⚠️ Waiting for user reports rather than proactively detecting error rate spikes through automated monitoring

Patterns

Where to look	What good looks like
Platform \| Source Control	✅ Store prompt templates in Git alongside application code treating prompts as critical application logic deserving code review, testing, and controlled deployment
Einstein \| Prompt Builder	✅ Use Prompt Builder for iterative prompt development with version history and testing capabilities, then export to version control for deployment pipelines
Platform \| Source Control	✅ Store configurations (model selection, temperature, token limits, retrieval configs, action permissions, feature flags) as code enabling environment-specific values and drift detection
Platform \| Source Control	✅ Apply semantic versioning to agent components: major (breaking changes), minor (features), patch (bug fixes) with version numbers appearing in logs
Platform \| Documentation	✅ Document what changed, why it changed, what testing validated the change, and what rollback procedures exist for every prompt and configuration change

Anti-Patterns

Where to look	What bad looks like
Einstein \| Prompt Builder	⚠️ Making prompt changes without code review, testing, or controlled deployment processes
Platform \| Configuration	⚠️ Storing agent configurations outside of source control where drift cannot be detected and deployments cannot be automated
Platform \| Documentation	⚠️ Making prompt changes without documenting rationale, test results, or rollback procedures
Platform \| Source Control	⚠️ Failing to apply semantic versioning, making it impossible to correlate incidents with specific deployments

Patterns

Where to look	What good looks like
Platform \| Documentation	✅ Maintain a model registry recording which model versions are deployed where including model family, specific version, fine-tuning state, environment, deployment date, and responsible team
Platform \| Documentation	✅ Log every agent deployment including version, environment, timestamp, deploying user, approvals obtained, and deployment outcome
Platform \| Documentation	✅ Map dependencies between agents, prompts, knowledge bases, actions, and integrations explicitly rather than discovering them during incidents
Business \| Process	✅ Establish a deprecation schedule with a policy-backed minimum notice period communicated sufficiently in advance, enabling planned migrations rather than emergency responses to forced deprecations
Platform \| Documentation	✅ Document rollback procedures for each agent including target version, execution steps, validation testing, and notification requirements
Platform \| Staging	✅ Test rollback procedures in non-production environments validating they work before depending on them during production incidents

Anti-Patterns

Where to look	What bad looks like
Platform \| Documentation	⚠️ Discovering dependencies between agents, prompts, and knowledge bases only during incidents rather than maintaining explicit dependency maps
Business \| Process	⚠️ Providing insufficient deprecation notice forcing emergency migrations rather than planned transitions
Platform \| Documentation	⚠️ Having untested rollback procedures that fail when needed most during production incidents
Platform \| Deployment	⚠️ Deploying agents without logging version, environment, approvals, or outcome, losing ability to correlate incidents with deployments

Patterns

Where to look	What good looks like
Platform \| UI	✅ Collect explicit feedback (thumbs up/down, quality scores, comments) and implicit feedback (completion rates, escalation frequencies, retry patterns) systematically
Data 360 \| Data Curation	✅ Curate conversation logs into training datasets by filtering for quality, removing problematic examples, ensuring representativeness, and deduplicating
Data 360 \| Grounding	✅ Adapt agents to domain vocabulary, business processes, and organization-specific knowledge primarily through RAG grounded in Data 360 and prompt tuning — the Salesforce-native adaptation path
Einstein \| BYOLLM	✅ When grounding and prompt tuning are insufficient, fine-tune a model externally and connect it through BYOLLM rather than expecting in-platform generative fine-tuning
Einstein \| Prompt Builder	✅ Establish regular prompt review cadence examining recent conversations, quality metrics, and user feedback identifying improvement opportunities
Einstein \| Prompt Builder	✅ Implement prompt changes incrementally with A/B testing validating improvements before full rollout
Agentforce \| Agent Configuration	✅ Maintain test sets covering happy paths, edge cases, adversarial inputs, and historical problem scenarios, expanding continuously as new edge cases are discovered

Anti-Patterns

Where to look	What bad looks like
Platform \| Data	⚠️ Attempting improvement without systematic feedback collection, making improvement guesswork rather than data-driven
Data 360 \| Data Curation	⚠️ Using uncurated conversation logs for training including safety violations, errors, and non-representative examples
Data 360 \| Grounding	⚠️ Reaching for external fine-tuning before exhausting RAG grounding and prompt tuning, the native path that resolves most domain-adaptation needs without a custom model
Einstein \| Prompt Builder	⚠️ Treating prompt engineering as a one-time exercise rather than continuous practice adjusted as usage patterns evolve
Agentforce \| Agent Configuration	⚠️ Having static test sets that never expand to cover new edge cases discovered in production

Patterns

Where to look	What good looks like
Platform \| Custom Metadata Types	✅ Implement feature flags using custom metadata types or custom settings to decouple deployment from release for high-risk agent changes
Platform \| Permission Sets	✅ Implement canary deployments releasing agent changes to a small user subset first using permission sets controlling agent access
Agentforce \| Agent Configuration	✅ Monitor the canary cohort intensively measuring success rates, quality, error rates, latency, token consumption, and satisfaction, and only expand rollout when the canary matches or improves on the control group running the previous agent version
Platform \| Analytics	✅ Define success criteria before starting A/B tests preventing ambiguous results, aligned with business objectives
Agentforce \| Agent Configuration	✅ Deploy champion-challenger testing maintaining proven version as fallback while comparing against improved challenger version
Business \| Process	✅ Establish objective promotion criteria including minimum sample size, quality thresholds, and evaluation duration before promotion decisions

Anti-Patterns

Where to look	What bad looks like
Agentforce \| Agent Configuration	⚠️ Rolling out agent changes to all users simultaneously without canary testing, maximizing blast radius of issues
Platform \| Deployment	⚠️ Deploying high-risk agent changes without feature flags enabling instant rollback without redeployment
Platform \| Analytics	⚠️ Running A/B tests without pre-defined success criteria, leading to ambiguous results and poor decisions
Platform \| Analytics	⚠️ Running experiments with insufficient sample size producing noisy results that lead to incorrect conclusions
Agentforce \| Agent Configuration	⚠️ Using feature flags for every minor prompt change rather than reserving them for high-risk changes to business-critical agents

Patterns

Where to look	What good looks like
Platform \| Monitoring	✅ Detect agent failures through automated monitoring alerting on error rates, quality metric degradation, cost anomalies, latency increases, and user escalations
Platform \| Monitoring	✅ Supplement automated detection with user reporting mechanisms enabling escalation of issues automated monitoring misses
Business \| Process	✅ Define severity criteria objectively based on user impact, blast radius, data integrity risk, and recovery urgency
Platform \| Documentation	✅ Document initial response procedures as runbooks executable by on-call engineers without deep agent expertise
Agentforce \| Agent Configuration	✅ Include in initial response: disable failing agent (circuit breaker), route to fallback, collect diagnostics, notify stakeholders, initiate incident communication

Anti-Patterns

Where to look	What bad looks like
Platform \| Monitoring	⚠️ Relying on user reports as primary detection mechanism rather than automated monitoring for agent failures
Business \| Process	⚠️ Using subjective severity classification that leads to under-escalating real problems or over-escalating minor issues
Platform \| Documentation	⚠️ Having no runbooks, requiring incident-specific investigation by engineers with deep agent expertise during every incident
Business \| Process	⚠️ Treating agent failures identically to traditional application failures without accounting for hallucinations, off-topic responses, or safety violations

Patterns

Where to look	What good looks like
Platform \| Apex	✅ Implement circuit breakers with configurable error threshold (e.g., 50% error rate), duration window (e.g., 5 min rolling window), and recovery testing (periodic trial requests)
Agentforce \| Agent Configuration	✅ Design fallback hierarchies: Primary sophisticated agent → Simpler backup agent → Static FAQ responses → Human handoff
Platform \| Orchestration	✅ Implement fallback logic in orchestration layer rather than within individual agents enabling consistent behavior across agent ecosystem
Agentforce \| Agent Configuration	✅ Design human handoff workflows preserving conversation context, communicating handoff reason, and routing to appropriate human agent with relevant expertise
Platform \| Monitoring	✅ Monitor handoff rates by topic, agent, and failure reason to identify agent capability gaps
Agentforce \| Agent Configuration	✅ When sophisticated reasoning fails, fallback to simpler logic; when real-time data unavailable, operate on cached data; when integrations fail, operate in read-only mode
Platform \| Orchestration	✅ Test fallback paths regularly validating they work when needed

Anti-Patterns

Where to look	What bad looks like
Agentforce \| Agent Configuration	⚠️ Having no circuit breakers, allowing failing agents to repeatedly impact users while issues are resolved
Platform \| Orchestration	⚠️ Implementing fallback logic within individual agents rather than at the orchestration layer, creating inconsistent behavior
Agentforce \| Agent Configuration	⚠️ Designing binary pass/fail agents that either work perfectly or completely fail without graceful degradation options
Platform \| Orchestration	⚠️ Never testing fallback paths, discovering they don’t work only during actual incidents
Agentforce \| Agent Configuration	⚠️ Human handoff that loses conversation context forcing users to repeat information

Patterns

Where to look	What good looks like
Business \| Process	✅ Conduct blameless post-incident reviews focusing on system improvements rather than individual fault creating psychological safety
Business \| Process	✅ Create detailed timeline from initial conditions through detection, investigation, response, recovery, and validation with timestamps
Business \| Process	✅ Use “Five Whys” technique drilling from symptoms to fundamental causes rather than simplistic conclusions
Business \| Process	✅ Analyze contributing factors including monitoring gaps, testing gaps, documentation gaps, automation gaps, and architectural gaps
Business \| Process	✅ Evaluate detection: what signal should have detected problem earlier, what monitoring enables faster detection
Business \| Process	✅ Specify concrete, assigned, timebound action items preventing recurrence (e.g., “add alert for error rate exceeding 5% over 10min - assigned: Alex - due: 2 weeks”)
Business \| Process	✅ Track action item completion in subsequent reviews ensuring continuous improvement actually happens
Platform \| Documentation	✅ Document post-incident reviews in shared knowledge base with tagging enabling pattern analysis across multiple incidents
Business \| Process	✅ Share post-incident reviews broadly beyond immediate incident responders through team meetings or lunch-and-learn sessions

Anti-Patterns

Where to look	What bad looks like
Business \| Process	⚠️ Blame-focused postmortems that reduce psychological safety and prevent honest assessment of failures
Business \| Process	⚠️ Simplistic root cause conclusions (“the agent had a bug”) rather than specific findings enabling effective remediation
Business \| Process	⚠️ Vague action items like “improve monitoring” instead of specific, assigned, timebound tasks
Business \| Process	⚠️ Low action item completion rates indicating continuous improvement process failure
Business \| Process	⚠️ Keeping post-incident learnings siloed to immediate responders rather than sharing broadly for organizational learning

Patterns

Where to look	What good looks like
Business \| Process	✅ Validate pre-deployment checklist including testing, security review, compliance validation, documentation, monitoring configuration, and stakeholder approval
Business \| Process	✅ Evaluate risk based on blast radius, data sensitivity, action authority, integration dependencies, recovery complexity, and compliance implications
Business \| Process	✅ Scale approval requirements based on risk: high-risk requires executive approval, security review, and gradual rollout; low-risk requires peer review and standard testing
Platform \| Approval Processes	✅ Implement automated, tracked, and auditable approval workflows using Salesforce approval processes or external deployment tools
Business \| Process	✅ Define emergency bypass procedures requiring executive approval, rationale documentation, and expedited post-deployment review
Platform \| Monitoring	✅ Track emergency deployments separately; high emergency deployment frequency indicates process problems requiring investigation

Anti-Patterns

Where to look	What bad looks like
Business \| Process	⚠️ Applying uniform approval depth regardless of agent risk profile, wasting effort on low-risk agents or under-reviewing high-risk ones
Business \| Process	⚠️ Deploying production agents without pre-deployment checklist validation of testing, security, compliance, and monitoring
Platform \| Deployment	⚠️ Having no emergency bypass procedures, causing critical fix delays during production incidents
Business \| Process	⚠️ Frequently requiring emergency deployments, indicating broken standard processes rather than exceptional circumstances

Patterns

Where to look	What good looks like
Agentforce \| Agent Configuration	✅ Test output characteristics (contains required information, maintains appropriate tone, includes citations) rather than exact text for non-deterministic agent outputs
Agentforce \| Agent Configuration	✅ Establish minimum quality thresholds (e.g., “95% accuracy on held-out test set, zero safety violations, fairness audit shows no demographic disparities exceeding 5%“)
Agentforce \| Agent Configuration	✅ Perform safety testing with adversarial inputs including prompt injection, jailbreak attempts, request escalation, and out-of-scope requests
Agentforce \| Agent Configuration	✅ Have safety testing performed by team members not involved in agent development providing fresh perspective on vulnerabilities
Platform \| Performance Testing	✅ Test under realistic load including concurrent conversations, typical conversation lengths, and projected usage volume
Platform \| Security	✅ Validate that agents cannot access unauthorized data, cannot perform unauthorized actions, and that prompt injection cannot escalate privileges
Platform \| CI/CD	✅ Integrate security scanning in deployment pipeline using Salesforce Code Analyzer and specialized AI security tools

Anti-Patterns

Where to look	What bad looks like
Agentforce \| Agent Configuration	⚠️ Testing agents for exact text match on non-deterministic outputs rather than testing output characteristics
Platform \| Performance Testing	⚠️ Only testing with single-user scenarios, missing governor limit issues, resource constraints, and scalability bottlenecks
Agentforce \| Agent Configuration	⚠️ Skipping safety testing or having developers test their own agents without fresh perspective on vulnerabilities
Platform \| Security	⚠️ Not testing whether prompt injection can escalate agent privileges or access unauthorized data
Agentforce \| Agent Configuration	⚠️ Having static test sets that don’t represent real usage diversity including different user types and conversation complexities

Patterns

Where to look	What good looks like
Platform \| Documentation	✅ Document every change: what changed, why, what testing validated it, and what rollback procedure exists
Business \| Communication	✅ Notify all stakeholders (users, ops teams, support, business) describing what is changing, when, what benefits, what risks, and who to contact
Business \| Communication	✅ Establish communication templates for different change types standardizing messages and reducing preparation overhead
Platform \| Event Monitoring	✅ Time agent deployments during low-usage periods informed by Event Monitoring data showing actual usage patterns
Business \| Process	✅ Deploy major agent changes early to midweek avoiding weekends with reduced incident response capability
Business \| Process	✅ Establish change freeze periods before major business events (fiscal year end, product launches, major marketing campaigns)
Business \| Process	✅ Document change freeze calendar annually communicating restricted periods sufficiently in advance

Anti-Patterns

Where to look	What bad looks like
Platform \| Deployment	⚠️ Deploying during peak usage, month-end close, quarter-end close, or major business events
Platform \| Deployment	⚠️ Deploying on Mondays (ramping up) or Fridays (reduced staffing) when incident response capability is limited
Business \| Process	⚠️ Having excessive change freezes that slow improvement velocity without proportionate risk reduction
Business \| Communication	⚠️ Deploying agent changes without notifying affected users, operations teams, support teams, or business stakeholders
Platform \| Documentation	⚠️ Making changes without documenting rationale, creating organizations that rediscover lessons rather than building on experience

Patterns

Where to look	What good looks like
Platform \| UI	✅ Implement feedback collection within conversation UI enabling immediate response while interaction is fresh rather than delayed collection
Platform \| Analytics	✅ Query conversation logs analyzing question types, conversation lengths, success rates by topic, and user satisfaction by conversation characteristics
Platform \| Analytics	✅ Analyze error logs grouping by error type, affected users, conversation patterns, and temporal distribution to reveal systematic issues
Platform \| Analytics	✅ Prioritize error remediation by frequency and impact: high-frequency errors affecting many users warrant immediate attention
Platform \| Monitoring	✅ Establish quality baselines after initial deployment and monitor ongoing quality against baselines alerting when trends decline

Anti-Patterns

Where to look	What bad looks like
Platform \| UI	⚠️ Delaying feedback collection (e.g., follow-up emails days later) reducing response rates and introducing recall bias
Platform \| Analytics	⚠️ Not analyzing usage patterns to determine whether agents justify operational complexity or whether capabilities are underutilized
Platform \| Analytics	⚠️ Treating all errors equally rather than prioritizing by frequency and business impact
Platform \| Monitoring	⚠️ Making improvement decisions without data, lacking quality baselines and trend monitoring

Patterns

Where to look	What good looks like
Business \| Process	✅ Provide consistent experiment structure: hypothesis, experimental design, metrics, sample size, and success criteria
Business \| Process	✅ Document experiments before execution preventing ambiguous interpretation of results
Platform \| Analytics	✅ Track variant assignment in conversation metadata enabling outcome correlation by variant
Platform \| Analytics	✅ Calculate required sample size before experiments based on expected effect size, statistical power, and acceptable error rates
Platform \| Analytics	✅ Use multi-armed bandit optimization to dynamically allocate more traffic to better-performing variants reducing opportunity cost
Platform \| Documentation	✅ Maintain experiment catalog with variants tested, outcomes measured, results achieved, and decisions made to prevent retesting failed approaches

Anti-Patterns

Where to look	What bad looks like
Business \| Process	⚠️ Running experiments without pre-defined hypotheses and success criteria leading to ambiguous results
Platform \| Analytics	⚠️ Using insufficient sample sizes producing noisy results that lead to poor improvement decisions
Platform \| Documentation	⚠️ Having no experiment catalog, causing teams to retest previously failed approaches and lose organizational knowledge
Platform \| Analytics	⚠️ Not tracking which users received which variants, making outcome correlation impossible