Reliability for the Agentic Enterprise

Autonomous agents introduce non-deterministic execution patterns fundamentally different from traditional Salesforce automation. While Salesforce Flow and Apex follow predictable paths that produce repeatable results under controlled conditions, agents reason through problems dynamically. For example, the same question asked twice can take different execution paths, consume different resources, and produce different results. This non-determinism creates reliability challenges that traditional testing and monitoring approaches don't address.

Agent reliability failure modes are categorically different from Salesforce Flow or Apex failures. Large language model (LLM) inference services represent external dependencies with availability characteristics separate from the Salesforce platform. Agents generate responses that occasionally violate expected structure despite prompt instructions. Agent context preparation consumes governor limits unpredictably. Unbounded data retrieval affects agents more severely because agents can request open-ended context while deterministic queries have explicit scope control.

Reliable agent architecture requires a multi-tier fallback chain that degrades gracefully—from sophisticated agents, to simpler agents with reduced context, to deterministic rule engines, and finally to human handoff via Omni-Channel. Platform Events power durable retry queues with checkpoint patterns, allowing failed workflows to resume from the last successful step. Circuit breakers track LLM service health in Platform Cache for real-time status, backed by Custom Metadata Types for thresholds and configuration, routing to fallback patterns after consecutive failures. Monitoring requires tracking conversation success rates, mean time between failures (MTBF), mean time to recovery (MTTR), and grounding accuracy. Agentforce Observability provides session traces, health and latency metrics, and escalation and deflection rates, but you'll still need to instrument reliability metrics like MTBF, MTTR, and grounding accuracy yourself.

This document covers what changes when agents are in the picture. For foundational reliability patterns applying to all Salesforce solutions (service level object (SLO) design, fault tolerance, scaling, and monitoring), see the Reliability pillar.

Agentforce agents fail in different ways than Salesforce Flow or Apex. Understanding these failure modes is what shapes reliability architecture.

  • LLM service unavailability: LLM inference endpoints experience occasional latency spikes or availability issues. Agentforce is listed as a monitored product on trust.salesforce.com alongside Analytics and other Salesforce services. However, granular LLM inference latency and per-request performance metrics aren’t exposed there. Implement circuit breakers by using Platform Cache for low-latency real-time state, backed by Custom Metadata Types for thresholds and configuration, to track LLM service health. Platform Cache is best-effort–entries can be evicted before their time-to-live (TTL)—so persist the open/tripped state in a durable store, such as a Custom Object, rather than the cache alone. Tune the trip condition to the endpoint: a count of consecutive failures (for example, five) or an error rate crossing a threshold over a rolling window. When the circuit trips, open it and route to fallback patterns, such as simpler prompts, cached responses, or human handoff, until a successful test request indicates recovery.
  • Response validation failures: Agents occasionally generate responses violating expected structure despite prompt instructions. An agent sometimes returns malformed JSON, omits required fields, or includes unexpected data types. Platform Cache can store validated response schemas, enabling fast validation. When validation fails, retry with refined prompts, including the validation error and an example of correct structure. Set a maximum number of retries (3—5) to prevent infinite retry loops.
  • Hallucination detection: When grounding data is incomplete, agents produce plausible but incorrect information. Unlike deterministic queries that return "not found," agents fill gaps with inference. The Einstein Trust Layer Audit Trail logs every prompt, masked prompt, response, toxicity detection result, and user feedback for post-hoc governance review. However, it doesn’t capture step-by-step reasoning traces. For step-by-step reasoning capture, use Agentforce Session Tracing, which records reasoning-engine executions. Require agents to cite Data 360 (formerly Data Cloud) retrieval-augmented generation (RAG) retrieval sources. Cross-reference agent claims against retrieved sources to catch factual deviations before action execution. An inline check adds little latency, since it runs against sources already in context. A check that needs a separate round-trip, such as a second model call or an external service, costs more. Reserve these checks for high-impact writes, including financial or order updates, and run it asynchronously during real-time conversation.
  • Governor limit spikes: Agent context preparation consumes Salesforce object query language (SOQL) queries, CPU time, and heap memory unpredictably based on conversation flow. Proactive Monitoring is a Signature Success (formerly Signature Support) entitlement service in the Salesforce Success Plan, where Salesforce proactively monitors customer accounts. Proactive Monitoring isn’t a self-service configurable alert available to all customers. For self-service governor limit monitoring, use Scale Center to identify resource-intensive agent operations approaching limits. Implement pagination in agent actions retrieving large record collections. For frequently accessed reference data, use Platform Cache.
  • Data skew in agent context: Agents retrieving context for accounts with more than 10,000 opportunities, or cases with comparably large activity histories, encounter the same data skew challenges as Batch Apex. Unlike deterministic queries where you control the scope, agent reasoning can request "all history" unpredictably. To prevent this, design agent actions with sensible hard limits (for example, a cap on the order of 200 children per parent) and implement sampling strategies when limits are exceeded. The cap bounds how much data enters the agent's context. Pair it with selective, indexed filters so the query itself stays cheap, since an unindexed filter or aggregate on a skewed parent scans the full child set before the cap applies.

Design multi-tier fallback chains, enabling agents to continue functioning with reduced capability rather than failing completely.

  • Primary agent with full context: Sophisticated agent with complete Data 360 grounding, extensive conversation history, and complex reasoning. Highest quality but most resource-intensive and highest failure risk.
  • Secondary agent with reduced context: Simpler agent using condensed context (for example, last 30 days vs. all history, top 5 vector search results vs. top 20). Smaller context means fewer tokens, faster inference, and a lower chance of failure, but only if the reduced context still covers the data the task needs.
  • Deterministic rule engine: Traditional Salesforce Flow or Apex logic that handles common scenarios when agent reasoning fails. Rules can't adapt to new situations, but they reliably handle known patterns. The order validation agent falls back to standard validation rules, while the lead scoring agent falls back to rule-based score calculation.
  • Human handoff via Omni-Channel: Route to human queue when automated options are exhausted. Configure skill-based routing to make sure escalations reach appropriate expertise. Pass full conversation context—including attempted strategies, any confidence scores you compute for the agent, and failure reasons—to enable efficient human intervention.

Implement fallback logic in orchestration Salesforce Flow or Apex triggered by Platform Events. Each tier logs which strategy succeeded, creating visibility into failure patterns and fallback effectiveness.

All Platform Event patterns in this section assume high-volume Platform Event types, which provide the 72-hour replay retention window. Standard-volume Platform Events are retained for only 24 hours. Use high-volume event types for all agent resilience, retry queue, and checkpoint patterns where replay durability is a reliability requirement. Platform Events provide durable messaging enabling agent workflows to survive failures and recover automatically.

  • Checkpoint agent progress: Multi-step agent workflows publish checkpoint events after each successful step. Contract processing agent completes document extraction, publishes ContractParsed event with extracted data, then proceeds to clause analysis. If clause analysis fails, replay the ContractParsed event to resume from the extracted data without re-parsing. This works when the checkpoint payload carries that data and the subscriber is idempotent, because replay redelivers events rather than resuming a workflow mid-step. Store checkpoint state in event payload including correlation ID, completed steps, and state needed for continuation.
  • Retry queue with exponential backoff: Failed agent requests publish to AgentRetryRequested event with retry count in payload. Event subscriber processes retries with exponential backoff delays (for example, 30 sec, 2 min, 8 min, 30 min). After maximum retries (typically 5), route to dead-letter queue and alert operations. High-volume Platform Events preserve a 72-hour replay window. Use high-volume event type for agent retry queues so subscribers can catch up after org maintenance windows or deployment downtime. Standard-volume Platform Events retain for only 24 hours.
  • Event-driven orchestration: Design agent workflows as loosely coupled event chains. Lead creation publishes LeadCreated. Lead scoring agent subscribes, scores, and publishes LeadScored. Lead routing agent subscribes to LeadScored and assigns to the appropriate queue. Each agent can fail and retry independently. A single agent failure doesn't block the entire workflow—downstream agents process when dependencies recover.
  • Correlation ID tracking: Include a correlation ID as a custom field in all related event payloads. Replay events by using the Pub/Sub API by replayId, an opaque, non-contiguous identifier, to reconstruct workflow history within the 72-hour retention window. To enable correlation ID-based debugging, implement a subscriber-side pattern that logs incoming events with their replayId and correlation ID to a custom object, enabling cross-event traceability. This logging pattern is a custom implementation you build yourself, not a native platform query capability.

Move agent operations that exceed synchronous limits to an asynchronous context, which allows higher governor limits and longer timeouts.

  • Batch Apex for bulk agent operations: An agent analyzing 1,000+ records benefits from Batch Apex providing 200 SOQL queries, 150 DML statements (up to 10,000 DML rows), and 60,000 milliseconds (60 seconds) CPU time per execute method. Lead scoring agent processing nightly lead imports. Case sentiment analysis across historical cases. Opportunity forecasting analyzing complete pipelines.
  • Queueable chains for multi-step workflows: Complex agent workflows requiring multiple external API calls, large dataset analysis, or extended processing time use Queueable Apex chains. Each Queueable job gets 60 CPU time and 12 MB heap. Chain to next Queueable for workflows exceeding single-job limits. Track progress in Custom Objects so a failed job can resume from the last completed step.
  • @future for simple async handoffs: Lightweight agent operations, like sending notifications, logging to external systems, or non-urgent updates, use @future methods. The fire-and-forget pattern fits when a workflow doesn't need results back and can tolerate failure.

Monitor async processing capacity via the Apex Jobs page (Setup → Apex Jobs) and the Apex Flex Queue. A maximum of 5 concurrent batch jobs limits parallel agent processing; additional jobs queue in the Apex Flex Queue, which holds up to 100 jobs in Holding status. All async Apex types, Batch, Future, Queueable, and Scheduled Apex, share the same org-wide daily limit (DailyAsyncApexExecutions) on asynchronous Apex executions: the greater of 250,000 or 200 times the number of user licenses. When you run high-frequency agent polling or retry subscriber queues, Queueables can rapidly exhaust this quota. To avoid this, batch or throttle these enqueues to stay within the limit—which applies org-wide across all async Apex types, not as a separate allocation for Queueable alone. Alert when consumption approaches these limits so teams can manage capacity proactively.

Validate agent resilience before production through testing strategies that address non-deterministic behavior.

  • Load testing in Full Copy Sandbox: Test agent performance under production-scale load by using full data volumes. Simulate concurrent conversations matching peak demand. Measure p95 and p99 latency, identifying performance degradation under stress. Validate governor limit consumption stays below thresholds at peak load. Load testing in a Partial Copy sandbox with reduced data provides misleading confidence, since data volume directly affects agent performance. Before running these tests, get approval from Salesforce (at least a week prior for performance test requests). Unapproved testing can be throttled or blocked.
  • Fault injection: Deliberately introduce failures to validate recovery mechanisms. Simulate LLM inference failures in the service layer your circuit breaker monitors, to verify the breaker activates. Inject SOQL exceptions to test error handling. Corrupt agent responses to test validation logic. Introduce data skew (accounts with 10,000+ children) to test pagination logic. Set aggressive timeouts to test timeout handling and retry strategies.
  • Chaos engineering: Randomly disable Platform Event subscribers to test replay recovery. Terminate async jobs mid-execution to test checkpoint restart. Introduce variable latency to external systems to test progressive timeout strategies. Chaos experiments validate resilience in realistic failure combinations. Schedule regular chaos days in staging environments, maintaining resilience as agents evolve.
  • Long-running stability tests: Execute agent workloads continuously for 72+ hours, monitoring for time-dependent failures. Accumulating errors and gradually degrading performance surface only under sustained operation. Track CPU time trends to identify gradual degradation over the run.

Define service level indicators (SLIs) enabling objective reliability measurement. Establish SLOs before building agents, not after production failures.

  • Conversation success rate: Percentage of agent conversations completed without errors or timeouts. Count a graceful handoff to a human as a success, not a failure—escalation is the final fallback tier by design. Count failed or unintended escalations against the metric—it's a useful reliability signal, but an imperfect proxy for user experience, since a polite sign-off can mask an unresolved issue. Track separately by agent type and use case since lead scoring success criteria differ from contract review. Alert when the success rate drops below SLO (typically 95—99%).
  • Mean time between failures (MTBF): Average operational time between agent failures. Higher MTBF indicates fewer failures and better reliability. Calculate as total agent operational hours divided by failure count. Track per agent type to identify least reliable agents that require optimization investment.
  • Mean time to recovery (MTTR): Average time from failure detection to service restoration. Automated recovery via Platform Event replay and circuit breakers restores service much faster and with less manual error than hands-on intervention. Lower MTTR reduces business impact per incident.
  • Grounding accuracy: Percentage of agent responses factually correct based on source data validation. Sample random agent responses to validate claims against Data 360 sources. Grounding accuracy below 95% indicates hallucination issues requiring prompt refinement or better context retrieval.
  • Platform Event replay gaps: Count of events missed or undelivered, for example a subscriber offline beyond the retention window. Zero gaps is a positive signal of a healthy event-driven architecture. Gaps indicate subscriber issues, event publishing failures, or capacity constraints. Track these gaps with the subscriber-side logging pattern described above, and record each event's replayId and correlation ID to a custom object.

Reliable agents come from designing for failure up front—defining SLIs/SLOs, governor-limit-aware context handling, and multi-tier fallback chains before building, not bolting them on afterward. Layer circuit breakers and Platform Event-based retry and checkpoint logic so workflows recover automatically, with human handoff as the last resort. Validate that design through systematic load, fault-injection, and chaos testing, then keep monitoring the same SLIs in production to confirm it holds up over time.

The reliability guidance in the main Reliability pillar applies to agents with platform-specific considerations covered here. Successful agent architectures balance autonomous capability with appropriate oversight, enabling self-recovery from transient failures while escalating when confidence drops or edge cases emerge.

Share your feedback on the Well-Architected Framework.