Operational Excellence - DevOps Patterns
Learn more about Well-Architected Operational Excellence → DevOps Practices
Patterns
| Where to look | What good looks like |
|---|---|
| Version Control Setup | ✅ All metadata stored in Git source format using Salesforce DX project structure with .forceignore configured to exclude irrelevant files; all team members clone from central repository |
| Scratch Org Usage | ✅ Developers create ephemeral scratch orgs from source for feature work, commit changes back to version control, and discard scratch orgs after feature completion |
| Commit Messages | ✅ Every commit includes clear, concise message explaining what changed and why; messages follow team convention (e.g., “feat:”, “fix:”, “docs:“) |
| Change Tracking | ✅ Every metadata modification captured in version control with author identity, timestamp, and diff showing exact changes |
| Code Review Process | ✅ All production-bound changes flow through pull requests requiring review approval before merge; declarative configuration receives same scrutiny as code |
| Rollback Procedures | ✅ Previous versions recoverable from Git history through checkout or revert operations; team has documented rollback procedures tested in non-production |
| Audit Trail | ✅ Complete change history preserved in version control supporting compliance requirements and security investigations |
| Branching Strategy | ✅ Consistent Git branching model (GitFlow, GitHub Flow, or trunk-based) appropriate to team size and release frequency; documented and enforced |
| Migration from Org-Based | ✅ Transition to source-driven development completed early in solution lifecycle before habits solidify around org-based development |
| .gitignore Configuration | ✅ Repository excludes generated files, user-specific settings, and credentials; includes only source files teams intend to version |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Org as Source of Truth | ⚠️ Development happens directly in shared sandboxes with metadata exported to version control as afterthought; orgs and version control diverge creating deployment failures |
| Inadequate .gitignore | ⚠️ Repository includes generated files, user-specific settings, large binaries, or credentials; bloats repository and risks security exposure |
| Vague Commit Messages | ⚠️ Commits like “fix bug” or “updates” provide no context for what changed or why; hinders audit investigations and future maintenance |
| Large Infrequent Commits | ⚠️ Developers commit large changesets weekly rather than small incremental commits multiple times daily; difficult to review, hard to isolate failures |
| Commit Directly to Main | ⚠️ Developers push changes directly to main/production branch bypassing review and validation; eliminates quality gates |
| No Branching Strategy | ⚠️ Team lacks consistent branching model; developers create arbitrary branch names and merge patterns; creates confusion and merge conflicts |
| Binary Files in Git | ⚠️ Large binary files (PDFs, videos, archives) committed to Git bloating repository size and slowing clone operations |
| Committing Credentials | ⚠️ API keys, passwords, integration credentials, or security tokens committed to version control; creates security vulnerability even after removal from later commits |
| Ignoring Merge Conflicts | ⚠️ Merge conflicts resolved by accepting one side without analysis; loses changes or introduces bugs |
| No Commit Verification | ⚠️ Team never verifies that committed metadata actually represents intended changes; metadata format errors or unintended changes slip through |
Patterns
| Where to look | What good looks like |
|---|---|
| Developer Sandboxes | ✅ Individual developers use Developer sandboxes for feature work, unit testing, and rapid iteration; each developer has dedicated environment |
| Developer Pro Sandboxes | ✅ Teams developing managed packages or requiring higher storage limits use Developer Pro sandboxes for collaborative work |
| Partial Copy Sandboxes | ✅ Integration testing uses Partial Copy sandboxes with representative data samples (10,000 records per object) configured via data templates |
| Full Copy Sandboxes | ✅ Performance testing, production-like staging, and data migration validation use Full Copy sandboxes containing complete production data and configuration |
| Data Sensitivity Controls | ✅ Full Copy sandbox access restricted to roles requiring production data; Shield Platform Encryption data masking applied for non-production use when appropriate |
| Integration Endpoint Configuration | ✅ Each sandbox tier connects to corresponding test endpoints for external integrations; production integrations never connect from sandboxes |
| Environment-Specific Config | ✅ Integration architecture uses custom metadata types or custom settings to externalize endpoint URLs, credentials, and feature flags per environment |
| Post-Refresh Automation | ✅ Sandbox refreshes trigger automated Salesforce CLI scripts restoring sandbox-specific settings including integration credentials, test data seeding, feature flag states, and email deliverability |
| Refresh Scheduling | ✅ Sandbox refresh schedule documented and communicated; refreshes planned during low-activity periods to minimize team disruption |
| Data Templates | ✅ Partial Copy sandboxes use well-designed data templates selecting representative data across objects maintaining referential integrity |
| Naming Conventions | ✅ Sandboxes follow clear naming convention indicating purpose, owner, and refresh cycle (e.g., “dev-integration-q2”, “uat-release-2-4”) |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Shared Development Sandbox | ⚠️ All developers work in single shared sandbox overwriting each other’s changes; creates confusion about what is deployable state |
| Stale Sandboxes | ⚠️ Sandboxes not refreshed for months diverging significantly from production configuration; testing in unrepresentative environment leads to deployment failures |
| No Data Governance in Full Copy | ⚠️ Full Copy sandboxes containing production PII, financial data, HIPAA data accessible to entire development team without access controls or data masking |
| Production Integrations from Sandbox | ⚠️ Sandboxes connect directly to production integration endpoints creating risk of test transactions affecting production data or external systems |
| Manual Post-Refresh Configuration | ⚠️ Sandbox refreshes require hours of manual reconfiguration restoring settings; overhead discourages refreshing sandboxes as frequently as needed |
| No Sandbox Documentation | ⚠️ Team lacks documentation about sandbox purposes, refresh schedules, data content, or access procedures; developers guess which environment to use for what purpose |
| Single Environment Type | ⚠️ Team uses only one sandbox type (typically Developer or Full Copy) for all purposes; misses benefits of appropriate sandbox strategy per activity |
| Sandbox Hoarding | ⚠️ Sandboxes allocated to individuals or projects and never released when no longer needed; artificial scarcity despite available licenses |
| Inadequate Test Data | ⚠️ Test data in sandboxes doesn’t represent production data characteristics (volumes, relationships, edge cases); testing in unrealistic environment misses production issues |
Patterns
| Where to look | What good looks like |
|---|---|
| Source Commit Stage | ✅ Developers commit changes to feature branches in Git with clear commit messages explaining context and rationale |
| Static Analysis Stage | ✅ Salesforce Code Analyzer scans Apex for security vulnerabilities (OWASP Top 10), performance anti-patterns (SOQL in loops, DML in loops), code quality issues (complexity, duplication), and governor limit risks; runs in seconds providing immediate feedback |
| Unit Test Stage | ✅ Apex tests execute validating business logic and governor limit compliance in ephemeral scratch org or dedicated integration sandbox; properly isolated tests complete in under 5 minutes |
| Validation Deployment Stage | ✅ Metadata deploys to integration sandbox using Salesforce CLI with –checkonly flag for validation-only deployment running tests without committing changes |
| Integration Test Stage | ✅ Automated tests validate multi-component workflows, external integration contracts, platform event flows, and async processing chains against shared integration environment with test data management |
| Security Scanning Stage | ✅ Additional security tools scan for hardcoded credentials, vulnerable dependencies, CRUD/FLS violations, and compliance policy violations; critical findings break builds |
| UAT Deployment Stage | ✅ Changes passing all automated gates deploy to user acceptance testing environment for business validation; deployment may be automated or require manual approval based on change risk |
| Production Deployment Stage | ✅ After UAT approval, changes deploy to production with automated one-click execution, comprehensive logging, and immediate rollback capability |
| Pipeline Execution Time | ✅ End-to-end pipeline completes in under 15 minutes for typical changes enabling rapid feedback; longer for large deployments or extensive test suites |
| Build Artifacts | ✅ Pipeline preserves deployment packages, test results, security scan reports, and execution logs for compliance and troubleshooting |
| Pipeline Tooling | ✅ Teams use Salesforce DevOps Center for platform-native CI/CD, or integrate with external platforms (GitHub Actions, GitLab CI, Azure DevOps, CircleCI, Jenkins, Copado, Gearset, AutoRABIT) |
| Failure Notifications | ✅ Pipeline failures trigger notifications to commit author and relevant channels with diagnostic context enabling rapid remediation |
| Branch Protection | ✅ Main/production branches protected requiring passing CI checks and approved reviews before merge; direct commits prevented |
| Environment Promotion | ✅ Changes flow through defined promotion path (dev → integration → UAT → production) with validation at each stage |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Manual Deployments | ⚠️ Changes deployed manually via change sets or Salesforce CLI commands with inconsistent steps; human errors, missed steps, lack of audit trail |
| No Static Analysis | ⚠️ Pipelines skip Salesforce Code Analyzer or similar tools; security vulnerabilities, performance anti-patterns, code quality issues reach production |
| Insufficient Testing | ⚠️ Pipeline only validates minimum 75% coverage without running integration tests or validating key user flows; regressions slip through |
| No Security Scanning | ⚠️ Pipelines never scan for hardcoded credentials, vulnerable dependencies, or CRUD/FLS violations; security issues discovered in production or security reviews |
| Slow Pipeline Execution | ⚠️ End-to-end pipeline takes 45-60+ minutes discouraging frequent commits and slowing feedback loops; developers commit less frequently accumulating larger changes |
| Ignored Pipeline Failures | ⚠️ Teams ignore or override pipeline failures to meet deadlines; quality gates become theater rather than actual controls |
| No Test Environment Parity | ⚠️ CI/CD validates in environment with different configuration, limits, or data characteristics than production; passing validation doesn’t guarantee production success |
| Missing Rollback Capability | ⚠️ Pipeline automates deployment but provides no automated rollback; recovery from bad deployment requires manual emergency procedures |
| No Deployment Logs | ⚠️ Deployments executed without comprehensive logging; troubleshooting failures or auditing deployments becomes difficult |
| All-or-Nothing Testing | ⚠️ Pipeline runs all tests for every change regardless of what changed; 30-minute test suite runs even for documentation updates |
Patterns
| Where to look | What good looks like |
|---|---|
| Feature Flags | ✅ High-risk changes deploy with features disabled behind custom metadata type or custom setting configuration; enables testing in production without user exposure, gradual rollout to segments, A/B testing, and instant rollback via configuration change |
| Feature Flag Scope | ✅ Feature flags reserved for high-risk changes affecting critical business processes, performance-sensitive operations, or new integration patterns; not every feature needs flags |
| Canary Deployments | ✅ Business-critical changes release to small user subset first (5-10% via permission sets or user attribute filtering); canary cohort monitored intensively 24-48 hours measuring success rates, performance, error rates before expanding rollout |
| Deployment Windows | ✅ Deployments scheduled during low-usage periods informed by Event Monitoring API usage logs showing actual traffic patterns; avoids peak usage, month-end close, quarter-end close, major business events |
| Deployment Communication | ✅ Stakeholders receive advance notification of deployment windows including expected duration, affected functionality, and rollback decision timeline |
| Rollback Procedures | ✅ Documented and tested rollback procedures for each deployment type; team can execute rollback in under 30 minutes when needed |
| Rollback Testing | ✅ Rollback procedures validated in non-production environment before significant deployments ensuring recovery capability exists |
| Deployment Validation | ✅ Post-deployment smoke tests validate critical user flows, integration endpoints, and key functionality immediately after release |
| Change Coordination | ✅ Coordinated deployment strategy when changes span multiple systems or require synchronized releases across Salesforce and external platforms |
| Monitoring During Deployment | ✅ Enhanced monitoring active during deployment window and post-deployment observation period (24-48 hours) watching for error rate spikes, performance degradation, or user flow failures |
| Blue-Green Pattern for Integrations | ✅ External services fronting Salesforce use blue-green deployments maintaining two production-like environments, deploying to inactive first, switching traffic after validation |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Big Bang Deployments | ⚠️ Large releases containing many features, refactorings, and changes deployed simultaneously with no gradual rollout; failure impacts all users immediately |
| Peak Hour Deployments | ⚠️ Releases scheduled during peak business hours (mid-day, month-end, quarter-end) when usage is highest and user impact of problems maximized |
| No Rollback Plan | ⚠️ Deployments proceed without documented and tested rollback procedures; when problems emerge, team scrambles to figure out recovery |
| Feature Toggles as Afterthought | ⚠️ Feature flags added reactively when problems occur rather than designed in proactively for high-risk changes; limits rollback options to full deployment reversal |
| No Deployment Validation | ⚠️ Changes deployed to production without post-deployment smoke tests; team assumes deployment succeeded and moves on; problems discovered hours later by users |
| Untested Rollback Procedures | ⚠️ Rollback procedures documented but never tested in non-production; during actual incident, procedure fails or proves inadequate |
| Concurrent Multi-System Changes | ⚠️ Coordinated changes across Salesforce and external systems deployed without synchronization or compatibility testing; creates integration failures |
| No Monitoring Enhancement | ⚠️ Deployments proceed without enhanced monitoring during deployment window; problems detected through user complaints rather than proactive alerts |
| Weekend Deployments | ⚠️ Major releases deployed Friday evening or weekends when on-call coverage is reduced and vendor support availability limited |
Patterns
| Where to look | What good looks like |
|---|---|
| Unit Test Coverage | ✅ Meaningful test coverage validating business rules, edge cases, error paths, and governor limit approach; significantly exceeds 75% minimum requirement |
| Unit Test Quality | ✅ Tests validate business logic, not platform functionality; well-written unit tests are fast (milliseconds), isolated (no shared state), and focused (each test validates one behavior) |
| Test Data Strategy | ✅ Tests use @TestSetup methods for common test data; avoid SOQL queries in tests where possible; use Test.loadData() for loading static resources |
| Mock External Dependencies | ✅ Unit tests mock external callouts using test stubs and dependency injection enabling fast execution without actual HTTP requests |
| Integration Tests | ✅ Validate interactions between triggers, flows, platform events, batch jobs, and external callouts running in sandboxes with representative test data |
| Integration Test Reliability | ✅ Integration tests are repeatable and deterministic; sporadic failures addressed through synchronization mechanisms like platform event replay IDs and batch job polling |
| End-to-End Tests | ✅ Automated browser testing (Selenium, Playwright, Provar) validates complete user journeys for business-critical flows exercising system through UI exactly as users interact |
| E2E Test Focus | ✅ E2E tests focus on highest-value user paths due to slower execution (minutes per test) and higher maintenance burden from UI changes |
| Load Testing | ✅ Performance validated under realistic concurrent user loads and data volumes using Salesforce Load Testing services or third-party tools; reveals governor limit issues, locking contention, performance degradation |
| Load Test Frequency | ✅ Load tests run before major releases and periodically (quarterly) to validate performance at 3x-5x current production volumes |
| Test Pyramid | ✅ Balanced test distribution: many unit tests (hundreds, minutes), fewer integration tests (dozens, minutes), few E2E tests (under 20, minutes); fast feedback at base, comprehensive validation at top |
| Test Isolation | ✅ Tests do not depend on each other; can run in any order; each test leaves system in clean state or uses @TestSetup for consistent starting point |
| Negative Test Cases | ✅ Tests validate error handling, validation rule enforcement, permission enforcement, and graceful degradation under adverse conditions |
| Governor Limit Testing | ✅ Tests specifically validate behavior approaching governor limits including SOQL query limits, DML limits, CPU time, heap size |
| Continuous Test Execution | ✅ Tests run automatically on every commit via CI/CD pipeline; failing tests prevent deployment to higher environments |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| Testing for Coverage | ⚠️ Tests written solely to achieve 75% minimum coverage without validating business rules or edge cases; tests that assert nothing or test trivial code paths |
| Testing Platform Functionality | ⚠️ Tests validate that Salesforce saves records or SOQL queries work rather than validating business logic; wastes effort on platform guarantees |
| No Mock Strategy | ⚠️ Unit tests make actual callouts to external systems; tests are slow, brittle to external system availability, and consume API limits |
| Sporadic Test Failures | ⚠️ Tests pass or fail unpredictably based on execution timing, data state, or test execution order; teams learn to ignore failures and re-run until tests pass |
| No Integration Testing | ⚠️ Only unit tests exist; team never validates that triggers, flows, platform events, and batch jobs work together correctly |
| Manual E2E Testing Only | ⚠️ Critical user flows validated manually before each release; no automated E2E tests; regressions discovered in production when manual testing misses scenarios |
| No Load Testing | ⚠️ Performance never validated under realistic concurrent loads; governor limit issues, locking contention, or performance degradation discovered in production |
| Test Data Dependencies | ⚠️ Tests depend on specific records existing in org or assume particular user configuration; break when sandbox refreshed or configuration changes |
| No Negative Test Cases | ⚠️ Tests only validate happy path scenarios; never test error handling, validation enforcement, or permission checks; failure modes undiscovered |
| Skipped Tests | ⚠️ Failing tests commented out of @isTest and skipped rather than fixed; test suite degrades over time as more tests are ignored |
Patterns
| Where to look | What good looks like |
|---|---|
| Review Coverage | ✅ All production-bound changes flow through code review via pull requests requiring approval before merge; includes Apex, Aura, LWC, Flows, validation rules, permission sets, sharing rules |
| Review Criteria Checklist | ✅ Explicit checklist covering security (CRUD/FLS enforcement, input validation), performance (bulkification, SOQL optimization), governor limit compliance (processing patterns, recursion prevention), maintainability (naming, documentation, complexity), test coverage (meaningful tests, edge cases), architectural consistency (design patterns, separation of concerns) |
| Review Size Limits | ✅ Pull requests kept under 400 lines of meaningful change enabling thorough review; large features broken into smaller incremental changes that are individually reviewable and independently deployable |
| Review Timing | ✅ Reviews completed within one business day maintaining development momentum and providing rapid feedback; delayed reviews create context switching overhead |
| Automated Pre-Review Checks | ✅ Static analysis, unit tests, and security scanning run automatically before human review requested; automated checks catch mechanical issues so reviewers focus on design and business logic |
| Distributed Review Responsibility | ✅ Code review responsibility distributed across team preventing bottlenecks and sharing knowledge; junior developers review senior code and explain their own code during reviews |
| Review Comments Quality | ✅ Reviewers provide specific, actionable feedback with rationale; distinguish between required changes (blocking issues) and suggestions (nice-to-have improvements) |
| Review Documentation | ✅ Pull request descriptions explain what changed, why it changed, what testing was done, and any special deployment considerations or rollback procedures |
| Declarative Config Review | ✅ Flows, validation rules, and permission changes receive same review rigor as code; configuration is architectural and deserves scrutiny |
| Follow-up on Comments | ✅ All review comments addressed before merge; questions answered, changes implemented, or rationale provided for why suggestion not adopted |
Anti-Patterns
| Where to look | What bad looks like |
|---|---|
| No Code Review | ⚠️ Changes committed directly to production branch without any peer review; eliminates primary defect detection mechanism |
| Rubber Stamping | ⚠️ Reviewers approve pull requests within seconds without actually examining changes; review becomes checkbox compliance rather than quality gate |
| Single Reviewer Bottleneck | ⚠️ Only one senior developer authorized to review code; creates bottleneck and single point of failure; knowledge concentrated rather than distributed |
| Massive Pull Requests | ⚠️ Pull requests containing thousands of lines of changes spanning multiple features; impossible to review thoroughly leading to superficial examination |
| Delayed Reviews | ⚠️ Pull requests sit unreviewed for days or weeks; developers lose context and move to other work; review comments arrive when developer has switched mental context |
| No Review Criteria | ⚠️ Team lacks explicit criteria for what reviewers should examine; review quality varies wildly by reviewer; some reviewers focus on style, others on logic, none comprehensive |
| Review Without Testing | ⚠️ Reviewers approve changes based on code inspection alone without pulling branch and testing locally or validating automated tests pass |
| Ignored Automated Checks | ⚠️ Reviewers approve pull requests despite failing static analysis, security scans, or unit tests; defeats purpose of automated quality gates |
| Hostile Review Culture | ⚠️ Reviews become personal criticism or competitive exercise rather than collaborative quality improvement; discourages learning and psychological safety |
| No Declarative Review | ⚠️ Flows, validation rules, permission sets, and sharing rules deploy without review while Apex code receives scrutiny; creates quality gap in declarative configuration |