ENGINEERING NOTES
A Practical Production Debugging Checklist
A layer-by-layer checklist for scoping incidents, verifying recovery and documenting root cause.
1. Reproduce and scope
Identify the affected operation, time window and environment. Is the failure universal, intermittent or limited to particular users? Use a safe reproduction path and record expected versus actual behavior before changing configuration.
2. Read logs and trace the request lifecycle
Correlate timestamps and request identifiers across available logs. Follow the request from entry point to application and dependencies. Keep sensitive payloads and credentials out of debugging notes.
3. Inspect the application layer
Check error paths, validation, authorization and dependency calls. Distinguish a business rejection from an exception or timeout. Compare the failing path with a successful request to narrow the search.
4. Check the database
Verify connectivity, connection-pool behavior, locks and slow queries where that evidence is available. Avoid running expensive diagnostics or mutating production data just to test a theory. Confirm that schema and application expectations match.
5. Check cache and sessions
Look for expired or inconsistent state, connection failures and differences between instances. Understand the impact before clearing shared state: invalidating sessions or caches can create additional load and user disruption.
6. Review proxy, load balancer and infrastructure
Check NGINX responses, load-balancer health and WAF decisions. Verify that requests reach the expected service. Review AWS/RDS health and resource pressure with the infrastructure team when the evidence points there.
7. Verify deployment and configuration
Confirm the actual running release, environment settings and recent changes. Compare healthy and unhealthy instances. Consider a rollback only with a clear understanding of database compatibility and the operational procedure.
8. Verify the fix
Retest the original failure and related paths after the corrective action. Observe errors and latency over an appropriate window; a single successful response is not enough to establish stabilization. Share confirmed recovery status with stakeholders.
9. Document root cause
Write down impact, the evidence that isolated the cause, the corrective action and a concrete prevention task. Separate confirmed facts from unresolved hypotheses. This checklist supports the portfolio’s production-engineering focus without disclosing private incidents.