Magento Root Cause Analysis: Five Whys That Stick

lead

An alert fires: server load 95%, pages crawling, AI crawlers all over the access log. The obvious fix takes ten minutes — block the bots in robots.txt — and it works, in the sense that the load graph comes down and nobody is paged again that night. It is also the wrong fix, and the reason it is wrong is the entire subject of this article: the bots were not causing slow pages, they were revealing them. Real customers had been waiting five seconds for a category page the whole time; there were simply too few of them at once to show up on a graph. Magento root cause analysis is the discipline of not stopping at the explanation that makes the alert go away — and in a Magento 2 codebase specifically, the causes cluster into a handful of recognisable shapes that live one layer away from wherever the symptom appeared.

The stopping rule that makes five-whys work in practice: stop at the last layer you can act on and verify. Not at five. If the answer at depth six is a cultural aspiration rather than a change someone can ship this quarter, depth five was your root cause. And if the answer to why-1 is a specific person’s action, you have found the trigger and not started the analysis.

baseline: the five whys, and why teams stop at one

A. Stop at Symptom (Bad)

Incident: "Server load 95%, site slow"
Decision: "Block AI bots in robots.txt"
Result: Bots blocked ✓, but legitimate users STILL slow (root cause not fixed)

B. Dig to Root Cause (Good)

Incident: "Server load 95%, site slow"
Why 1: "High CPU usage when AI bots crawl"
Why 2: "Each page takes 5 seconds to render"
Why 3: "Rendering takes 5s because of N+1 queries (50 queries for catalog page)"
Why 4: "N+1 queries because product attributes loaded individually, not batch"
Why 5: "No index on product_id in attribute loader"

ROOT CAUSE: Missing database index (developer error in migration)
REAL FIX: Add index to attribute_id foreign key
BONUS: Legitimate users also benefit (pages 5s → 0.5s for everyone)

C. The 5-Why Process (Step-by-Step)

1. STATE THE SYMPTOM (observable problem):
   "Load average 15.2, checkout times out"

2. ASK WHY #1 (one layer deeper):
   "Why is load so high?"
   → "Checkout process making 200+ database queries"

3. ASK WHY #2 (next layer):
   "Why so many queries?"
   → "Order totals calculated by looping through items (1 query per item)"

4. ASK WHY #3 (next layer):
   "Why not batch-load?"
   → "Observer was written before batch-loading was available in Magento"

5. ASK WHY #4 (next layer):
   "Why wasn't it updated?"
   → "No performance regression tests, so slow queries weren't caught"

6. ROOT CAUSE IDENTIFIED:
   "Missing performance regression tests in CI"

FIXES (in order of impact):
├─ Immediate: Batch-load order items (5 min fix, fixes checkout)
├─ Short-term: Add performance regression test (prevents repeat)
└─ Long-term: Audit all observers for N+1 patterns (systemic issue)

working_example: the AI-bot load spike that was never about bots

Scenario: "AI bots hammering the site, server load 95%"

Layer 1: Observation (Symptom)

2026-08-21 15:30 UTC
├─ Alert: Load average 15.2 (normal: 2.5)
├─ Notice: Bots identified in logs (ChatGPT crawler, GoogleBot, Perplexity)
├─ Impact: Legitimate users report slow pages (3-5s load time)
└─ Decision point: Block bots? Or investigate?

Layer 2: Bot Analysis (First Why)

Why is load high when bots crawl?

Answer: Each bot request takes 5 seconds to render
├─ ChatGPT crawler: 1000 requests/hour × 5s/request = 5000 seconds load
├─ GoogleBot: 200 requests/hour × 5s/request = 1000 seconds load
└─ Total: ~6000 seconds of CPU per hour for bots

Comparison:
├─ Legitimate users: 500 requests/hour × 0.5s/request = 250 seconds load
└─ Ratio: Bots create 24x more load than users (not normal!)

Layer 3: Rendering Performance (Second Why)

Why does each page take 5 seconds?

Profile with Blackfire:
├─ Database queries: 3500ms (70% of time!)
├─ Template rendering: 800ms (16%)
├─ Image optimization: 300ms (6%)
├─ Cache lookup: 400ms (8%)
└─ Total: ~5000ms per page

ROOT BOTTLENECK: Database queries (70% of rendering time)

Layer 4: Query Analysis (Third Why)

Why so many database queries?

EXPLAIN + slow_query_log analysis:
├─ catalog_product table scan (no index): 1200ms
├─ product_attribute loop (1 query per attribute): 1400ms (200 attributes × 7ms)
├─ inventory check (no index): 600ms
├─ category load (N+1 pattern): 300ms
└─ Total: 3500ms in queries

PATTERN DETECTED: Multiple N+1 queries + missing indexes

Layer 5: Code Review (Fourth Why)

Why N+1 queries and missing indexes?

Code investigation:
1. Product loader (AttributeLoader.php:45)
   └─ Loop: foreach($attributes as $attr) { $model->load($attr->getId()); }
   └─ Issue: 200 iterations = 200 queries (N+1 pattern)

2. Inventory checker (InventoryValidator.php:82)
   └─ Query: SELECT * FROM inventory WHERE product_id = $id (no index)
   └─ Issue: Full table scan on every product check

3. Category loader (CategoryLoader.php:33)
   └─ Loop: foreach($products as $product) { $category->load($product->getId()); }
   └─ Issue: Another N+1 (N categories × N products)

ROOT CAUSES IDENTIFIED:
├─ Observer written in 2020 before batch-loading was standard
├─ Inventory query never indexed (added in 2023, index forgotten)
└─ Category loader written by contractor, never reviewed for performance

Layer 6: Prevention (Fifth Why)

Why weren't these caught before production?

Audit findings:
├─ No CI performance regression tests (can deploy slow code)
├─ No code review checklist for N+1 patterns
├─ No monitoring alert for query count per page (10 normal, this had 200)
└─ No bot-specific analysis (bot crawls are different from user traffic)

SYSTEMIC ISSUES:
├─ Process: Code review checklist missing performance items
├─ Testing: No performance baseline tests
├─ Monitoring: No alerts for query spike
└─ Culture: Performance treated as "nice to have", not required

The Fix (Not Just Blocking)

Immediate (5 min fix)

// FIX: Batch-load attributes instead of looping
// BEFORE (N+1 pattern):
foreach ($attributes as $attribute) {
  $model->load($attribute->getId());  // 200 queries!
}

// AFTER (batch-load):
$ids = array_column($attributes, 'id');
$models = $modelCollection->addFieldToFilter('id', ['in' => $ids])->getItems();
// 1 query instead of 200!

Short-term (2 hours fix)

-- Add missing indexes
ALTER TABLE inventory ADD INDEX idx_product_id (product_id);
ALTER TABLE product_attribute ADD INDEX idx_product_id (product_id);

-- Verify with EXPLAIN
EXPLAIN SELECT * FROM inventory WHERE product_id = 123;
-- type should be "ref" (indexed), not "ALL" (full scan)

Long-term (1 week)

Add performance regression tests to CI:
├─ Track query count per page (alert if >50 queries)
├─ Track page load time (alert if >2 seconds)
├─ Track AI bot metrics separately (alert if >1000 bot requests/hour)
└─ Auto-block deployment if metrics regress >10%

Results (Before vs After)

Before RCA fix:

Load average: 15.2 (alert threshold: 8.0)
Page load time: 5000ms (target: 1000ms)
Bot requests: 1000/hour (each takes 5s = 5000s load)
User complaints: "Site is slow" (every bot crawl cycle)
Decision: Block bots (temporary band-aid)

After RCA fix:

Load average: 2.3 (healthy)
Page load time: 500ms (well below target)
Bot requests: 1000/hour (each takes 0.5s = 500s load, acceptable)
User complaints: None (legitimate users fast too)
Decision: Welcome bots, let them index your content naturally
Bonus: Legitimate users also 10x faster (5s → 0.5s)

Four more chains, compressed

Example 1: "Checkout Timeout at Peak Hours"

Symptom: Checkout timeouts 2-4 PM (peak traffic)

Why 1: Checkout process slower at peak?
→ Database connection pool exhausted (max_connections = 100)

Why 2: Why exhausted at peak?
→ Each checkout holds connection for 30 seconds (too long)

Why 3: Why 30 seconds?
→ Payment processor API call takes 25 seconds (slow third-party)

Why 4: Why not timeout earlier?
→ No timeout configured, request just waits

Why 5: Why no timeout?
→ Configuration missing from payment module

ROOT CAUSE: Missing payment API timeout configuration
FIXES:
├─ Immediate: Set timeout to 5 seconds (fail fast instead of hang)
├─ Short-term: Use async payment queue (don't block user)
└─ Long-term: Switch payment processor or negotiate SLA

Example 2: "Memory Leak in Background Jobs"

Symptom: Server memory grows 1% per hour, OOM crash after 48 hours

Why 1: Memory growth per hour?
→ Background job creates objects but never releases them

Why 2: Which job?
→ Price recalculation job (runs every hour)

Why 3: What objects aren't released?
→ Product models loaded in loop without cleanup

Why 4: Why not cleaned up?
→ Developer called $product->load() in loop, forgot unset()

Why 5: Why not caught?
→ No memory profiling in CI, only functional tests

ROOT CAUSE: Missing memory profile testing in CI
FIXES:
├─ Immediate: unset($product) in loop (1-line fix)
├─ Short-term: Add memory profiling to CI
└─ Long-term: Use collection->walk() instead of explicit loop

Example 3: "Elasticsearch Crashes Every 6 Hours"

Symptom: Elasticsearch down 6 hours after index reindex starts

Why 1: Why does reindex crash ES?
→ Reindex uses 100% of JVM heap memory

Why 2: Why use so much?
→ Reindex job loads ALL products into memory at once (20,000 products)

Why 3: Why all at once?
→ Developer forgot to add batch limit to query

Why 4: Why not caught?
→ Reindex only runs on production (batch size works on dev)

Why 5: Why production-only test?
→ No environment parity (dev has 100 products, prod has 20,000)

ROOT CAUSE: Environment parity issue (dev != prod data volumes)
FIXES:
├─ Immediate: Reindex in batches (1000 products at a time)
├─ Short-term: Increase ES JVM heap (4GB → 8GB)
└─ Long-term: Run staging with production-scale data

Magento-specific failure shapes worth knowing before you start asking

Generic five-whys walks you down a ladder. Knowing the local terrain tells you which ladder to pick up. Four failure shapes recur in Magento 2 often enough that recognising them saves an entire round of questions — and all four share a property that makes them nasty: the symptom appears somewhere other than the cause.

Two observers, one entity, no defined order

Two modules subscribe to catalog_product_save_after. Both mutate the product. Magento resolves observer order from the merged events.xml across the module load sequence, which is derived from <sequence> declarations — and if neither module declares a sequence relative to the other, the order is stable on your machine and stable on production and not necessarily the same order. The result is a bug that reproduces on staging one week in three.

The tell: intermittent data corruption on save, no error, no stack trace, and a diff between environments that has nothing to do with data. The root cause is almost never "observer B is wrong" — it is "two modules own the same write and neither declared that". The fix is a <sequence> entry, or better, collapsing both writes into one owner.

Two plugins on the same method, one of them silently losing

Same shape, different mechanism. Two around plugins wrap the same public method. The outer one does not call $proceed() on some branch — a cache hit, an early return, a guard clause — and the inner plugin simply never runs. Nothing errors. The behaviour of module B disappears whenever module A takes its short-circuit path.

The tell: "feature works, except on cached pages" or "works for guests, not logged-in customers". Start from bin/magento dev:di:info on the class, list every plugin on that method, and read the around ones for missing $proceed() calls. This is also the reason around plugins on hot paths are a code smell rather than merely a performance note — they make other people’s code conditionally invisible.

The cache invalidation that was never declared

A block renders stale data. Not always — only between an update and the next full cache flush. The cause is a custom block or ui component whose cache tags do not include the entity it actually reads, so nothing invalidates it when that entity changes. Five whys will march you from "customers see the old price" through "the block is cached" to "the block’s identities do not include the price entity", and the fix is a getIdentities() implementation, not a cache flush in the deploy script.

The tell worth internalising: any bug whose workaround is "flush the cache" has a root cause in cache tags. Flushing is the band-aid; the identity list is the fix.

GraphQL resolvers rediscovering N+1

Resolvers compose. A product resolver calls a price resolver per item, which calls a tier-price resolver per item, and a request for 24 products issues several hundred queries. It looks like a database problem and profiles like one, which is why teams add indexes and see nothing improve. The root cause sits one layer up in the resolution graph, and the fix is batching — a data loader per level — not query tuning.

The general lesson under all four: in Magento the symptom surfaces at the layer that renders, and the cause usually lives at the layer that wires. Ask "who else touches this?" before you ask "what is slow?".

tradeoff: what the band-aid actually costs

Band-Aid vs Root Cause

Approach	Example	Cost	Time	Recurrence
Band-Aid	"Block bots in robots.txt"	€0	10 min	Every time bots update crawlers (monthly)
Root Cause	"Fix N+1 queries + add indexes"	€200 (2 hours dev)	2 hours	Never (structural fix)

Band-aid cost over 1 year: 12 incidents × 30 min each = 6 hours = €600

Root cause cost: 2 hours once = €200

Net savings: €400 per year, plus users benefit 10x

Culture Shift

Old way (reactive):

Incident happens
→ Team panics
→ Apply quick fix (band-aid)
→ Move on
→ Same incident 2 weeks later
→ Repeat

New way (proactive via RCA):

Incident happens
→ Team investigates root cause (5-why framework)
→ Apply structural fix (code change)
→ Add monitoring to prevent recurrence
→ Fix never happens again
→ Team learns, performance improves

Where root cause analysis goes wrong

The method is simple enough that its failure modes are all human, and every one of them produces a confident, wrong answer.

Asking why past the point of usefulness. Five is a heuristic, not a target. Every chain eventually terminates in something true and useless — "because the team was under deadline pressure", "because software has bugs". The stopping rule that works: stop at the last layer you can act on and verify. If the proposed fix at depth six is a cultural aspiration rather than a change someone can make this quarter, depth five was your root cause.

Groupthink dressed as consensus. The most experienced person in the room says "this smells like the indexer" in minute two, and every subsequent why gets steered toward the indexer. The chain looks rigorous and confirms a hypothesis nobody tested. The cheap countermeasure is ordering: gather the data before anyone states a theory, and have the least senior person present their reading first. If your five-whys reached the conclusion the loudest person opened with, run it again from the evidence.

Confusing the trigger with the cause. "The deploy caused it" is nearly always the trigger. The cause is whatever made the system fragile enough for that deploy to break it — the missing test, the untyped config, the assumption two modules both made. A useful check: if the answer to why-1 is a specific human action, the analysis has not started yet.

Skipping the prevention layer. Every genuine root cause has a companion question that teams routinely skip: what should have caught this and didn’t? A test that does not exist, an alert whose threshold is wrong, a review checklist item nobody wrote. Answering it converts a fix into a fix plus a guardrail, which is the entire difference between an incident that happens once and one that happens quarterly.

Blame framing, which quietly destroys the input data. "Who broke this?" and "why did this happen?" produce different information from the same people. Under the first, engineers who know something inconvenient say less, and the chain runs on partial evidence. This is not a morale point; it is a data-quality point. The team that feels safe describing exactly what they did at 16:40 produces analyses the blame-framed team cannot.

Documenting the answer but not the chain. A post-mortem that records "root cause: missing index" teaches nobody. The chain is the reusable artefact — it shows the next person which questions were worth asking. Write down the whys, including the ones that dead-ended.

verification: the RCA checklist

Step 1: State the Symptom

[ ] What's observable? (load high, page slow, error spike, etc.)
[ ] When did it start? (exact timestamp)
[ ] How many users affected? (1 or 1000?)
[ ] Business impact? (revenue lost, reputation risk, etc.)

Step 2: Gather Data (Don’t Hypothesize Yet)

[ ] Sentry events: What errors spiked?
[ ] Loki logs: Full request breadcrumb (entry → error)
[ ] Grafana: Load, CPU, memory, disk, database connections
[ ] Git: What changed recently? (deployment, config change, etc.)
[ ] Database: Query performance (EXPLAIN, slow_query_log)
[ ] Application profiler: Blackfire, XDebug (where's time spent?)

Step 3: Ask "Why?" 5 Times (Dig Deeper Each Time)

Why 1 (surface): What triggered the symptom?
  Answer: ___________________________________

Why 2 (process): Why did that trigger it?
  Answer: ___________________________________

Why 3 (system): Why is the system designed that way?
  Answer: ___________________________________

Why 4 (code): Why wasn't this prevented in code review?
  Answer: ___________________________________

Why 5 (culture): Why don't we have checks to catch this?
  Answer: ___________________________________

[ ] ROOT CAUSE IDENTIFIED: ________________________

Step 4: Propose Multi-Level Fixes

IMMEDIATE (< 1 hour):
[ ] _________________________________

SHORT-TERM (1 day - 1 week):
[ ] _________________________________

LONG-TERM (1-4 weeks, systemic):
[ ] _________________________________

PREVENTION (never happens again):
[ ] Add monitoring alert
[ ] Add CI check (test, lint, performance)
[ ] Add code review checklist item
[ ] Add documentation/runbook

Step 5: Validate the Fix

[ ] Reproduce original symptom (before fix)
[ ] Apply fix
[ ] Verify symptom gone
[ ] Monitor metrics (load, latency, errors) for 24 hours
[ ] Check if legitimate users also benefit (not just bots)

Getting the habit to stick

1. Week 1: Train team on 5-why framework (1 incident walkthrough)

2. Week 2-3: Practice on past incidents (retrospective RCA)

3. Week 4+: Apply to every incident (becomes standard practice)

ROI: After 3 incidents prevented via RCA → cost of initial training recovered.

Sources & References

Adobe Commerce: events and observers — observer ordering and <sequence> resolution.
Adobe Commerce: plugins (interceptors) — around plugin semantics and $proceed().
Adobe Commerce: cache and getIdentities() — the identity list behind stale-block bugs.
MySQL EXPLAIN output format — reading the query plan referenced in the worked example.
Taiichi Ohno’s five whys, as described in Toyota Production System — the origin of the method, and of the warning that it is a thinking tool rather than a form to fill in.

Disclaimer

The cost figures in the tradeoff section are illustrative arithmetic at a mid-range European contractor rate, not accounting from a specific engagement — the ratio is the point, not the absolute numbers. The worked example is a composite drawn from a real incident with details generalised.