Inside the Microsoft 365 outage and why outages happen
The Microsoft 365 outage knocked out Outlook worldwide into a second day. What happened, why outages like this happen and what engineers do.
On this page 3 sections ▾
TL;DR
- Microsoft says the underlying issue is fixed and that it wasn't a cyberattack, but the root cause will only come in a full post-incident report.
- It's the year's third big Microsoft 365 outage, after a nearly 10-hour one in January and a days-long one in September blamed on a core authentication configuration.
For hours on end, and into a second day, a big chunk of the working world couldn't open Outlook. While millions waited for their inboxes to come back, engineers on the other side of the Microsoft 365 outage were working a job most people never see, and here's what it looks like.
Quick facts
Microsoft says the elevated load on its servers wasn't caused by an increase in traffic.
Teams chats and calls mostly kept working, even while Outlook was down.
Government systems in Singapore and Japan were disrupted, and Australia's spy agency and federal police were among those hit.
#What happened in the Microsoft 365 outage
Problems started around 16:00 GMT on Thursday, November 12, first in the Americas. Within about an hour, Downdetector had reports from 74 countries and territories. Outlook took the hardest hit, with many emails arriving late, and the trouble spread to Teams, Copilot and integrations that companies like OpenAI and Anthropic rely on.
Early on, Microsoft said some of its infrastructure was under "elevated load" and that it was redistributing connection traffic across its worldwide infrastructure to balance it out. That didn't hold.
- Outlook problems begin in the Americas
- Downdetector shows reports from 74 countries and territories
- 12+ hours in, Microsoft reports only "marginal success", and servers it had restored start failing again
- Microsoft rolls back a network change, checks that new patches are working and looks for ways to add capacity
Microsoft's later updates filled in more. Servers had been failing automated health checks, and at 21:00 UTC on Thursday a different way of checking server health brought some of them back. At 17:50 UTC on November 13, Microsoft reported broad recovery and said the underlying issue had been fixed, with a full post-incident report to follow.
#Why Microsoft 365 outages happen
Saying a bad change caused it doesn't explain much. Here's what actually starts big outages, and what turns a small fault on one service into email going down on every continent:
- Health checks that backfirePulling a sick server out of rotation is the right call for one machine, but when many fail at once, each removal piles its users onto the survivors, which then fail their own checks. That's one way load can climb with no extra traffic, which fits Microsoft's statement that the elevated load wasn't caused by more traffic.
- Retry stormsEvery Outlook client, phone and app that loses its connection typically tries again, often right away, so a struggling service gets hit with more requests at the moment it can handle the fewest.
- Changes that reach everywhereA configuration change can spread to servers worldwide far faster than new code, so a mistake is everywhere before anyone notices. September's Microsoft 365 outage was blamed on a core authentication configuration.
- Shared building blocksSign-in and DNS sit underneath nearly everything, so when one stumbles, services with nothing else in common fail together. Two big Azure incidents were exactly this: multi-factor sign-in problems in November 2018 and DNS maintenance issues in May 2019.
- Physical problemsPower, cooling and hardware still fail. In September 2018, a datacenter outage in Azure's South Central US region took down the services running there.
- Never just one failureMark Russinovich, Azure's CTO, wrote in 2019 that those three incidents were each the result of multiple failures that only caused an outage through intricate interactions. The change is often just the last piece to fall into place.
We will never be able to avoid outages entirely.
#What life is like inside Microsoft during an outage
For your information
Microsoft hasn't described its internal response to this outage. This section is based on Microsoft's own published incident-management guidance and the steps it has reported publicly.
- The page goes outMonitoring spots the problem and alerts the on-call engineers for the affected services, wherever they are and whatever time it is. Microsoft's guidance says teams should keep on-call schedules and escalation plans ready so first-line support can reach the right people fast.
- Everyone joins the bridgeThe engineers pulled in form what Microsoft calls the bridge: one triage team of technical experts and decision-makers working the problem together. For an outage this size, that would mean teams from mail, networking and every service that depends on them.
- One person owns itAn incident response manager owns the incident from detection to root cause. They don't fix servers themselves. Their job is keeping the process on track, getting decisions made and making sure the right people are informed.
- Stop the bleeding firstThe first goal is getting service back, not explaining what broke. That fits the network rollback in the timeline, done before Microsoft said anything about the cause.
- Make the hard callsFew choices are clean. Roll the change back, or push a new fix forward? Shift traffic to healthy regions and risk overloading them too, or keep it where it is? Each option can make things worse, and the incident manager has to pick one with only part of the picture.
- Write it up, without blameAfter the fix, a retrospective rebuilds the timeline and turns what went wrong into fixes. Microsoft's guidance says it should be blameless: the goal is better systems.
ELI5: explain it simply▾
THE WORDS
- On-call
- Being the person who has to answer an alert, day or night, during your scheduled shift.
- Roll back
- Undo a recent change and go back to the last version that worked.
THE IDEA
Think of a kitchen during a dinner rush when the oven dies. The head chef doesn't stop to figure out why it broke: everyone switches to the backup oven to get plates out, and the repair person works out the cause after the rush.
Assume the change is the likely cause and roll it back immediately instead of spending a long time investigating first.
The people you don't see
- The comms teamSomeone writes every status update, under pressure to say something fast without saying something wrong. Microsoft's own explanation moved over the day, from infrastructure under elevated load to rolling back a network change. For Azure, Microsoft's stated goal is to notify every affected customer within 15 minutes, and Sami Kubba, the program manager who ran its outage communications, has said that can't be done by people alone, so automated notices handle much of the first wave.
- Support and account teamsWhile engineers work the bridge, support staff absorb the flood of tickets, and account managers take calls from big customers, government agencies included, who want answers Microsoft doesn't have yet.
- Whoever pushed the changeIf the network change turns out to be the trigger, someone made it. A blameless review is built for exactly this person: the goal is finding out why the safety checks let the change through, not punishing the engineer who clicked deploy.
Why the fix took so long
Friday morning showed the hardest part of recovery: servers Microsoft brought back started failing again. At this scale, a restored server typically gets hit with a backlog of waiting users and delayed mail at once, so teams have to bring capacity back gradually instead of flipping everything on together.
Incident response is a big part of cloud and DevOps jobs, and someone has to be on that bridge. Not sure which path fits you? Answer 4 quick questions and Ping suggests a cert and a ProCourse that fit you: find your path.
How do I check if Microsoft 365 is down?
Why did Teams and Copilot break when Outlook went down?
Will Microsoft give refunds for the outage?
What this means for you
If your company runs on Microsoft 365, agree now on a backup way to reach people (phone, text or another chat tool) for the day email and Teams go down together, and write it down somewhere that doesn't live in Outlook.
Sources