Cloudflare CEO Matthew Prince has confirmed that the major global internet outage was not the result of a cyberattack, but was caused by an internal configuration mistake
Share
NEW DELHI — Cloudflare CEO Matthew Prince has confirmed that the major global internet outage was not the result of a cyberattack, but was caused by an internal configuration mistake.
The outage disrupted several major platforms, including X, ChatGPT, Canva, Discord, and many other websites and apps around the world.
In a postmortem, Prince explained that the problem began when the company made a change to permissions on a ClickHouse database cluster.
Related: Cloudflare outage hits X and several digital platforms globally, including India
The update was meant to improve access to data, but a faulty query caused the system to pull in far more information than it should have.
This error made a key “feature file” used by Cloudflare’s Bot Management system grow too large.
This feature file is refreshed and shared across Cloudflare’s network every five minutes. When the file suddenly doubled in size, it crossed the software’s limit, causing routing software at the network edge to crash.
The issue became unpredictable because the faulty file was only generated on parts of the cluster that had been updated. This meant that every five minutes, Cloudflare’s network either received a correct file and briefly recovered or a corrupted file and failed again.
This cycle of recovery and failure continued for about three hours from around 11:20 UTC, causing widespread service interruptions across the internet. Prince stressed that there was no cyberattack involved and admitted that the company initially misread the symptoms as a massive DDoS attack before identifying the real cause.
Engineers eventually stopped the faulty file from spreading, replaced it with an older correct version, and restarted the affected systems. Cloudflare said the issue was fully resolved by 17:06 UTC and described the incident as its most serious outage since 2019.
Prince apologised for the disruption and said Cloudflare will introduce stronger safeguards, including stricter limits on file sizes, global kill switches for critical updates, and a broader review of how its core systems can fail.