Cloudflare is struggling with another outage - here's what to know - Matthew Prince’s worst nightmare came true

Article / Archive

Cloudflare, one of the top web and internet services businesses, is having trouble again. In this storm, Cloudflare Dashboard and its related application programming interfaces (API) have gone down. The silver lining to this trouble is that these issues are not affecting the serving of cached files via the Cloudflare Content Delivery Network (CDN) or the Cloudflare Edge security features. No, those had gone down last week.

On Oct. 30. Cloudflare had rolled out a failed update to its globally distributed key-value store, Workers KV. The result was that all of Cloudflare's services were down for 37 minutes. Today's problem isn't nearly as serious, but it's been going on for several hours.

As of 7:15 PM Eastern time, Cloudflare reported, that it was "seeing gradual improvement to affected services."

Still, it's bad enough. Cloudflare disclosed that the snags have affected a slew of products at the data plane and edge level. These include Logpush, WARP / Zero Trust device posture, Cloudflare dashboard, Cloudflare API, Stream API, Workers API, and Alert Notification System.

Other programs are still running, but you can't modify their settings. These are Magic Transit, Argo Smart Routing, Workers KV, WAF, Rate Limiting, Rules, WARP / Zero Trust Registration, Waiting Room, Load Balancing and Healthchecks, Cloudflare Pages, Zero Trust Gateway, DNS Authoritative and Secondary, Cloudflare Tunnel, Workers KV namespace operations, and Magic WAN.

Cloudflare failures are a big deal. As John Engates, Cloudflare's field CTO, recently tweeted, "Cloudflare processes about 26 million DNS queries every SECOND! Or 68 trillion/month. Plus, we blocked an average of 140 billion cyber threats daily in Q2'23."

The root cause of these problems is a data center power failure combined with a failure of services to switch over from data centers having trouble to those still functioning.

Late in the day, Cloudflare gave ZDNET a fuller explanation of what happened:

We operate in multiple redundant data centers in Oregon that power Cloudflare's control plane (dashboard, logging, etc). There was a regional power issue that impacted multiple facilities in the region. The facilities failed to generate power overnight. Then, this morning, there were multiple generator failures that took the facilities entirely offline. We have failed over to our disaster recovery facility and most of our services are restored. This data center outage impacted Cloudflare's dashboards and APIs, but it did not impact traffic flowing through our global network. We are working with our data center vendors to investigate the root cause of the regional power outage and generator failures. We expect to publish multiple blogs based on what we learn and can share those with you when they're live.

Cloudflare is still working to resolve this problem. But, since the problem was with data center power outages rather than its software, solving it may be outside its control. Hang in there, folks. Fixing this may take a while.

Trust me, Cloudflare really wants to fix this as soon as possible. Cloudflare's earning call is today.

-

By the way, unrelated, if you check cloudflarestatus.com right now, A LOT is affected. A LOT. Ouch.

EDIT: better picture (thanks)

 

[cybertoaster]

That explains all the people complaining about WARP.

This is why google's censorship is now so blatantly obvious: their talent no longer has the knowledge to make it more subtle than "this search is banned"

That has more to do with ideological capture within the company, see how they dropped the "don't be evil" part.

 

[That_Guy]

Cloudflare gone down: Kiwifarms most not at all effected.

This big dark cloud's all nice and shiny inside. Must've been the headache of the century to get here, But what a nice place to be. Cloudflare isn't working. Grab some popcorn, sit down and watch the train wreck. Maybe have a Coke.

 

[Big moth tiddies]

Pardon my ignorance, because I am not a cyber security expert (tranny) but isn't having "multiple redundant data centers in Oregon" rather than multiple states a really retarded idea? If your company is that important to the internet, wouldn't you want at least three facilities in three different states? Having all your shit in one state doesn't sound very redundant.

While we're on the topic of "redundancies", these generators... any of us in the US and even outside of such can go buy a generator right now that will power on and do a self test every month to ensure the fucking thing works, and can even text you saying "Hey, I'm working."
Why don't these stinkditches who are supposed to be internet wizards have generators that do this?

I know this isn't how it works, but in my head I'm imagining the trannies standing in the building with gas cans refusing to go outside and fill up the generators so they can keep running.

Even github has an east coast and a west coast... I don't know what you call it... a hub? Even they have that going for redundancy, and the company covering most of the internet can't think that having one facility out of state could be a good idea.

Fucking disgraceful. I hope the earnings call is disastrous and that Matthew Prince has to suck tranny dick for cock for the rest of his shitty life.

Edit: Just looked up the earnings call. It was nothing but fluff because get this:

Cloudflare, Inc. (NYSE:NET) Q3 2023 Earnings Call Transcript November 2, 2023

Cloudflare, Inc. beats earnings expectations. Reported EPS is $0.16, expectations were $0.1.

OMFG GUYS! THEIR EPS WAS FIFTEEN CENTS MORE THAN THEY THOUGHT IT WOULD BE! BREAK OUT THE GODDAMN CHAMPAGNE!

Furious fedpost here.

 

[Heckler1]

Cloudflare are niggers.

I hear they also love girldick more than making profits. Making them, in actuality, retarded nigger faggots.

 

[ZazietheBeast]

This is the price of taking the girldick. A series of disasters that arrive by capitulating to perverted loons. Troons you will.

Pardon my ignorance, because I am not a cyber security expert (tranny) but isn't having "multiple redundant data centers in Oregon" rather than multiple states a really retarded idea? If your company is that important to the internet, wouldn't you want at least three facilities in three different states? Having all your shit in one state doesn't sound very redundant.

This is run by a guy who in turn follows the ideas of mutilated men that play pretend. Disaster was bound to follow. Also the idea of that biz based in a soon-to-be zone of exclusion is hilarious. Meaning some hobo can utterly screw things up for big tech.

 

[REGENDarySumanai]

Pardon my ignorance, because I am not a cyber security expert (tranny) but isn't having "multiple redundant data centers in Oregon" rather than multiple states a really retarded idea?

Yes. You want it spread out as possible. Preferably in different states and built to withstand anything you can throw at it.

 

[Snekposter]

Yes. You want it spread out as possible. Preferably in different states and built to withstand anything you can throw at it.

I will give anyone who thinks they actually spend more than the bare minimum on physical protection for their data centers a most excellent payout on their bet.

 

[Shroom King]

Why don't these stinkditches who are supposed to be internet wizards have generators that do this?

Because they are trannies. People who cut off their dicks for reddit updoots aren't always thinking in the long-term.

 

[TheRedChair]

Remember... The Progressives eat their own. This what happens when a business goes woke.

 

[Twinkletard]

Pardon my ignorance, because I am not a cyber security expert (tranny) but isn't having "multiple redundant data centers in Oregon" rather than multiple states a really retarded idea?

Oregon is the best strategically-speaking, in terms of kwh ratios and its location on the busiest section of the internet backbone in North America, the NWAX. Many internet service companies locate their infrastructure there for those very reasons.

 

[CarlosDanger]

Kiwiflare seems to be working just fine though.

The virgin Cloudflare vs the Chad Kiwiflare.

 

[NoReturn]

Gentlekiwis, might I remind you all that poor CloudFlare is just trying to be a responsible business?
We need to help them!
I suggest anyone with time take some of it to file helpful forms at https://www.cloudflare.com/trust-hub/reporting-abuse/ of all the nasty websites out there who host icky content like porn. Surely a fine, upstanding service provider like CloudFlare wouldn't be reliant upon pornographic websites for a significant percentage of their revenue.

 

[Übertroon]

It looks like CloudFlare has been "googled".
When tech companies reach a certain age and level of dominance, they become too big and unwieldy to manage themselves.
The old talent tire of the red tape introduced by so many chefs in the kitchen and leave.
The people left behind lack the depth of understanding the founders had (basically, warhammer 40k 'dark age of technology' in micro).
The solution being delivered continues on inertia, but the pace of feature development slows to a crawl, and the further away from the original knowledge of the product you get, the more it's "spitwads and bailing wire" levels of "maintenance".
This is why google's censorship is now so blatantly obvious: their talent no longer has the knowledge to make it more subtle than "this search is banned"

It's like twitter where Elon fired 70% of the staff and nothing happened, actual parasites

 

[Zbk3VmLfujjzjxBd43tL]

Oh no, how will I access all my favorite DDoS for hire websites that Cloudflare shields!

 

[innocent jogger]

Cloudflare are niggers.

Troons too I would imagine, but niggers nonetheless.

I suggest anyone with time take some of it to file helpful forms at https://www.cloudflare.com/trust-hub/reporting-abuse/ of all the nasty websites out there who host icky content like porn.

I can think of one particularly disgusting site.

 

[The Littlest Shitlord]

It looks like CloudFlare has been "googled".
When tech companies reach a certain age and level of dominance, they become too big and unwieldy to manage themselves.
The old talent tire of the red tape introduced by so many chefs in the kitchen and leave.
The people left behind lack the depth of understanding the founders had (basically, warhammer 40k 'dark age of technology' in micro).
The solution being delivered continues on inertia, but the pace of feature development slows to a crawl, and the further away from the original knowledge of the product you get, the more it's "spitwads and bailing wire" levels of "maintenance".
This is why google's censorship is now so blatantly obvious: their talent no longer has the knowledge to make it more subtle than "this search is banned"

See also: Microsoft.

 

[JULAY]

Oh no, how will I access all my favorite DDoS for hire websites that Cloudflare shields!

Not to mention your ISIS recruitment, your cartel operations, your CP rings, your Zoosadist sites, your organized ransomware sites... All of those things are totally less evil and insidious than Kiwi Farms.

 

[moocow]

"37 minute outage" my aching ass. Their shit broke yesterday morning and it's still fucked good 'n plenty right now. It's made the last two workdays a pain in my ass. Their APIs either time out, respond with errors, or just "do nothing." The dashboard is unusable, assuming you can even log in. Even if you can get it to load, none of the widgets work because of all the API errors it flashes up as little toasts at the bottom of the screen.

Can't update DNS records, change firewall rules, purge caches, nothing.

On the plus side, since it broke our CI/CD pipelines too (no DNS updates == no passing health checks == failed pipelines and no deployment), it prevented my colleagues from doing their usual 4:45pm Friday production release.

Fuck you, Cloudflare. The moment I can migrate my employer off your shit, I will.

 

[A Random Lurker]

Suffer prince

 

[Cpt. Stud Beefpile]

Pardon my ignorance, because I am not a cyber security expert (tranny) but isn't having "multiple redundant data centers in Oregon" rather than multiple states a really retarded idea? If your company is that important to the internet, wouldn't you want at least three facilities in three different states? Having all your shit in one state doesn't sound very redundant.

My shiny new CompTIA certs say "damn straight".

True redundancy means distribution across significant geographical distances, and generally the further the better. So that even if a 9.0 earthquake in California flattens one site to a pancake AND another site in Nebraska gets creamed by a tornado, your third site in Maine can at least pick up most of the slack.

At absolute bare minimum you want a backup site in another US state, and that doesn't mean 10 miles away because your current site is right by the state line.