Article / Archive
Over the last few years, Google's DDoS Response Team has observed the trend that distributed denial-of-service (DDoS) attacks are increasing exponentially in size. Last year, we blocked the largest DDoS attack recorded at the time. This August, we stopped an even larger DDoS attack — 7½ times larger — that also used new techniques to try to disrupt websites and Internet services.
This new series of DDoS attacks reached a peak of 398 million requests per second (rps), and relied on a novel HTTP/2 “Rapid Reset” technique based on stream multiplexing that has affected multiple Internet infrastructure companies. By contrast, last year’s largest-recorded DDoS attack peaked at 46 million rps.
For a sense of scale, this two minute attack generated more requests than the total number of article views reported by Wikipedia during the entire month of September 2023.
Google mitigated a DDoS attack which peaked at 398 million requests per second
The most recent wave of attacks started in late August and continue to this day, targeting major infrastructure providers including Google services, Google Cloud infrastructure, and our customers. Although these attacks are among the largest attacks Google has seen, our global load-balancing and DDoS mitigation infrastructure helped keep our services running. In order to protect Google, our customers, and the rest of the Internet, we helped lead a coordinated effort with industry partners to understand the attack mechanics and collaborate on mitigations that can be deployed in response to these attacks.
Generally, DDoS attacks attempt to disrupt internet-facing websites and services, making them unreachable. Attackers direct overwhelming amounts of Internet traffic to targets, which can exhaust their ability to process incoming requests.
DDoS attacks can have wide-ranging impacts to victim organizations, including loss of business and unavailability of mission critical applications, which often cost victims time and money. Time to recover from DDoS attacks can stretch well beyond the end of an attack.
We observed the attack campaign continued over the course of September 2023
We were able to mitigate the attack at the edge of Google's network, leveraging our significant investment in edge capacity to ensure our services and our customers’ services remained largely unaffected. As we understood more details about the attack methodology, we developed a set of mitigations and updated our proxies and denial-of-service defense systems to efficiently mitigate this technique. Since Google Cloud’s Application Load Balancer and Cloud Armor use the same hardware and software infrastructure that Google relies on to serve its own internet-facing services, the Cloud customers who use those services have their Internet-facing web apps and services similarly protected.
This cross-industry collaboration has resulted in patches and other mitigation techniques used by many large infrastructure providers. The collaboration helped to pave the way for today’s coordinated responsible disclosure of the new attack methodology and potential susceptibility across a multitude of common open source and commercial proxies, application servers, and load balancers.
The collective susceptibility to this attack is being tracked as CVE-2023-44487 and has been designated a High severity vulnerability with a CVSS score of 7.5 (out of 10).
Google expresses sincere gratitude to all of the cross-industry stakeholders who have collaborated, shared information, accelerated patching of their infrastructure, and rapidly made patches available to their customers.
Google Cloud customers exposing their services using the global or regional Application Load Balancer benefit from Cloud Armor always-on DDoS protection, where attacks exploiting vulnerabilities such as CVE-2023-44487 are quickly mitigated.
Even though with Cloud Armor always-on DDoS protection we are able to efficiently absorb most of the hundreds of millions of requests per second at the edge of Google’s network, millions of unwelcome requests per second can still make it through. To protect against this and other layer 7 attacks, we also recommend deployment of Cloud Armor custom security policies with proactive rate limiting rules and AI-powered Adaptive Protection to more comprehensively detect, analyze, and mitigate attack traffic.
We provide more technical information on this current wave of DDoS attacks here, and you can learn more about Google Cloud Armor’s DDoS protection here.
---------
Thoughts: Was this also the reason Tier 1 ISPs have been going down lately?
Over the last few years, Google's DDoS Response Team has observed the trend that distributed denial-of-service (DDoS) attacks are increasing exponentially in size. Last year, we blocked the largest DDoS attack recorded at the time. This August, we stopped an even larger DDoS attack — 7½ times larger — that also used new techniques to try to disrupt websites and Internet services.
This new series of DDoS attacks reached a peak of 398 million requests per second (rps), and relied on a novel HTTP/2 “Rapid Reset” technique based on stream multiplexing that has affected multiple Internet infrastructure companies. By contrast, last year’s largest-recorded DDoS attack peaked at 46 million rps.
For a sense of scale, this two minute attack generated more requests than the total number of article views reported by Wikipedia during the entire month of September 2023.
Google mitigated a DDoS attack which peaked at 398 million requests per second
The most recent wave of attacks started in late August and continue to this day, targeting major infrastructure providers including Google services, Google Cloud infrastructure, and our customers. Although these attacks are among the largest attacks Google has seen, our global load-balancing and DDoS mitigation infrastructure helped keep our services running. In order to protect Google, our customers, and the rest of the Internet, we helped lead a coordinated effort with industry partners to understand the attack mechanics and collaborate on mitigations that can be deployed in response to these attacks.
Generally, DDoS attacks attempt to disrupt internet-facing websites and services, making them unreachable. Attackers direct overwhelming amounts of Internet traffic to targets, which can exhaust their ability to process incoming requests.
DDoS attacks can have wide-ranging impacts to victim organizations, including loss of business and unavailability of mission critical applications, which often cost victims time and money. Time to recover from DDoS attacks can stretch well beyond the end of an attack.
Our investigation and response
Our investigation revealed that the attack was using a novel “Rapid Reset” technique that leverages stream multiplexing, a feature of the widely-adopted HTTP/2 protocol. We provide further analysis of this new Rapid Reset technique and discuss the evolution of Layer 7 attacks in a companion blog.
We observed the attack campaign continued over the course of September 2023
We were able to mitigate the attack at the edge of Google's network, leveraging our significant investment in edge capacity to ensure our services and our customers’ services remained largely unaffected. As we understood more details about the attack methodology, we developed a set of mitigations and updated our proxies and denial-of-service defense systems to efficiently mitigate this technique. Since Google Cloud’s Application Load Balancer and Cloud Armor use the same hardware and software infrastructure that Google relies on to serve its own internet-facing services, the Cloud customers who use those services have their Internet-facing web apps and services similarly protected.
Industry coordination and response for CVE-2023-44487
Soon after detecting the earliest of these attacks in August, Google applied additional mitigation strategies and coordinated a cross-industry response with other cloud providers and software maintainers who implement the HTTP/2 protocol stack. We shared intelligence about the attack and mitigation methodologies in real time as the attacks were underway.This cross-industry collaboration has resulted in patches and other mitigation techniques used by many large infrastructure providers. The collaboration helped to pave the way for today’s coordinated responsible disclosure of the new attack methodology and potential susceptibility across a multitude of common open source and commercial proxies, application servers, and load balancers.
The collective susceptibility to this attack is being tracked as CVE-2023-44487 and has been designated a High severity vulnerability with a CVSS score of 7.5 (out of 10).
Google expresses sincere gratitude to all of the cross-industry stakeholders who have collaborated, shared information, accelerated patching of their infrastructure, and rapidly made patches available to their customers.
Who is susceptible and what to do about it
Any enterprise or individual that is serving an HTTP-based workload to the Internet may be at risk from this attack. Web applications, services, and APIs on a server or proxy able to communicate using the HTTP/2 protocol could be vulnerable. Organizations should verify that any servers they run that support HTTP/2 are not vulnerable, or apply vendor patches for CVE-2023-44487 to limit impact from this attack vector. If you are managing or operating your own HTTP/2-capable server (open source or commercial) you should immediately apply a patch from the relevant vendor when available.Next steps
Defending against massive DDoS attacks such as those described here is difficult. With or without patches, organizations would need to make significant infrastructure investments to keep services running in the face of attacks of any moderate size and larger. Instead of bearing that expense themselves, organizations running services on Google Cloud can take advantage of our investment in capacity at global scale in our Cross-Cloud Network to deliver and protect their applications.Google Cloud customers exposing their services using the global or regional Application Load Balancer benefit from Cloud Armor always-on DDoS protection, where attacks exploiting vulnerabilities such as CVE-2023-44487 are quickly mitigated.
Even though with Cloud Armor always-on DDoS protection we are able to efficiently absorb most of the hundreds of millions of requests per second at the edge of Google’s network, millions of unwelcome requests per second can still make it through. To protect against this and other layer 7 attacks, we also recommend deployment of Cloud Armor custom security policies with proactive rate limiting rules and AI-powered Adaptive Protection to more comprehensively detect, analyze, and mitigate attack traffic.
We provide more technical information on this current wave of DDoS attacks here, and you can learn more about Google Cloud Armor’s DDoS protection here.
---------
Thoughts: Was this also the reason Tier 1 ISPs have been going down lately?
