# DDoS Update



## Null (Nov 25, 2014)

I've taken some measures against DDoS attacks. I'd like to explain what a DDoS is and how it affects services, and what has been done to help protect against them.

The most simple form of attack is called a Layer 7, or application-level "slow attack". It relies on exploiting the victim's resources. An attacker serves a large number of requests that the server works very hard to respond to. The result is overworking itself to the point of not being able to serve real visitors.






To protect against these sorts of attacks, we employ Cloudflare. Cloudflare is a massive application that sits between the request and our servers. They absorb malicious traffic and help filter out attacks. Depending on the cleverness of the attack, sometimes Cloudflare has to respond by being more restrictive. This is when you see things like browser checks and CAPTCHAS.






Cloudflare works in a particular way. Clever people can find ways around it. Some applications are vulnerable to identification attacks, that reveal the true source of the servers. With this information, attacks could potentially bypass our front line of defense.

To handle this, I have done two things. The server now throws away any connections not from Cloudflare. That way, a direct Level 7 attack will not work. Additionally, vulnerabilities have been moved to a smaller and less important Linode. If that server is attacked, no one will care.





Since attackers will be forced to use Cloudflare to even talk to the forum, they will have their options limited. Attacks can still work, depending on what methods they use. To help combat this, I've set up three different domain names. If one of the domains is attacked, I close it and will let people know what other domain to use for the time being.





That's a very inconvenient thing to do, so it's a last resort. Still, the option is there.

Most bases are covered. Uncovering the forum's true source won't help much, and is going to be very hard to do. Attacking Orange (the sub-server) won't bring down the website. Attacking Cloudflare is almost completely useless. Even if we can't stop attackers from breaking through Cloudflare, I can shut them off like valves to keep the server online (at the cost of annoying people).

Is there a weakness? Of course. Everyone reading this already knew what a DDoS was, or at least had heard of them. That's because the Internet is very vulnerable to this sort of attack. It's a big deal. That's why it's a felony in many western countries, including the United States.

A large-scale botnet attack with thousands of infected computers directly hitting the server would look something like this.





At this point, there's nothing to be done. An attack of this magnitude shakes the very foundation of the world wide web to its bones and reveals the innate flaws of its being. The best I could do is explain to my service host the situation. I could obtain a new IP, or they could blackhole any non-Cloudflare IPs (like with what I've done on a server level).

The blue bubble in my earlier diagram ceases to work at this point because, by the time the traffic reaches the server, it's already done its damage. Compared to a Level 7 attack each connection is massively inefficient, but because there's so many of them, it doesn't matter. The Internet, as we all know, is a series of tubes. This sort of massive attack will clog them no matter what sort of fancy garbage disposal you have.

A botnet attack like that costs about $200 dollars a day, from what I've seen.


----------



## champthom (Nov 25, 2014)

I like the pictures.


----------



## Colress (Nov 25, 2014)

have we identified who is trying to attack us, to any extent? 

i believe your methods will indeed likely assist in protecting the forums, but since i believe a large-scale botnet is unlikely (because only the most advanced malware-creator would likely have access to such a group, and anyone angry at these forums likely has no such access) then we likely don't have to worry about more violent attacks.  plus, a botnet infected by malware would have already been destroyed, as far as i know. we would be grossly aware of such a piece of code (or probably those would lurk on antimalware blogs would.) so a botnet attack is unlikely. i have heard of people handing money to those who create botnets, and control them. that's the only real way a botnet like that would basically attack, as far as i know. this is extremely unlikely.

knock on wood in case though.


----------



## CatParty (Nov 25, 2014)

have diagrams in all your posts pls


----------



## Null (Nov 25, 2014)

Wilhelm said:


> have we identified who is trying to attack us, to any extent?


I've answered this question at least 5 times in public.

*Yes, *I know who it is.
*Yes,* I know why.
*No,* I'm not making it public. If he/she/they/xir want to take credit, they can.


----------



## Colress (Nov 25, 2014)

Null said:


> I've answered this question at least 5 times in public.
> 
> *Yes, *I know who it is.
> *Yes,* I know why.
> *No,* I'm not making it public. If he/she/they/xir want to take credit, they can.


that's fine. since DDoSing is a felony i, and probably a lot of kiwis here, can understand why you wouldn't want that info out.


----------



## Silver (Nov 25, 2014)

Are we allowed to ask if it's an established Kiwi or a lolcow/brethren?


----------



## Hyperion (Nov 25, 2014)

Null is so helpful!


----------



## Pikonic (Nov 25, 2014)

Null said:


> A botnet attack like that costs about $200 dollars a day, from what I've seen.


Who the fuck would pay that?


----------



## exball (Nov 25, 2014)

Pikonic said:


> Who the fuck would pay that?


----------



## Null (Nov 25, 2014)

I disabled DDoS protection for a second to fix a caching error with Cloudflare and immediately the server dropped. Normally, the attacks only happening during events. This is the first time it's gone on for more than an hour.

Buckle down, my Kiwis, this is the long haul.

As an immediate fix, I've forced stylesheets based on domain.


----------



## Blueberry (Nov 25, 2014)

I'm scared


----------



## EI 903 (Nov 25, 2014)

$200 a day to keep a bunch of autists from talking about other, slightly worse autists. That's Charb-level financial strategy, right there.


----------



## Flowers For Sonichu (Nov 25, 2014)

Pikonic said:


> Who the fuck would pay that?



Golden Knight's mom


----------



## exball (Nov 25, 2014)

Hellblazer said:


> $200 a day to keep a bunch of autists from talking about other, slightly worse autists. That's Charb-level financial strategy, right there.


TJ has plenty of money, he works for the bank.


----------



## Surtur (Nov 25, 2014)

Hellblazer said:


> $200 a day to keep a bunch of autists from talking about other, slightly worse autists. That's Charb-level financial strategy, right there.


It makes me wonder if we're laughing at the right people.


----------



## 4Macie (Nov 25, 2014)

Love the pics, but they need more kiwis


----------



## Jaimas (Nov 26, 2014)

$200 a day to essentially ween an entire forum that essentially shrugged it off?

Fuck, that's some money well-spent.


----------



## Null (Nov 26, 2014)

You guys misunderstood me. $200/day is the "DDoS Deluxe" package in the last figure. The one that would completely obliterate everything.

More likely prices for attacks that we are currently able to endure are seen in this article:
http://www.computerworld.com/article/2837175/symantec-sees-rise-in-high-traffic-ddos-attacks.html






I'm not super knowledgeable about the actual logistics behind orchestrating an attack. I just know how they work, and thanks to some reading I know a little bit about stopping them.

It's possible the attacker owns their own small botnet and are carrying out the attacks themselves. It's possible they are paying. My guess is that they are. I'm actually very suspicious that the person bought a service similar to the very one I've linked, because:


Nov 23th, we experience three separate attacks of escalating intensity. One during the Jace stream, one immediately after, and one much later in the afternoon.

Nov 24th, we experience an attack late at night.

Nov 25th, we experience a much longer attack during the Jace stream.


I'm thinking that, what they're doing, is buying the very short timeframe attacks.

Day 1, $4 for Package A 1 hour, $6 for Package B 1 hour, $15 for Package C 1 hour. Each attack got much harder to deal with. $25 total.

Day 2, Package C 1 hour. $40 total.

Day 3, Package C 4 hours. $70 total. This is also the first attack that lasted _much_ longer. Hours later I went to fix a caching issue by messing with Cloudflare and immediately the server went down again because the attack was still going on.


Now again, this is all crapshooting. I have no idea. If it's just a few friends attacking, _maybe_ they could achieve these results? It just doesn't seem likely. Even if they did buy an attack, I don't know if this is the pricing. Again, I've seen everything from $5/day to $1000/day, based on the attack.

I'm certain they don't spend too much money, because _we're still here_, but I'm also doubtful they're carrying out the attack with personal devices. If they _were_, it wouldn't be over ever. It would be a perpetual siege. Money is more fleeting than time, however, so it would make sense to pick your attacks wisely if you had to pay for it.


Again, can't be certain of the costs, but this is what I know.


----------



## champthom (Nov 26, 2014)

Hellblazer said:


> $200 a day to keep a bunch of autists from talking about other, slightly worse autists. That's Charb-level financial strategy, right there.



Well shit, people are actually _paying _$200 to attack our forums? All we do is talk about online weirdos, no idea why they feel so strongly that they need to stop us.

EDIT: Okay, so they're probably not spending $200. Still, spending any amount of money to attack our forums is just silly.


----------



## Null (Nov 26, 2014)

Okay, the "blue bubble" in my diagrams are up. No non-cloudflare connections are allowed to the server. This is very experimental, and I will be keeping close tabs on the denials, but no legitimate traffic should be getting blocked.

If you _are_ getting blocked, it's probably only going to be intermittent. This because your cloudflare IP is suddenly not blacklisted. That should never happen, though.


----------



## ThatGuy (Nov 26, 2014)

Null said:


> Okay, the "blue bubble" in my diagrams are up. No non-cloudflare connections are allowed to the server. This is very experimental, and I will be keeping close tabs on the denials, but no legitimate traffic should be getting blocked.
> 
> If you _are_ getting blocked, it's probably only going to be intermittent. This because your cloudflare IP is suddenly not blacklisted. That should never happen, though.



Null"s the man!


----------



## Null (Nov 26, 2014)

Sorry for all the downtime this morning and afternoon.

I've completely rebuilt the following configurations:

iptables
apache
cloudflare
shell
ftp

I've also improved the relationship between Kiwi and Orange and resolved a few issues with the mailserver.

These are all efforts to secure the website on an application level. I've heard reports that we've gotten faster, which is nice side-effect of not doing things sloppily. If more people can chip in on if they think things are slower/faster, I'd like to hear.


----------



## c-no (Nov 26, 2014)

champthom said:


> Well shit, people are actually _paying _$200 to attack our forums? All we do is talk about online weirdos, no idea why they feel so strongly that they need to stop us.
> 
> EDIT: Okay, so they're probably not spending $200. Still, spending any amount of money to attack our forums is just silly.


Especially when the better option is to just ignore the forum. If lol-cows attacked this forum because we had a thread on them, they can just ignore us or even mock us in someway. Even then, the forum isn't like ED where its a wiki that chronicles every single thing about them.


----------



## Holdek (Nov 26, 2014)

You should report this to tech support at the hosting service.


----------



## Null (Nov 26, 2014)

Holdek said:


> You should report this to tech support at the hosting service.


I have. Linode has offered to work with me in preventing large-scale botnet attacks should they affect other servers on our network.


----------



## Blueberry (Nov 27, 2014)

Null is like Iron Man with that Robert Downey Jr Swag


----------



## Centipede (Nov 27, 2014)

I'm assuming it's someone we got a thread on who's trying to DDoS our beloved forums. Most lolcows get their money from their parents/welfare, there's no way that anyone would spend money they _earned_ on something as meaningless as this.


----------



## Null (Nov 27, 2014)

I can't verify this, but based on suspicious spikes in traffic and an hour long period where the mailserver was inaccessible, I believe Orange was attacked.

Mail services cannot work with Cloudflare, because attaching the outgoing IP of Cloudflare to an email means that they would get blamed for any spam attacks. Because of this, mail carriers are always vulnerable to what's called an Identification Attack, which reveals a real IP address. Having moved our SMTP and webmail off to Orange (a small, cheap box specifically hosted for the purpose of being an easy target that our server does not depend on), it looks like it did its job and was targeted instead of Kiwi.


----------



## Colress (Nov 27, 2014)

so whoever was responsible basically took the bait? that's pretty neat.

i've never actually quite been able to witness a server be attacked like that, so thank you for keeping us posted, Null. this has been an interesting thing to see.


----------



## Null (Nov 27, 2014)

There is a Jace Stream tonight. I am going to preemptively add more stringent Cloudflare security measures.


----------



## Xarpho (Nov 30, 2014)

On Tuesday afternoon, I couldn't access Kiwi much at all, 404'd at a Cloudfare page most of the time. It seems to be better now.


----------



## ☻ (Nov 30, 2014)

Pikonic said:


> Who the fuck would pay that?


Monthly tugboat at work


----------



## Null (Dec 3, 2014)

The 5 minutes or so of downtime was due to an attack. This time, I believe it was on our TyceAndrews.com or the wiki


----------



## Watcher (Dec 3, 2014)

Since the attacks started, how much money has the attacker spent in total to try and take the forums down?


----------



## Null (Dec 3, 2014)

Cuddlebug said:


> Since the attacks started, how much money has the attacker spent in total to try and take the forums down?


No idea. It depends on what service they use. This was probably a 1 hour attack, so that 5 minutes of DT cost them anywhere between $4 and $15 (assuming they use a botnet and not a personal service).


----------



## Randall Fragg (Dec 3, 2014)

How many attacks have we had now? Just curious.


----------



## Null (Dec 3, 2014)

Randall Fragg said:


> How many attacks have we had now? Just curious.


Uh, 7 different instances of 1 ~ 4 hours each.


----------



## Randall Fragg (Dec 4, 2014)

Cuddlebug said:


> Since the attacks started, how much money has the attacker spent in total to try and take the forums down?


So, for the how much has this cost question, the most conservative estimate ($4 for 1 hour, all 7 instances have been one hour attacks) would be $28. (4x7).
Of course, given how I don't know the actual number of attack hours, or the rate paid, it's likely higher than that. 
The most he could have paid would be $420 (4x7 for the total hours, $15 dollars per hour). I find this unlikely.


----------



## MurcDusen (Dec 7, 2014)

Randall Fragg said:


> The most he could have paid would be *$420* (4x7 for the total hours, $15 dollars per hour). I find this unlikely.



No, that's totally sick and realistic!

Also, wonder whether there'll be attacks today again with the Jace/Tyce crossstreaming.


----------



## SU 390 (Dec 26, 2014)

Informative stuff. I still don't get who would want to crash this site.


----------



## CatParty (Dec 26, 2014)

WanderingVagabond said:


> Informative stuff. I still don't get who would want to crash this site.


----------



## SU 390 (Dec 26, 2014)

CatParty said:


>



Shamefully, I still haven't got around to see this movie yet. Personally I like Bane in the comics.


----------



## cypocraphy (Dec 26, 2014)

WanderingVagabond said:


> Informative stuff. I still don't get who would want to crash this site.



Whoever it is I'd like to shake their hand.


----------



## Flowers For Sonichu (Dec 28, 2014)

I'm getting server errors, is it my shitty internet, a DDoS, or the forums being overloaded due to Chris' arrest?


----------



## Van Darkholme (Dec 28, 2014)

Same here. 503's all around.


----------



## Null (Dec 28, 2014)

Please post Ray IDs. If it's cloud flare


----------



## Null (Jan 2, 2015)

same shit


----------



## KingofManga420 (Jan 2, 2015)

Null said:


> same shit


Doesn't TJ have anything else to spend his money &/or hard earned welfare on?


----------



## Null (Jan 2, 2015)

KingofManga420 said:


> Doesn't TJ have anything else to spend his money &/or hard earned welfare on?


It's become apparent that the mastermind TJ Church will stop at nothing to destroy the Kiwi Farms. We must be on guard &/or alert at all times.


----------



## Konstantinos (Jan 2, 2015)

I'm curious as to who would be putting this much effort into DDoSing the farm. I'd think your average ween would give up once he saw that it was costing hundreds of dollars to do this.


----------



## QueenMegan (Jan 2, 2015)

G-d bless lil orange!  We Kiwis will not be denied our farms!


----------



## Van Darkholme (Jan 3, 2015)

Trying to DDOS something is just a huge waste of time. The forum does feel a lot slower right now.
Maybe they're still using the ion cannon...


----------



## The I Scream Man (Jan 3, 2015)

TaterBot said:


> I got caught in a DDoS attack that lasted 3 days, p much around the clock. It could be worse. But you're better protected at this point.
> I'm surprised you're baffled that there are ppl who want to take you down. There are ppl who want you dead.


See this dog?
<---- that one.

That's Null's guard dog.

As you can see, we're an institution you can trust.  We take security very seriously.


----------



## Null (Jan 3, 2015)

TaterBot said:


> I got caught in a DDoS attack that lasted 3 days, p much around the clock. It could be worse. But you're better protected at this point.
> I'm surprised you're baffled that there are ppl who want to take you down. There are ppl who want you dead.


I never said I was surprised. Everyone else seems surprised.


----------



## Ruin (Jan 3, 2015)

Konstantinos said:


> I'm curious as to who would be putting this much effort into DDoSing the farm. I'd think your average ween would give up once he saw that it was costing hundreds of dollars to do this.



Most of the people we mock have  they don't give a shit about wasting other peoples money on pointless crap.


----------



## Null (Feb 11, 2015)

Another. I think this was kiwifarms.net being targeted, but it's hard to tell.

Edit: I'm full of shit. We didn't get a guest spike so this was the lolcow.wiki


----------



## ThatGuy (Feb 11, 2015)

Null said:


> Another. I think this was kiwifarms.net being targeted, but it's hard to tell.
> 
> Edit: I'm full of shit. We didn't get a guest spike so this was the lolcow.wiki


I blame  tjc.


----------



## Null (Feb 11, 2015)

I reviewed my logs and the IPs are coming from Venezuela and China, which confirms that this is probably a real paid-for DDoS coming from many infected computers in poorer countries. They still primarily use vulnerable issues of Windows XP Service Pack 0 because of how abundant cracked copies of the software are.



ThatGuy said:


> I blame  tjc.


Most definitely.


----------



## Jaimas (Feb 11, 2015)

CatParty said:


>


Bane makes everything better.

Interesting note, BTW: Brianna's been linking directly to Idlediletante's anti-/baph/ shit on Dkos.

I swear to christ, it's like they want to piss off that part of the internet that even veterans of the tubernet know better than to irritate.


----------

