This post was supposed to be a discussion of my migration back to Android from iOS but direction changed yesterday morning. At 9:30 am I found myself in the middle of a DDoS attack that was affecting the rack my server lives on. My sites slowed to a crawl and eventually went offline.
Fortunately, my hosting company (Hivelocity) was already aware of the situation and I was advised that I would be back online in 15 minutes or so. That wasn’t the case and after a brief phone call I found out that the attack was specifically targeting my server and they had me offline until the attack stopped. FUUUUUUUUUUUU
Generally speaking, my sites take quite a bit of flak from the rest of the Internet and we have been “raided” on at least a half dozen of times that I’m aware of. Never has our infrastructure been attacked directly like this, aside from some script kiddies scanning for exploits, and because of that, I had nothing in place to mitigate the issue.
After a few hours of downtime, which I am thankful was during a Saturday morning, not really peak usage for us, I started to consider being down at someone else’s mercy as a non-option and updated the DNS on my domains to point to a secondary IP address. I realized that Linode’s DNS manager defaults to 1 day for the TTL, I’ve since bumped that down to 1 hour to help us shift faster next time.
That’s lesson number one from all of this, set your DNS TTL to something reasonable in the off chance that you have to make a change that will impact whether or not your site is accessible. CloudFlare keeps their TTL at 5 minutes and you can too, but I find that to be a little too short and could result in way too many DNS lookups due to the frequent expirations. Personally, I’ve never run into any issues with it set that low, so I suppose it’s a YMMV or perceived issue.
Okay, so at this point I have about 50-60 users online thanks to the DNS change. I started to research DDoS mitigation solutions and actually stumbled upon the fact that Hivelocity does offer DDoS Protection for a nominal fee per month ($39 for 48 hours of protection in the event of attack, with overage fees). This actually never came up during the sales process to set up the server but it’s something I’ve since added and will continue to pay for indefinitely.
As soon as the engineers put the appliance in place to handle the traffic, my sites came back online and we’ve been good ever since. This all begged the question, why did no one mention this to me when I was repeatedly contacting them about the situation. Obviously for 40$ a month to get my sites back online would be worth it. Hell, I lost that much in revenue during the downtime. I suppose that’s the difference between an engineer and a sales guy 😉
All in all, I felt the situation was handled very well by both Hivelocity and myself (I didn’t even curse anyone out!!!) but it could have been resolved faster knowing about the DDoS Protection add-on. Now that I’ve been through this, I definitely know how to approach it in the future. Now the question is, who would have attacked my server and why? No demands have been made but a few folks have attempted to stake claim for the attack. I’ve been thanking everyone that’s come forward for exposing a hole in our infrastructure and helping us improve, but have had to take it all with a grain of salt considering multiple parties are attempting to claim responsibility.
Please keep in mind that the attack was network related and my server was not compromised. If you’ve ever been through a DDoS attack, I’d love to hear your stories, comment below!
Also, if you’re interested in Hivelocity Hosting (now with a fresh new logo and site!) please use my referral link. Don’t forget to add the DDoS Protection add-on during checkout 😉
