Last week seemed like an ordinary week. As ordinary as it gets while traveling the world and bootstrapping a startup.
Until we started getting some strange emails in our inbox:
What’s this about? I plugged it into Google translate out of curiosity. It’s an automated response. The recipient is on vacation.
Cool, someone in China is trying Canny out! They invited a teammate to their team, and they’re on vacation. No big deal.
But then the next day I woke up to a dozen more emails like this:
Okay, what’s going on? If this many people are sending automated responses, how many emails are they sending?!
I did a quick database query to count teammate invites. My heart sank. They had sent 80k over the past week. Holy heck. We’re under attack. They’re programmatically sending thousands of HTTP requests which are resulting in us sending emails.
But why? Why would someone want to invite random people to their team on Canny? What’s in it for them? Are they just having fun? Ugh.
That’s what I thought at first. Then I took a look at one of the emails they sent:
That’s a weird company name. That doesn’t look legit at all.
I shared it with a teammate who can read Chinese. He instantly identified it as spam. They’re telling people to join a QQ group to get discounts on online gambling.
They’re using Canny to send people email spam. We pay to send their emails. We take the deliverability hit when people mark our emails as spam. Oh hell no, I don’t think so!
Stopping the Attack
We took a few immediate steps to stop the attack.
First, we deleted their Canny account. This way their requests would fail instead of triggering emails. We found a small handful of user and company accounts associated with this bad behavior. We removed them all.
Next, we blacklisted their IP address. They were sending every HTTP request from the same IP address. We simply added it to our blacklist in AWS WAF (Web Application Firewall). Their requests could no longer hit our servers.
At this point, they would have to create a new account, and use a different IP address to start the attack again. This isn’t too hard, but buys us time to think about how to avoid this attack in the long term.
Finding an 80/20 Solution
We’re an early-stage startup. We don’t have a team of security engineers to devote weeks to this. Every minute spent protecting ourselves is a minute that could have been spent on growth.
That’s why it was important that we find an 80/20 solution. Something good enough to make Canny an undesirable target, but nothing more. No over-engineering.
We put ourselves in the shoes of the attacker. We realized that picking which SaaS you want to send your email spam is a lot like picking which bike to steal.
Nicer bikes are more valuable, but have more protection. The thief must optimize for the nicest bike with the least amount of protection.
As long as your bike has at least as much protection as other similarly valued bikes, it won’t get stolen. This is because there are easier targets – the thief can make more money with less effort.
This makes it clear why early-stage SaaS apps are a target for email spammers: they’re an easier target. They have fewer resources to invest into security than big companies.
So how do we make our SaaS harder to attack?
1. Block Obvious Attacks
Some behavior is obviously malicious. For example, someone sending requests faster than humanly possible. This can easily be blocked using rate-based rules in AWS WAF.
But you have to be careful. You don’t want to block a paying customer from doing something legitimate. You have to be thoughtful about what you can and can’t block.
We made a list of obviously bad behavior and blocked it all via either AWS WAF or our server application.
2. Better Awareness
The attack went on for several days before we even realized we were under attack. In that time they significantly ramped up volume. They probably thought we hadn’t noticed, so they could send more. They were right.
If we can immediately realize when we’re under attack, and act, this minimizes attacks. It also shows attackers we’re on to them. They’ll probably move on to another target then and there.
We created a new #fraud thread in Twist. We made a list of suspicious events, like a user sending more than 10 team invites in a day. Whenever this happens, the server pings our #fraud thread.
This behavior could be legit: maybe someone is inviting their entire team. That’s why we chose to send notifications for these events instead of outright blocking them. They warrant a quick manual review, but aren’t necessarily bad.
I hope you enjoyed our story. It was definitely unexpected and stressful, but we came out of it smarter and stronger.
If you work at an early-stage SaaS company, or someday want to, here’s a quick list of the most important takeaways:
- As an early-stage SaaS company, you are a target. You have less security than your larger peers. Think especially hard if you have a free tier/trial and send emails that contain user generated content.
- Awareness is crucial – know when you’re being attacked. Think about how people might attack you, and what would make you aware that it’s happening. It’ll probably only take a few minutes.
- Don’t over-engineer. It’s easy to dream up the perfect solution to every fraud problem you’ll ever have. Don’t. Your time is valuable. Find a solution that delivers 80% of the value in 20% of the time.
Thanks for reading. If you’d like to share some security best practices, leave a comment below!