Overview
Dating applications get sharp traffic spikes at different times of day and night all over the globe. So the goal of this project was a reliable back-end that scales automatically and stays highly available.
Auto scaling was the easy part, because AWS gives you everything you need to set up scaling policies and rules. The tricky part was protecting the platform from attacks, hackers and intrusion attempts. We did that by leaving a single entry point into the whole infrastructure and covering it with a firewall, traffic filtering and IP substitution.
That design created one problem, and we solved it with AWS Auto Scaling and Elastic Load Balancing.
Problem
If the single entry point, where the firewall, proxy and traffic filter run, goes down or gets stuck, the whole infrastructure becomes unavailable. One instance in front of everything is a single point of failure, and high availability is gone.
Solution
We put the entry-point instance into an EC2 Auto Scaling group with the minimum, maximum and desired capacity all set to 1, and placed an external load balancer in front of it.
The two services split the work:
- The load balancer checks the instance's health and only sends traffic to a healthy target. It doesn't launch or replace servers.
- Auto Scaling keeps exactly one healthy instance running. When the load balancer reports the instance as unhealthy, Auto Scaling replaces it with a new instance from the same launch template and registers the new one with the load balancer. Traffic then flows to the new instance.
Two details matter if you build this yourself:
- Turn on Elastic Load Balancing health checks for the Auto Scaling group. By default, Auto Scaling only uses EC2 status checks and ignores the load balancer's verdict. A frozen proxy on a running instance would then never be replaced.
- Recovery isn't instant. With one instance there's nothing to fail over to, so users wait for the replacement to boot and pass its health checks. That's usually a few minutes of downtime instead of hours, but it's not zero.
If you want to go deeper into high-volume traffic and how load balancers work, read our article on Load Balancers. The same Auto Scaling and load balancer pattern, with more than one instance, is the core of our scalable WordPress on AWS architecture.
Backup plan: a fresh environment from Docker images
You can't be 100% sure your back-end is protected from every kind of malware and hack.
We like to say: "The most protected server is the one that's turned off and lies underground under a six-foot layer of concrete." So if you want to be ready, you need a backup plan. Here's how we handled it in this project.
Running the stack in Docker containers makes it simple to bring the same technology stack up on a new server. The new server only needs Docker installed. So if the original infrastructure fails, we can switch to a brand-new one quickly.
We wrote automation scripts that call the API of the chosen server or cloud provider, create the server units we need and install Docker on them.
After that, we load the Docker images onto the servers, switch DNS, and we're live again.
Images only bring back the application. Two things decide how fast and how complete that recovery really is:
- DNS TTL. Resolvers keep serving the old address until the record's TTL expires. Keep the TTL on the records you plan to switch low, or the "switch DNS" step takes as long as your TTL.
- Data. Containers are disposable, the database and user uploads are not. The fresh environment is only useful if a recent copy of the data is already waiting in another location.
What we'd build today
The approach above worked for this project. If we designed the same platform on AWS today, we'd keep the ideas (one controlled entry point, self-healing, a ready second environment) and lean more on managed services:
- No single filtering instance. An Application Load Balancer runs in at least two Availability Zones, with AWS WAF attached for traffic filtering. AWS Shield Standard protects every AWS customer automatically against common network and transport layer DDoS attacks. If you need your own firewall appliances, Gateway Load Balancer spreads traffic across a fleet of them instead of one box.
- Containers on ECS or EKS. The ECS service scheduler replaces tasks that fail container or load balancer health checks, and it starts the replacement before stopping the unhealthy task when capacity allows. Kubernetes on EKS does the same with Deployments and probes.
- Images in Amazon ECR with cross-Region replication, so the recovery Region already has every image.
- A managed database (Amazon RDS or Aurora, Multi-AZ) with a cross-Region read replica or AWS Backup copies in another Region.
- Infrastructure as code. Terraform describes both environments, so the backup one comes from the same code instead of separate scripts. That's the kind of setup we build in DevOps automation projects.
- Route 53 failover routing with health checks. Route 53 answers with the secondary record when the primary is unhealthy, so the DNS switch doesn't wait for a person.
Not sure how your platform would survive a bad night? Talk to us and we'll check it before your users do.