Floorplanner Tech Blog

This is the third post in a series about how we scaled our product at Floorplanner. The first one was about how it all started with one dedicated server. In the second post I talked about how we reduced some of the risk by running two application servers on separate machines in the cloud.

This post is about our database. The application servers and the load balancer were running in the AWS cloud, but the MySQL database was still on the dedicated server outside of the AWS cloud. Not really ideal. It had to be in the AWS cloud, but we wanted a bit more.

Now that our app servers were running very smooth, our new biggest risk was our database. I can’t remember if we made backups at all back in the days, but if we were, the backups were probably stored on the same disk anyway (because we only had one). We needed a proper backup system too.

Percona Server

At that time, AWS didn’t had the Relational Database Service (or it was very expensive …), so we had to do it ourselves. We got a very good tip from Valery Visnakovs from Ask.fm about using the Percona MySQL Server for our setup. Impressed by the Percona Server and their MySQL Performance Blog we choose to take it for a ride.

At the same time, we got some other good advice. Cody Faustner from Shopify told me “The very worst part of AWS, is disk I/O, so whatever you can do to minimize disk activity will go a long way.” Related, and even more important was his tip to have a server with lots of memory, enough for MySQL to load the whole working set into memory. More info about that on the MySQL Perfomance Blog.

To meet the memory demand we picked a High-Memory Extra Large Instance and installed Percona Server on it.

Backup strategy

With the new database server in place we had to think about our backup strategy. We started out with the idea to backup our database on a daily basis. But Floorplanner was being used by more and more people every day and it’d be a disaster if we’d loose a whole day of data. So we needed a better way to backup our data, a real time way. That meant replicating our data to another (mirror) server.

We created the first database server in the US east region, the same place as all the other servers. To protect our data, we didn’t want our database backup to be stored in the same region. Therefor we created a second database server in the EU west region and replicated our data there. I’ll write a short post soon about the steps we took to setup this replication.

When for some reason our main database would loose or cripple data, these changes would directly be synced to our backup database, removing important data there too. So we figured, we still needed a daily backup to reduce this risk. Amazon’s Elastic Block Store (EBS) has the perfect feature for this, snapshots. With it you can easily create a snapshot of a whole EBS block. Therefor we used and instance with EBS for the backup database server. Don’t forget to clean up your snapshots after a while, say a couple of days, otherwise your AWS bill will grow.

See below a new scheme including the database changes.