There was a new challenge ahead. Moving over 10 million images from a single Linux machine to the Cloud.
It was quite a daunting task to move so many files from a running production server to the cloud. Before I will start digging into the technical part I would first like to explain why it was necessary.
The single Linux server for media storage is, of course, a small practice. It’s a single point of failure, plus we needed to add disk space ourselves as the data set kept growing. We also needed to keep the old data because it was used on the website.
Before the start of the project, there was about 1.5 TB of images stored on this server. Mostly Roomstyler 3D images. The thumbnails for these images were created on the fly and cached on the same machine, but they were stored on a separate disk.
The main problem, next to upload all those files to the Cloud, was generating thumbnails for all these images. S3 does not support on the fly thumbnailing and we didn’t want a extra server running only for generating thumbs. We decided we needed to upload all the images to S3 plus generate thumbnails for them.
For most images, we decided we needed two different size thumbnails, which would mean we needed to store over 30 million files in the Cloud.
Continue reading “The great image migration” →