This week, request-log-analyzer obtained its 100th watcher on GitHub!

Bart and I have worked hard to make r-l-a a useful product for many people in various situations. The fact that more than 100 people are following the project’s progress and that at this moment, the gem has been download almost 200 times, shows that we are somewhat successful in this regard. Numbers like these, in combination with the e-mail messages we have received, motivate us to keep spending time on the project and keep improving it, even if these improvements are not directly useful for our own projects. It has grown beyond scratching our own itch. 

On a related note, my Flickr photostream recently welcomed its 10,000th visitor.

What started as a convenient utility to backup and share my holiday photos with my family and friends, now has become somewhat of a showcase of what I am about and what I am up to. I would not consider myself a “professional” photographer and I am not an active member of the Flickr community, but still I get feedback on my photographs by visitors, because their visits increase the view counters of my photos.

These counters have really motivated me to make more of an effort when I put photos on Flickr. I started by adding titles, descriptions and tags, so that my photos are easier to find. I also became much more critical of the pictures I upload to Flickr: new uploads have to add something significant to my collection. Analyzing why some pictures got more attention than others made me a better photographer, although there still is a lot of room for improvement.  

Ignite the lazyweb, kick-start a quality improvement loop

What interests me in these examples is that simple counters like watchers on GitHub or views on Flickr can be valuable feedback and can motivate people to put in effort. The end result is quality improvement: write better software, make better pictures, etc.. Even an inherently lazy person like me can get motivated to keep putting in effort and to keep improving myself, because of such a simple feedback loop!

Additionally, it creates a dependence on the website in question. I look at my Flickr stats page every day, and I am subscribed to my activity feed on GitHub to get notified when new people start watching my projects. I now simply have to publish tools I write on GitHub to boost its quality, immediately and in the long run. And I have to upload pictures to Flickr as it is vital for my photography learning curve.

I guess I finally figured out what Web 2.0 is all about! Can we use a similar technique on Floorplanner to boost the quality of the designs our visitors make?

Gert-Jan is on the left side and Jaap on the right.

Nico is on the left side and Willem on the right.

There is a lot of hype these day about being in “the cloud”. However, “the cloud” seems to mean a lot of different things. Tim O’Reilly sees three types of cloud computing.  His first type, utility computing, is the type I’m talking about here. 

Utility computing. Amazon’s success in providing virtual machine instances, storage, and computation at pay-as-you-go utility pricing was the breakthrough in this category, and now everyone wants to play. Developers, not end-users, are the target of this kind of cloud computing.

Launched in July 2002, Amazon Web Services provide online services for other web sites or client-side applications. Each individual product is interesting by itself, but all the services together made it just the solution we were looking for. In this and coming posts I will talk about our experiences with Amazon’s version of “the cloud”. The first service we started to use was Elastic Cloud Computing (EC2).

Each day more than 2000 users from all over the world register at Floorplanner.com. At this moment we have over 800.000 registered users in our database. A lot of these users visit the site on a regular basis, which generates a lot of traffic. To meet this demand, we had a couple of dedicated servers running. When the demand rised above a certain level, we’d add a new one to spread the load. As logical as it might sound, it is far from ideal.

It takes a lot of time to install, configure and maintain a server. Precious time you can’t spend on the development of your product. It doesn’t scale fast. For example: when you are featured on a big site that generates a lot of additional traffic, your servers will probably have a very hard time (or crash) and there is nothing you can do about it. Buying additional servers takes too much time. Another thing about buying servers is that it’s an investment you have to finance up front. You have to pay the full price for it, but in the beginning you’ll only use a small part of it’s capacities. Once you’ve spend the money on a server, you can’t spend it on anything else. And maybe the biggest issue we had with dedicated servers was that it is hardware, and hardware gets old very fast. After a while you’re stuck with an outdated server, that you (a) won’t replace because “it’s still doing something” or (b) costs time (= money) to migrate to another server. This means a higher risk for failure, or just more work.   

Now with the EC2 service life has gotten better. Sure, we still had to install everything on the virtual instance, but only the first time. Once you’ve done this, you can add additional instances with a click of a button (based on an image of the first instance). When traffic spikes you can add instances within minutes instead of days. The pay-as-you-go model keeps us from financing every server up front. And because it’s a virtual server, we don’t have to think about the hardware anymore. Just deploy it and get on with the fun stuff!

I would like to thank you all for helping us build our thumbnail database!
I presume this statement might be in need of some clarification, so bear with me when I go into the technical details on this one.

For every design that is saved on Floorplanner, we create a thumbnail in JPEG format. We use these thumbnails for the gallery, and now we have included them on everyone’s dashboard. However, for various reasons, we do not have a thumbnail available for every design. However, with your help, we are improving the thumbnail database while you our browsing the site!

The thumbnail images are stored on Amazon AWS S3. We know the URL that a thumbnail of a design should have, but we do not know if it actually is available. In the latter case, the result is a nasty image not found placeholder on the dashboard. This of course is not acceptable! We cannot know if a thumbnail exists other than doing a request to the URL and see whether we get an image back from Amazon, or a HTTP 404 status. This is a very time-consuming procedure to run server side so we chose to find a client-side solution.

We found that javascript can be used to check if an image exist. An AJAX call cannot be used, because cross-site calls are not supported. However, the javascript Image object can be used for this purpose.

var img = new Image();
img.onload = function(event) {
  // image was found and loaded successfully
  document.getElementById('img-tag').src = img.src;
};
img.onerror = function(event) {
  // An error occured while loading the image
  document.getElementById('img-tag').src = '/images/thumb-unavailable.jpg';
}
 
// Setting the src property will trigger the events.
img.src = 'http://link.to.amazon.s3/design/thumbnail.jpg';

A nice thumbnail not available image will be shown if the thumbnail file cannot be found on S3. This is much nicer, and the check is completely done client-side! However, we found a way this could even be better by changing the onerror event handler. Instead of displaying a thumbnail not available image, we can simply load a small instance of the Floorplanner application to display a small version of the design. Moreover, we can instruct it to generate a thumbnail JPEG and save it to S3!

So, every time you see a small Floorplanner loading on your dashboard, you are creating a missing thumbnail. Next time, the thumbnail will be available on S3 and there will be no need to start the Floorplanner application. A nice example of distributed computing, mixed with a hint of SETI@home. I like it!