Tech Blog

This week, request-log-analyzer obtained its 100th watcher on GitHub!

Bart and I have worked hard to make r-l-a a useful product for many people in various situations. The fact that more than 100 people are following the project’s progress and that at this moment, the gem has been download almost 200 times, shows that we are somewhat successful in this regard. Numbers like these, in combination with the e-mail messages we have received, motivate us to keep spending time on the project and keep improving it, even if these improvements are not directly useful for our own projects. It has grown beyond scratching our own itch. 

On a related note, my Flickr photostream recently welcomed its 10,000th visitor.

What started as a convenient utility to backup and share my holiday photos with my family and friends, now has become somewhat of a showcase of what I am about and what I am up to. I would not consider myself a “professional” photographer and I am not an active member of the Flickr community, but still I get feedback on my photographs by visitors, because their visits increase the view counters of my photos.

These counters have really motivated me to make more of an effort when I put photos on Flickr. I started by adding titles, descriptions and tags, so that my photos are easier to find. I also became much more critical of the pictures I upload to Flickr: new uploads have to add something significant to my collection. Analyzing why some pictures got more attention than others made me a better photographer, although there still is a lot of room for improvement.  

Ignite the lazyweb, kick-start a quality improvement loop

What interests me in these examples is that simple counters like watchers on GitHub or views on Flickr can be valuable feedback and can motivate people to put in effort. The end result is quality improvement: write better software, make better pictures, etc.. Even an inherently lazy person like me can get motivated to keep putting in effort and to keep improving myself, because of such a simple feedback loop!

Additionally, it creates a dependence on the website in question. I look at my Flickr stats page every day, and I am subscribed to my activity feed on GitHub to get notified when new people start watching my projects. I now simply have to publish tools I write on GitHub to boost its quality, immediately and in the long run. And I have to upload pictures to Flickr as it is vital for my photography learning curve.

I guess I finally figured out what Web 2.0 is all about! Can we use a similar technique on Floorplanner to boost the quality of the designs our visitors make?

We have been working hard to implement an XML-based REST-style API for Floorplanner and some of our partners are using it already to access their users and plans. We now have started to use this API ourselves.

The Floorplanner Flash application communicates with our servers to load and save projects and designs. The backend for this functionality used to be written in PHP. Eliminating this PHP application simplifies our server setup, eases development and reduces our maintenance burden. To rewrite this functionality in Rails, we decided to eat our own dogfood and use the REST API to load and save designs. Why reinvent the wheel?

However, while implementing the changes it the Flash application, we found that it did not work out of the box, because of some limitations in ActionScript. Note that we still use ActionScript 2, so some of these issues may not be relevant for ActionScript 3. In this post, we will detail what these issues are and how we overcame them.

Using a separate format

Our REST API uses the XML format supplied by Rails. Because we had to make some changes to make it work from ActionScript, we decided to add a distinct format that we could implement differently without altering the behavior of the default XML API. Adding a new format, called “flash” can be done in Rails by adding a MIME alias to your environment:

Mime::Type.register_alias "application/xml", :flash

Now we can send different responses based on this format:

respond_to do |format|
  format.xml   { ... } # default API behavior
  format.flash { ... } # do something different
end

HTTP status codes

REST APIs use HTTP status codes to return whether a call succeeded, and if not, why. Flash however uses the browser to perform HTTP requests. The browser only returns something to Flash if the request was successful. So, if an error code is used if a request failed together with some error messages, these error messages will not be delivered to Flash and cannot be displayed to the user. We resolved this by always sending the HTTP OK status.

respond_to do |format|
  format.xml   { render :xml => @project.errors, :status => 422 }
  format.flash { render :xml => XML.failure(@project.errors, 422) }
end

Our XML.failure method will return something like:

<failure status="422">
  <error on="name">A project should have a name!</error>
</failure>

Note that other HTTP success statuses than 200 work in Safari and Firefox, but not in Internet Explorer. So, never return a 201 (:created), because Internet Explorer will not send the result to Flash!

PUT and DELETE requests

REST-style APIs use HTTP PUT requests to alter objects and DELETE requests to destroy objects. Most browsers do not support these request type. It is not supported by ActionScript either, because Flash uses the browser to send the request.

To overcome this problem, these types of requests can be simulated in Rails by sending a _method parameter along with a POST request. Unfortunately, this does not work when calling the REST API. The POST request body cannot be used to send additional variables, because it is used for the XML payload.

We solved this issue by creating additional routes for POST requests to the update and destroy actions of our resource controllers. Our routes.rb file now looks like this:

  map.resources :projects, :member => { :update => :post, :destroy => :post }

These routes route to exactly the same methods as the default REST actions (ProjectsController#update and ProjectsController#destroy), so no additional code is needed. The following calls are now equivalent:

PUT    /projects/123.xml
POST   /projects/123/update.xml
POST   /projects/123/update.flash

DELETE /projects/123.xml
POST   /projects/123/destroy.xml
POST   /projects/123/destroy.flash

The result

It requires some stretching of the pure REST principles, but doing so is worth it: we can now reuse the code we use for our API to handle Flash application calls and we can eliminate the PHP backend.

After a complete rewrite, Bart and I are proud to present request-log-analyzer version 1.0! Request-log-analyzer is an open-source command-line tool to analyze production log files from your Rails application to produce a performance report.

What’s new?

• More robust log parser. It parses more lines and it now combines all lines that belong to the same request, which greatly improves the amount of information available. 
• It produces more detailed and more beautiful reports. 
• A database builder is included, which will create an SQLite 3 database with all parsed request information, so you can roll your own queries.
• Request filtering options, so you can exclude irrelevant data. An example on how this can be applied in practice can be found in the wiki. 
• Better, more modularized design under the hood. The parser is now fully log file format-agnostic. Developing extensions and modifications, or adding support for other log file formats should be much easier now. See the development-page for some pointers.
• Documentation in the project’s wiki. Hopefully, this helps people get up to speed with the new version and answers most questions about using the tool. If you still have questions, please contact us so we can keep improving it!

Installation

Install or upgrade to the new version with the following command:

$ sudo gem install wvanbergen-request-log-analyzer 
                --source http://gems.github.com

To get the best results out of request-log-analyzer, it is important to configure logging correctly for your application. Some pointers on how to set things up correctly can be found in the wiki.

Good luck to the merged Rails team and hopefully Rails 3 will kick ass! Let’s hope git will really deliver on this gig! Try to refrain from using git blame too much when resolving merge conflicts.

My personal domain vanbergen.org is almost 10 years old. Besides the accompanying e-mail address, I’ve never used it extensively, but its one of the oldest pieces of my “legacy” that can be found on the Internet and it is the starting point of my online identity.

Lately, major internet companies have embraced the OpenID standard to allow other sites to login with the login credentials of these major sites, basically running an OpenID server. (Curiously, they do not support consuming OpenIDs themselves, so I still have to trust them with my password. What’s up with that?) I can use most of the sites I have an account on as OpenID identity. For example, my Flickr page is an OpenID identity URL. With more and more sites supporting consuming OpenIDs (including Floorplanner!), I will not have to remember all those passwords and the internet will become a better place because of it.

It would be nice to use my 10 year old domain name as my OpenID identity. This is actually very easy because of delegation. I do not have to run my own OpenID server: I can just add a header to my site that points to the OpenID server that I will be using. I have chosen to use my Google Account as OpenID server. To accomplish this, I simply added the following tag to the <head>-section of the vanbergen.org site:

<link rel="openid2.provider" 
    href="https://www.google.com/accounts/o8/ud" />

Now, I can simply enter “vanbergen.org” when my OpenID identity URL is requested on a site. How neat is that? That site will check my personal site for this tag and will redirect me to the Google Account page, on which I can confirm that I want to login to that site. Note: the site has to support OpenID 2.0, because Google does not offer an OpenID 1.0 provider. Luckily, most sites do.

I do not know what happens to the sites I have registered on with OpenID if I decide to change my OpenID provider and I change the value on vanbergen.org accordingly. Does anybody know if my existing OpenID accounts will keep working in this case?

Because Jaap finally convinced Gert-Jan, we have moved to the Git version control system for the main Floorplanner repository. Now we can use branches for different functionality far more easily. As an easy reminder to some common Git tasks I will need regularly, I have written down some Git recipes. This is basically meant for me and my fellow developers, but maybe it can help you as well. Suggestions to improve my workflow are welcome!

Displaying the current branch in your prompt

Because I will be using branches more regularly now, it is nice to know what branch I am currently working in. git branch will provide this information, but it can be more direct by including the current branch in your terminal prompt.

To display the current git branch in my terminal prompt, I have added the following to my Bash configuration file ~/.profile:

parse_git_branch() {
  git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ \[\1\]/'
}
 
PS1='\[\033[01;37m\]\w\[\033[00;35m\]$(parse_git_branch)\[\033[00m\] \$ '

I use a black, semi-transparent terminal with white text, and pink branch names. Change the colors to your own liking!

Working on a remote branch that is not available locally yet

If you want to help out on a branch that somebody else started and has pushed it to the remote repository, you can checkout this branch and make it “track” the remote branch.

$ git checkout -b newbranch origin/newbranch

In this example, origin/newbranch is the branch in the remote repository. Locally, this branch will be called newbranch. A simple git pull will update my branch with the latest changes from the remote branch later on, git push will send my changes to the remote server.

Creating a new branch and pushing it to the remote server

Sometimes I want to start a new branch myself if I want to work on a new feature or on disruptive refactoring of the main codebase.

$ git checkout -b feature

The feature branch is now available locally. After some time, I want to share my current changes in this branch with other developers. I should make the branch available in the remote repository so other can access it like I described above.

$ git commit -m "Added initial version of %feature%"
$ git push origin feature

The feature branch will now be available to other developers as well. Note that the local feature-branch is not tracking the remote branch of the same name. This can be enabled by changing the configuration of the repository.

Merging a branch

After work is completed on my feature-branch and it is tested thoroughly, I want to merge the branch with the master branch of the project. To make sure the merge with the master branch is seamless and all possible merge conflicts are handled beforehand, we first run git rebase. This ensures that the changes in the feature branch are relative to the latest commit to the master branch and can therefore be applied by a “fast forward”. It is best to run git rebase from time time to time while you are developing in the feature branch to make sure your work does not divert to much from the main development in the master branch.

  # make sure that the current branch is seamlessly
  # mergeable with the master branch
$ git rebase master
  # switch to the master branch
$ git checkout master
  # now, merge the feature branch
$ git merge feature
  # publish the merge to the remote server
$ git push origin master

I can now delete the local and remote feature branch, as the changes have been incorporated in the master branch:

$ git branch -d feature
$ git branch -d -r origin/feature

I just released version 0.2.0 of request-log-analyzer, our tool to analyze request log files that are generated by Rails and Merb for performance tweaking. This new version supports the new log format of Rails 2.2, which has changed slightly.

An updated gem should be available any minute now. Run sudo gem update to upgrade the newest version.

I would like to thank you all for helping us build our thumbnail database!
I presume this statement might be in need of some clarification, so bear with me when I go into the technical details on this one.

For every design that is saved on Floorplanner, we create a thumbnail in JPEG format. We use these thumbnails for the gallery, and now we have included them on everyone’s dashboard. However, for various reasons, we do not have a thumbnail available for every design. However, with your help, we are improving the thumbnail database while you our browsing the site!

The thumbnail images are stored on Amazon AWS S3. We know the URL that a thumbnail of a design should have, but we do not know if it actually is available. In the latter case, the result is a nasty image not found placeholder on the dashboard. This of course is not acceptable! We cannot know if a thumbnail exists other than doing a request to the URL and see whether we get an image back from Amazon, or a HTTP 404 status. This is a very time-consuming procedure to run server side so we chose to find a client-side solution.

We found that javascript can be used to check if an image exist. An AJAX call cannot be used, because cross-site calls are not supported. However, the javascript Image object can be used for this purpose.

var img = new Image();
img.onload = function(event) {
  // image was found and loaded successfully
  document.getElementById('img-tag').src = img.src;
};
img.onerror = function(event) {
  // An error occured while loading the image
  document.getElementById('img-tag').src = '/images/thumb-unavailable.jpg';
}
 
// Setting the src property will trigger the events.
img.src = 'http://link.to.amazon.s3/design/thumbnail.jpg';

A nice thumbnail not available image will be shown if the thumbnail file cannot be found on S3. This is much nicer, and the check is completely done client-side! However, we found a way this could even be better by changing the onerror event handler. Instead of displaying a thumbnail not available image, we can simply load a small instance of the Floorplanner application to display a small version of the design. Moreover, we can instruct it to generate a thumbnail JPEG and save it to S3!

So, every time you see a small Floorplanner loading on your dashboard, you are creating a missing thumbnail. Next time, the thumbnail will be available on S3 and there will be no need to start the Floorplanner application. A nice example of distributed computing, mixed with a hint of SETI@home. I like it!

Some time ago, I wrote about putting HTTP status code to use for your Rails application. For my reinvigorated project, I wanted to apply the same technique. Instead of re-implementing it once again, I created a Rails plugin called http_status_exceptions to easily add this functionality and I have put it on Github. For more information on how to install and use the plugin, see the project’s wiki.

I just started working on an old Rails project after having neglected it for 15 months. Most of the view files still had the good old .rhtml extension. I was too lazy to rename these files by hand, both on my file system and in the git repository. I used the following Bash commands to do the job:

First, I renamed all the partials to the .erb extension. Note: I am not using .html.erb, as some of these partials are used in js-formatted responses as well:

for i in `find app/views/**/_*.rhtml`; do \
  git mv $i `echo $i | sed s/\\.rhtml$/.erb/`; \
done

The remaining files could now be renamed to .html.erb with a similar command:

for i in `find app/views/**/*.rhtml`; do \
  git mv $i `echo $i | sed s/\\.rhtml$/.html.erb/`; \
done

Note that this technique works with Subversion as well: just substitute git with svn in the command above. A regular rename is possible as well by leaving out git altogether!

Now my file names are Rails-compliant again, I can start refactoring all the code that is not up to current Rails standards anymore. Ah, the virtues of developing with a rapidly evolving framework…