Floorplanner Tech Blog

After an almost complete rewrite, I am proud to present version 2.0 of scoped_search, the ActiveRecord plugin that makes it easy to find records using a simple query language. This new version support a new query language that supports more complex constructs, and can therefore be used to conduct more fine-grained queries on your models.

New query language

• Logical operators: AND (&, &&), OR (|, ||) and NOT (!, -) operators, and parentheses to structure the boolean logic: police AND (car || uniform), -"village people". By default, the AND operator is used to combine different segments of your query.
• Comparison operators: the most common comparison operators are supported, and to what you expect on integer and date field.
• Explicit field support: only search in the specified field instead of all fields: age >= 21, created < 2009-01-01, username != "root".
• Check for NULL fields: null? parent, set? error_message
• Commas are supported to separate the different parts of the query.

More information about the query language can be found in the project wiki on GitHub.

New definition syntax

The new version supports a new syntax to define what fields of your model can be searched and in what cases. An example:

class User < ActiveRecord::Base   
  belongs_to :account_type
 
  scoped_search :on => [:first_name, :last_name]
  scoped_search :on => :created_at, :alias => :created, :only_explicit => true
  scoped_search :in => :account_type, :on => [:name, :description]
end

After the fields have been defined, the search_for method can be used to search your models using a named scope, just like it was before. The project wiki has more information about this new syntax. The search syntax itself hasn’t changed:

@users = User.search_for(params[:q]).paginate(page => params[:page])

Installation or upgrade

Include the gem in your environment.rb configuration and run rake gems:install to install it:

config.gem 'scoped_search', :source => 'http://gemcutter.org'

Backwards compatibility

The new version has a new syntax to define the fields that can be searched with a query. This new syntax gives you more fine-grained control over the queries that will be generated, so I urge you to adopt this new syntax. However, the old searchable_on syntax is still available for backwards compatibility.

Please contact me if you have any issues with the new version.

(Updated with new gemcutter installation instructions.)

We have been working hard to implement an XML-based REST-style API for Floorplanner and some of our partners are using it already to access their users and plans. We now have started to use this API ourselves.

The Floorplanner Flash application communicates with our servers to load and save projects and designs. The backend for this functionality used to be written in PHP. Eliminating this PHP application simplifies our server setup, eases development and reduces our maintenance burden. To rewrite this functionality in Rails, we decided to eat our own dogfood and use the REST API to load and save designs. Why reinvent the wheel?

However, while implementing the changes it the Flash application, we found that it did not work out of the box, because of some limitations in ActionScript. Note that we still use ActionScript 2, so some of these issues may not be relevant for ActionScript 3. In this post, we will detail what these issues are and how we overcame them.

Using a separate format

Our REST API uses the XML format supplied by Rails. Because we had to make some changes to make it work from ActionScript, we decided to add a distinct format that we could implement differently without altering the behavior of the default XML API. Adding a new format, called “flash” can be done in Rails by adding a MIME alias to your environment:

Mime::Type.register_alias "application/xml", :flash

Now we can send different responses based on this format:

respond_to do |format|
  format.xml   { ... } # default API behavior
  format.flash { ... } # do something different
end

HTTP status codes

REST APIs use HTTP status codes to return whether a call succeeded, and if not, why. Flash however uses the browser to perform HTTP requests. The browser only returns something to Flash if the request was successful. So, if an error code is used if a request failed together with some error messages, these error messages will not be delivered to Flash and cannot be displayed to the user. We resolved this by always sending the HTTP OK status.

respond_to do |format|
  format.xml   { render :xml => @project.errors, :status => 422 }
  format.flash { render :xml => XML.failure(@project.errors, 422) }
end

Our XML.failure method will return something like:

<failure status="422">
  <error on="name">A project should have a name!</error>
</failure>

Note that other HTTP success statuses than 200 work in Safari and Firefox, but not in Internet Explorer. So, never return a 201 (:created), because Internet Explorer will not send the result to Flash!

PUT and DELETE requests

REST-style APIs use HTTP PUT requests to alter objects and DELETE requests to destroy objects. Most browsers do not support these request type. It is not supported by ActionScript either, because Flash uses the browser to send the request.

To overcome this problem, these types of requests can be simulated in Rails by sending a _method parameter along with a POST request. Unfortunately, this does not work when calling the REST API. The POST request body cannot be used to send additional variables, because it is used for the XML payload.

We solved this issue by creating additional routes for POST requests to the update and destroy actions of our resource controllers. Our routes.rb file now looks like this:

  map.resources :projects, :member => { :update => :post, :destroy => :post }

These routes route to exactly the same methods as the default REST actions (ProjectsController#update and ProjectsController#destroy), so no additional code is needed. The following calls are now equivalent:

PUT    /projects/123.xml
POST   /projects/123/update.xml
POST   /projects/123/update.flash

DELETE /projects/123.xml
POST   /projects/123/destroy.xml
POST   /projects/123/destroy.flash

The result

It requires some stretching of the pure REST principles, but doing so is worth it: we can now reuse the code we use for our API to handle Flash application calls and we can eliminate the PHP backend.

After a complete rewrite, Bart and I are proud to present request-log-analyzer version 1.0! Request-log-analyzer is an open-source command-line tool to analyze production log files from your Rails application to produce a performance report.

What’s new?

• More robust log parser. It parses more lines and it now combines all lines that belong to the same request, which greatly improves the amount of information available. 
• It produces more detailed and more beautiful reports. 
• A database builder is included, which will create an SQLite 3 database with all parsed request information, so you can roll your own queries.
• Request filtering options, so you can exclude irrelevant data. An example on how this can be applied in practice can be found in the wiki. 
• Better, more modularized design under the hood. The parser is now fully log file format-agnostic. Developing extensions and modifications, or adding support for other log file formats should be much easier now. See the development-page for some pointers.
• Documentation in the project’s wiki. Hopefully, this helps people get up to speed with the new version and answers most questions about using the tool. If you still have questions, please contact us so we can keep improving it!

Installation

Install or upgrade to the new version with the following command:

$ sudo gem install wvanbergen-request-log-analyzer 
                --source http://gems.github.com

To get the best results out of request-log-analyzer, it is important to configure logging correctly for your application. Some pointers on how to set things up correctly can be found in the wiki.

Good luck to the merged Rails team and hopefully Rails 3 will kick ass! Let’s hope git will really deliver on this gig! Try to refrain from using git blame too much when resolving merge conflicts.

I just released version 0.2.0 of request-log-analyzer, our tool to analyze request log files that are generated by Rails and Merb for performance tweaking. This new version supports the new log format of Rails 2.2, which has changed slightly.

An updated gem should be available any minute now. Run sudo gem update to upgrade the newest version.

Some time ago, I wrote about putting HTTP status code to use for your Rails application. For my reinvigorated project, I wanted to apply the same technique. Instead of re-implementing it once again, I created a Rails plugin called http_status_exceptions to easily add this functionality and I have put it on Github. For more information on how to install and use the plugin, see the project’s wiki.

I just started working on an old Rails project after having neglected it for 15 months. Most of the view files still had the good old .rhtml extension. I was too lazy to rename these files by hand, both on my file system and in the git repository. I used the following Bash commands to do the job:

First, I renamed all the partials to the .erb extension. Note: I am not using .html.erb, as some of these partials are used in js-formatted responses as well:

for i in `find app/views/**/_*.rhtml`; do \
  git mv $i `echo $i | sed s/\\.rhtml$/.erb/`; \
done

The remaining files could now be renamed to .html.erb with a similar command:

for i in `find app/views/**/*.rhtml`; do \
  git mv $i `echo $i | sed s/\\.rhtml$/.html.erb/`; \
done

Note that this technique works with Subversion as well: just substitute git with svn in the command above. A regular rename is possible as well by leaving out git altogether!

Now my file names are Rails-compliant again, I can start refactoring all the code that is not up to current Rails standards anymore. Ah, the virtues of developing with a rapidly evolving framework…

Last week we had to convert our existing Rails application, which uses Gettext to the new I18n API in combination with the SimpleBackend. I personally never liked Gettext, there was simply not enough control over translations as PO/MO files are not native ruby or at least can be easily accessed by Ruby (like YAML files).

We therefore decided to switch to the brand new, not even released, I18n API. But now we had a serious problem, our code base isn’t small and all that code had to be converted in some way. We could do it by hand, but hey, that’s a lot of work and especially very error-prone. Some convertor had to be written. Here it is as a rails plugin: GettextToI18n. 

What does the I18n convertor do?

It scrapes your whole application and searches for gettext calls, like this:

_("to be translated")

It will convert this gettext call to the newly I18n format: 

I18n.t :message_id

Then it builds up a big hash containing all the the translations. We decided it was handy to use the scopes that are introduced in the new I18n api. So it stores the translations in the following format: 

For models:

["model"]["model_name"]={:message_1 => "to be translated"}

For controller:

["controller"]["controller_name"]={:message_1 => "to be translated"}

After this hash of translations has been built up, the convertor writes it as a YAML file to:  config/locales/template.yml.That’s all!

What’s supported?

It supports basic gettext calls. We have run it over our code base and it converts all gettext calls we use without any problem. 

A normal gettext call

_("to be translated")

converts to:

I18n.t :message_0, :scope => [:txt, :controller, :controller_name])

A gettext call with variables

_("My name is %{name}" % {:name => "Jaap"})

converts to:

I18n.t :message_0, :name => "Jaap", :scope => [:txt, :controller, :controller_name])

A gettext call with variables that contain gettext calls

_("Click %{link} to go to the homepage" % {:link => link_to(_("Here"), root_path)})

converts to:

I18n.t :message_0, :link => link_to(I18n.t(:message_1, :scope => [:txt, :controller, :controller_name]), root_path), :scope => [:txt, :controller, :controller_name])

Installation

./script/plugin install git://github.com/japetheape/gettext_to_i18n.git

Usage

To convert your application:
rake gettext_to_i18n:transform

Please make sure you backup your complete application as it can screw things up.

Contribution

Please contribute to this plugin and make it better, as I won’t use it anymore, cause we are not going to convert another time (I think ). Things that has to be done are:

unnamed variables:

_("I play the %s" % "saxophone")

Go to the development location at github and fork this plugin!

Since I announced rails-log-analyzer some weeks ago, quite a lot has happened! Apparently there is some interest in such a tool: on this blog we get a lot of traffic looking for more info, the github project already has 22 watchers and it even has been forked!

In the mean time, Bart and I worked hard to add new functionality and refactored the internal design. As a result, I have released request-log-analyzer 0.1.0 today!

Changes: 

• The project is renamed to request-log-analyzer, because we plan to support log files from other frameworks as well; Merb is planned to be supported in the near future.
• The tool is distributed as a gem, making it much easier to install and update.
• More reports, colorized output, parsing progress bars, command line arguments, etc…
• Added a tool to create a SQLite database with all the parsed info from the log file, so you can do your own analysis.

Installation:

gem sources -a http://gems.github.com
sudo gem install wvanbergen-request-log-analyzer

Usage:

request-log-analyzer  [LOG FILES*]
request-log-analyzer -c 20 -z log/production.log

Please let me know what you think! If you have any problems using the tool, do not hesitate to contact me!

My friend Bart from movesonrails.com just blogged about Rails log analyzer, a command line tool to get performance statistics for your Rails application by parsing its log file.

What started as an exercise for me to write a command line ruby program, has been extended and improved by Bart to be actually useful! We decided to release it under an MIT license. You can found the source on github. The project’s wiki contains usage information and an example of the output it will produce.