To send a file to a Rails application, using a RESTful XML API call, use the following XML snippet:

<field type="file" name="filename.ext" content-type="mime/type">
  base64-encoded file contents
</field>

The file contents should be base64 encoded, and may be encapsulated in a CDATA block:

<greeting type="file" name="greeting.txt" content-type="text/plain">
  <![CDATA[SGVsbG8gd29ybGQh]]>
</greeting>

You can test this by opening a console and feed the XML to Hash.from_xml:

>> greeting = <<-EOXML
    <greeting type="file" name="greeting.txt" content-type="text/plain">
      <![CDATA[SGVsbG8gd29ybGQh]]>
    </greeting>
  EOXML
 
>> data = Hash.from_xml(greeting)
=> {"greeting"=>#<StringIO:0x10617d9d0>}
>> data['greeting'].content_type
=> "text/plain"
>> data['greeting'].original_filename
=> "greeting.txt"
>> data['greeting'].read
=> "Hello world!"

Now you can handle files using your XML API just like you would use file uploads in your HTML forms.

Redirecting

After a complete rewrite, we released the new version of Floorplanner over a year ago. We had to find a way to deal with the thousands of links that were in use by our customers and indexed by Google.

Some of these could simply redirect to their new location, e.g. http://www.floorplanner.com/tryit can now be found on http://www.floorplanner.com/demo. These redirects can of course simply be implemented in either Rails or Apache.

However, we also had a long list of URLs on which floor plans were published using the old version. These plans cannot be migrated to the new version easily, so we had to keep an instance of the old Rails application running. Requests to these URLs should be redirected to this separate instance, now hosted on a different domain name. Moreover, it is important that it only tries to redirect to the old version if the URL cannot be resolved in the new version, because the plan may have been migrated to or recreated on the new version.

We previously implemented this by adding a catch all route to our application, and a custom handler for the ActiveRecord::RecordNotFound exception in our application. This was a rather ugly solution, and we rather not have a dependency on the old version’s database in the new version’s code.

I decided to rewrite this functionality with a piece of Rack middleware. It basically executes every request with our Rails application, and if it returns a 404 response, it checks if the URL exists on the old version. If so, it redirects the user to the old version, otherwise, it will respond with the 404 response that was generated by Rails.

class RedirectToOldVersionMiddleware
  def initialize(app)
    @app = app
  end
 
  def call(env)
    # execute the request using our Rails app
   status, headers, body = @app.call(env)
 
    if status == 404 && url = find_redirect(env['REQUEST_URI'])
 
      # Issue a "Moved permanently" response with the redirect location
      [301, {"Location" => url}, 'Redirecting you to the new location...']
    else
 
      # Not a 404 or no redirect found, just send the response as is
      [status, headers, body]
    end
  end
 
  REDIRECT_HOST = 'http://old.floorplanner.version'
 
  def find_redirect(path)
    # See if we can find a valid URL on the old version.
    Redirect.exists?(path) ? "#{REDIRECT_HOST}#{path}" : nil
  end
end

This way, all the code that is related to redirecting users to the old version is contained in a single file, and doesn’t pollute our application anymore. Moreover, when all the old plans are migrated, or we decide we do not longer support these old plans, we can simply disable the middleware in our environment.rb, without having to alter any code.

Logging complete requests for debugging purposes

Another use for Rack middleware is logging: one of our customers notified us that an API call they were using was failing, but only some of the time. Unfortunately, we did not find any errors in our exception log, which made it very hard to debug this issue. After some time, we discovered that Rails was returning a 422 response, without the request ever “arriving” in our own code. Apparently, the request was stopped from processing somewhere in the Rails framework.

Unfortunately, we were not able to recreate this issue ourselves, and we started suspecting that the request was not well-formed, because Rails sends a 422 response in this case. To discover if this really was the issue, I wrote a piece of middleware that logs the complete request, before it is passed to the Rails framework for further processing.

class RequestLoggerMiddleware
  def initialize(app)
    @app = app
  end
 
  REQUEST_LOG_DIR = Rails.root.join('log', 'api_calls')
 
  def call(env)
    # log the request if it is a troublesome API call.
    if env['REQUEST_URI'] == '/failing_api.xml'
 
      filename = "api_call_#{Time.now.strftime('%Y%m%d%H%M%S')}.log"
      File.open(REQUEST_LOG_DIR.join(filename), 'wb') do |file|
        file.puts(env['rack.input'].read)
      end      
    end
 
    # now, execute the request using our Rails app
    response = @app.call(env)
  end
end

Using these request log files, API calls can be inspected exactly before any of the Rails magic happens. After resolving the issue, removing this logging was as simple as commenting a line in environment.rb.

• PostgreSQL query log support;
• Delayed::Job log support;
• Small fixes in the Rails file format;
• Various other small fixes and improvements.

As always, run the following command to install or upgrade to the latest version:

sudo gem install request-log-analyzer

• MySQL slow query log format support to analyze what queries are slowing down your database.
• Format autodetection: with all those supported file formats, remembering the right --format parameter gets tricky. With format autodetection, this usually is not needed anymore!

As always, run the following command to install or upgrade to the latest version:

$ gem install request-log-analyzer

For example, we use validates_uniqueness_of to check for duplicate e-mail addresses. Because email addresses are case-insensitive, adding :case_sensitive => false seems like a natural choice. However, this results in the following queries:

# For a new User instance:
SELECT id FROM users 
 WHERE LOWER(users.email) = BINARY 'user@example.com'
 
# For an existing User instance:
SELECT id FROM users 
 WHERE LOWER(users.email) = BINARY 'user@example.com' 
   AND users.id <> 42

This query cannot be optimized by a (unique) index on the email field and thus has to scan the full table. As our users table grew larger, these queries started to show up in our slow query log.

However, MySQL uses case-insensitive comparison by default. (To be exact, case-sensitiveness depends on the current collation, which can vary. Rails generates the weird query to make sure the check works, regardless of the current collation.) The conversion to lowercase therefore is not necessary for a uniqueness check (as long as the field has a case-insensitive collation like utf8_general_ci). I decided to write my own validation method that issues a query that can be optimized by a query.

  # Alternative for validates_uniqueness_of :email, :case_sensitive => false
  validate do |user|
    conditions = "users.email = :email"
    conditions << " AND users.id != :id" unless user.new_record?
    conditions = [conditions, { :email => user.email, :id => user.id }]
    if User.find(:first, :select => :id, :conditions => conditions)
      user.errors.add(:email, 'Already in use')
    end
  end

There is a ticket for this issue in Rails’s Lighthouse, but as of yet this issue is unresolved. For now, this solution works to keep our slow query log nice and quiet!

• New and improved log formats: r-l-a can now handle Apache access logs, Rack CommonLogger logs and Amazon S3 access logs. Moreover, the Rails format has been restructured to offer more flexibility.
• Improved database support: the database supports other databases than SQLite3 as well, and r-l-a can append information to an existing database instead of overwriting it. Moreover, a console tool similar to Rails’s script/console is bundled to inspect the database and run queries on it easily.
• Added standard deviation to reports: the standard deviation measure has been added to duration and traffic reports to get some feel of the variation in values besides the mean.
• E-mailing reports: r-l-a can email the performance report to a given e-mail address. This can be useful when running r-l-a in a cron job.
• Compressed log support: r-l-a will decompress compressed logs automatically.
• Speed improvements: we have profiled request-log-analyzer itself and significantly improved its performance.
• API: we created a basic API so it is possible to use the r-l-a engine as a library as well.
• Monitoring integration: integrate performance information into your Munin dashboard or your Scout account.

As always, use sudo gem install request-log-analyzer to install or upgrade.

Ruby en Rails 2009 conference

Bart and I will be presenting request-log-analyzer and performance tuning of Rails applications in general at the Ruby en Rails conference in Amsterdam, October 30-31 2009. We hope to see you there!

Currently, the gem provides the following:

• Simple configuration and setup.
• Uses Adyen’s test or production environment based on your Rails environment.
• Generating hidden form fields for redirecting to Adyen for a payment.
• Calculating the signature to sign these redirects.
• Checking Adyen’s signature when the user gets redirected back.
• Matchers to easily test your payment forms using RSpec.
• Receiving and storing notifications from Adyen.
• Calling the Adyen SOAP services (requires the Handsoap gem).

Currently, not all SOAP services are implemented (because we didn’t need them all). It should be quite easy to implement them as well based on the other services that are implemented already. Don’t hesitate to submit patches!

New query language

• Logical operators: AND (&, &&), OR (|, ||) and NOT (!, -) operators, and parentheses to structure the boolean logic: police AND (car || uniform), -"village people". By default, the AND operator is used to combine different segments of your query.
• Comparison operators: the most common comparison operators are supported, and to what you expect on integer and date field.
• Explicit field support: only search in the specified field instead of all fields: age >= 21, created < 2009-01-01, username != "root".
• Check for NULL fields: null? parent, set? error_message
• Commas are supported to separate the different parts of the query.

More information about the query language can be found in the project wiki on GitHub.

New definition syntax

The new version supports a new syntax to define what fields of your model can be searched and in what cases. An example:

class User < ActiveRecord::Base   
  belongs_to :account_type
 
  scoped_search :on => [:first_name, :last_name]
  scoped_search :on => :created_at, :alias => :created, :only_explicit => true
  scoped_search :in => :account_type, :on => [:name, :description]
end

After the fields have been defined, the search_for method can be used to search your models using a named scope, just like it was before. The project wiki has more information about this new syntax. The search syntax itself hasn’t changed:

@users = User.search_for(params[:q]).paginate(page => params[:page])

Installation or upgrade

Include the gem in your environment.rb configuration and run rake gems:install to install it:

config.gem 'scoped_search', :source => 'http://gemcutter.org'

Backwards compatibility

The new version has a new syntax to define the fields that can be searched with a query. This new syntax gives you more fine-grained control over the queries that will be generated, so I urge you to adopt this new syntax. However, the old searchable_on syntax is still available for backwards compatibility.

Please contact me if you have any issues with the new version.

(Updated with new gemcutter installation instructions.)

What’s new?

• More robust log parser. It parses more lines and it now combines all lines that belong to the same request, which greatly improves the amount of information available. 
• It produces more detailed and more beautiful reports. 
• A database builder is included, which will create an SQLite 3 database with all parsed request information, so you can roll your own queries.
• Request filtering options, so you can exclude irrelevant data. An example on how this can be applied in practice can be found in the wiki. 
• Better, more modularized design under the hood. The parser is now fully log file format-agnostic. Developing extensions and modifications, or adding support for other log file formats should be much easier now. See the development-page for some pointers.
• Documentation in the project’s wiki. Hopefully, this helps people get up to speed with the new version and answers most questions about using the tool. If you still have questions, please contact us so we can keep improving it!

Installation

Install or upgrade to the new version with the following command:

$ sudo gem install wvanbergen-request-log-analyzer 
                --source http://gems.github.com

To get the best results out of request-log-analyzer, it is important to configure logging correctly for your application. Some pointers on how to set things up correctly can be found in the wiki.