log file ordering: assuming each request is stored into a single array of hashes, my first reaction would be to build an index based on request time. I’d use a two-dimensional array [[time, position], [time, position]], then sort based on the time to give me an index into request data array. Even though there’s forked processes, the original process is the one storing the results and creating the index.

gains from forking: depending on how much work you’re doing for those 6 lines, the tradeoff does indeed need to be verified. But, you could also split the file by regex into requests, then send batches of requests to each fork. Send each fork 10, 50, 1000 requests or whatever.

code file is “in the mail”

What you suggest seems like an approach with potential as it uses processes instead of threads. I definitely going to try this out at some point when I have some more time available. Can you send me your test file (willem AT floorplanner DOR com)? And, if you have time to spare, please fork the project and try it out yourself

I do expect some problems though, all of which can probably be worked around. First of all: Rails log files are not always ordered correctly. R-l-a uses some heuristics to detect this, but this may not work when parsing is split out in different processes. Moreover, in production mode every request, every requests consists of about 6 lines. Is the overhead of forking not bigger than the gains that can be made? Interesting stuff to try out.

Looking at a log file (for Rails anyway), it’s pretty easy to identify requests. They start with “Processing…” and have some empty lines between them. It would be pretty easy to fork processing on a per request basis. Assuming the organization of the rest of your processing wouldn’t result in huge RAM duplication (our processes are organized to minimize this).

I can see a workflow like: load log file, split into requests, iterate requests forking for each one, parse request, when all requests are parsed, main thread starts analysis (probably forking for each one), save results of each analysis to file, when all forks completed, read the file, stitch the results together for group analysis / presentation.

I have a cheesy test file to show you how to do the forking if you’re interested. I would expect it to speed up your processing quite a bit.

Ruby 1.9.1 does not make much difference, because all threads still block on IO operations. Of course, r-l-a does a lot of file-reading, so it needs to be rewritten using non-blocking IO to make a difference.

JRuby is actually a lot faster using the threaded approach, but is so much slower to begin with, that it is still slower than Ruby 1.8 in the end.

With the different approaches taken by the different interpreters, the different concurrency APIs that are available and the small speed gains to be made at this moment, I decided to not use threading for now.

Fortunately, the other tweaks did improve performance significantly, and they work with every Ruby version and interpreter.

What you say about Ruby not using multiple cores in a multiprocessor system is true for versions of Ruby prior to 1.9, when it still used so called “Green Threads”. If you want multicore performance with Ruby 1.8, you can use processes instead (or upgrade to 1.9 and benefit from native threading support).