Tech Blog

I have been working on request-log-analyzer quite a lot recently. One of the things I focused on was improving the parsing performance: because it parses log files that often are very big, processing times tend to be long. So all savings are very welcome.

Improving the performance of a command line application that does a lot of processing is very different from optimizing the performance of a web application. Request-log-analyzer basically reads a file line by line and processes it, so small performance improvements in the line processing algorithm can really add up when the file has a lot of lines. I used ruby-prof to get information about the performance of our algorithms split out by method to focus my tweaking efforts. I have written down some of my findings below; hopefully they can be helpful.

Parallelization

My first thought for improving performance was parallelization: parse multiple lines at the same time. Unfortunately, this did not yield the results I was hoping for: it instead become slower. Probably, this is because Ruby implements its own in-process threading and thus only uses one core of my processor.

Block overhead

The overhead of using a block should not be underestimated. Consider the following simple change, which improved the performance of request-log-analyzer by 1.5-2% on large log files:

# with block
io.each_line { |line| process_line(line) }
 
# without block
process_line(line) while line = io.gets

Regular expressions

If you’re using complex regular expressions, and you do not expect that every string will match it successfully, it can be beneficial to test the string with a simpler regexp first. For example, request-log-analyzer uses a complex regexp to see if a line in a log file is a “Completed in…” line with the request duration in it:

# Check every line to see if it is a "completed" line and capture the values
if line =~ /Completed in (\d+)ms \((?:View: (\d+), )?DB: (\d+)\) \| (\d\d\d).+\[(http.+)\]/
  # do something with the captured values
end

I improved this by first checking for a superficial regexp that tells me with 99% certainty that the complex regexp will match the line successfully as well:

if line =~ /Completed in/
  if line =~ /Completed in (\d+)ms \((?:View: (\d+), )?DB: (\d+)\) \| (\d\d\d).+\[(http.+)\]/
    # do something with the captured values
  else
    $stderr.puts "#{line.inspect} expected to match 'Completed' regexp!"
  end
end

Depending on the log file, this can increase performance by ~3%. Another benefit of this approach is that it will give feedback on lines that matched the simple regexp, but not the complex one. This information can be used to correct the regular expressions.

Calculate things that do not change only once

Calculate things that do not change only once is easier said than done. For instance, request-log analyzer can aggregate durations in a category. A category can be based on a request field that is parsed from the log, or a Proc that calculates it:

# during parsing:
if categorizer.respond_to?(:call)
  category = categorizer.call(request)
else
  category = request[categorizer]
end
categories[category].aggregate_request(request)

With this implementation, categorizer will be checked to be a Proc for every request, but as the value of categorizer will not change, the result of the check will be constant as well. I solved this my making sure that it is always a Proc beforehand, so the check is no longer necessary during parsing:

# before parsing:
if categorizer.respond_to?(:call)
  categorizer_proc = categorizer
else
  categorizer_proc = lambda { |request| request[categorizer] }
end
 
# during parsing:
category = categorizer_proc.call(request)
categories[category].aggregate_request(request)

Performance gain: ~1%! And because on several instances a similar technique could be applied, the performance got improved by about 4% in total.

Check the most common case first

Consider the following example, which converts a number in any traffic unit (kilobytes, MB, etc) to bytes:

def convert_traffic(value, unit)
  case unit
  when :GB, :G, :gigabyte      then (value.to_f * 1000_000_000).round
  when :MB, :M, :megabyte      then (value.to_f * 1000_000).round
  when :KB, :K, :kilobyte, :kB then (value.to_f * 1000).round
  # ... even more units here
  else value.to_i
  end
end

In most cases, the value will simply given in bytes, which will be returned by the final else after all possibilities have been checked. This can be improved by checking for this possibility first:

# Converts traffic in any unit to bytes.
def convert_traffic(value, unit)
  case unit
  when nil, :B, :byte          then value.to_i
  when :GB, :G, :gigabyte      then (value.to_f * 1000_000_000).round
  when :MB, :M, :megabyte      then (value.to_f * 1000_000).round
  when :KB, :K, :kilobyte, :kB then (value.to_f * 1000).round
  # ... even more units here
  else raise "Unknown unit: #{unit.inspect}!"
  end
end

Again this change adds up to a ~1% performance increase if this method is called very often.

These kinds of changes really improved the performance of request-log-analyzer by quite a bit, so upgrade to the latest version to get some of your valuable time back!

The text editor of choice of our Rails team is TextMate. Our Flash team is a bit divided between Eclipse+FDT and FlexBuilder. I wanted to see if TextMate is a good alternative for building Flash/ActionScript projects.

There are a couple of good sources on using TexMate for ActionScript projects, but it’s a little fragmented and sometimes outdated. I found pixelate’s blog post and Simon’s blog very useful.

1. TextMate

First of all, you need have MacroMates’ TexMate of course. If you don’t already own a copy, you can download a 30-day-trial.

2. ActionScript3 bundle

Simon Gregory did some fine work by creating a TextMate bundle for ActionScript3. The most important things that the bundle handles are auto-completion and auto-import. You can download it directly from his blog, or fetch it with Git or SVN.

To install via Subversion:

export LC_CTYPE=en_US.UTF-8
cd ~/"Library/Application Support/TextMate/Bundles/"
svn co http://svn.textmate.org/trunk/Review/Bundles/ActionScript\ 3.tmbundle
osascript -e 'tell app "TextMate" to reload bundles'

To install via Git:

cd ~/"Library/Application Support/TextMate/Bundles/"
git clone git://github.com/simongregory/actionscript3-tmbundle.git "ActionScript 3.tmbundle"
osascript -e 'tell app "TextMate" to reload bundles'

3. Flex SDK

The actual building of your project is done by the free Flex 3 SDK. Download it and move the extracted folder into your /Developer/SDKs/ folder. The SDK has to be accessible throughout your whole system. You can do this by adding it to the PATH variable in the /etc/profile file. Open terminal and type:

sudo mate /etc/profile

Then add the folder “/Developers/SDKs/flex_sdk_3/bin” to the file and save it. It should look something like this:

PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/mysql/bin:/Developer/SDKs/flex_sdk_3/bin"

4. TextMate settings

Before we can start the ActionScript fun we have to setup a few things in TextMate. Open TextMate and select File→New Project. Click on the info button located in the bottom of the Project Drawer. Add two shell variables so that the ActionScript Bundle knows where to look for your files:

TM_FLEX_FILE_SPECS    src/Main.as
TM_FLEX_OUTPUT        bin/Main.swf

We also need to let TextMate know where the Flex SDK is located. Go to TextMate→Preferences→Advanced→Shell Variables and add a new global variable:

TM_FLEX_PATH    Developer/SDKs/flex_sdk_3

5. Hello World

With all that out of the way, we can finally start working on a ActionScript project: Hello World. Create a folder on your system that will hold this project. Drag this folder to TextMate’s Project Drawer. Create two new folders named bin and src in your project folder. Then create a new file in the src folder and name it Main.as. It should look something like this:

package {        
    import flash.display.Sprite;
    import flash.text.TextField;
 
    [SWF( backgroundColor='0xFFFFFF', frameRate='30', width='200', height='200')]
 
    public class Main extends Sprite {
        private var textField: TextField;
 
        public function Main() {
            textField = new TextField();
            textField.text = "Hello World.";
 
            addChild(textField);
        }        
    }
}

Choose Bundles→ActionScript 3→Build to build your project. A terminal window pops up which runs the Flex SDK to build your project. You can find the resulting Main.swf file in your bin folder. Yay!

Notes

If you want to build Flash Player 10 specific stuff (like me) you have to let the Flex SDK know. Open /Developer/SDKs/flex_sdk_3/frameworks/flex-config.xml and set the target-player tag to 10.0.0

<target-player>10.0.0</target-player>

The easiest way to work with libraries, the so called SWC files, in this setup is to drop them directly into /Developer/SDKs/flex_sdk_3/frameworks/libs folder. That way the Flex SDK has no problem finding them.

Just committed rev. 911 to the Papervision3D SVN trunk. Rev. 911 and upwards will become Papervision3D 2.1 because the changes made are quite big.
Major changes where made to the DAE, MD2 and animation classes.

NOTE:
This revision is considerably different then previous revisions. Use with care!
At this point its not advised to use rev. 911 for production.

This revision fixes several issues regarding the DAE class:

1. vertex-animation
2. nested animations
3. Cinema4D support
4. morph-weight animation
5. splines
6. cloning
7. play(), play(”clipName”), stop(), pause(), resume()
8. more…

The whole org.papervision3d.core.animation.* package has been revamped completely to allow for the changes in the DAE class.

DAE Example:

var autoPlay : Boolean = false; // don't play animations automatically
 
var dae : DAE = new DAE( autoPlay, "myCollada" );
 
dae.addEventListener(FileLoadEvent.LOAD_COMPLETE, onDaeComplete);
dae.addEventListener(FileLoadEvent.LOAD_PROGRESS, onDaeLoadProgress);
 
// optionally pass materials to DAE
// NOTE: here's a change with previous revs :
// 1. lookup the <material> elements in the COLLADA file (inside <library_materials>).
// 2. write down / remember the @id attribute of the <material> element.
// 3. materials.addMaterial( myMaterial, materialElementID ).
// ==> this will probably change in future revs
var materials : MaterialsList = new MaterialsList();
 
// If textures fail to load optionally add some search-paths 
// (relative to the swf):
dae.addFileSearchPath( "images" );
dae.addFileSearchPath( "textures" );
 
// set to true if you get a script-timeout error
var asyncParsing : Boolean = false;
 
// load it!
dae.load( "/path/to/dae", materials, asyncParsing );
 
/**
 * The DAE has loaded completely
 */
private function onDaeComplete(event : FileLoadEvent) : void
{
     var dae : DAE = event.target as DAE;
 
     // add to scene
     scene.addChild( dae );
 
     // start playing animation (if any available)
     // other animation controls include :
     // 1. play( "clipName ")
     // 2. stop()
     // 3. pause()
     // 4. resume()
     // 5. playing (getter: bool indicating if playing)
     dae.play();
 
     // lets create a clone
     // NOTE: DAE#clone() is somewhat bugged still, 
     // but seems to work in most cases
     var clone : DAE = dae.clone() as DAE;
 
     // add clone to scene
     scene.addChild( clone );
 
     // move it a bit
     clone.x = 200;
}
 
private function onDaeLoadProgress(event : FileLoadEvent) : void
{
 
}

MD2 Example:

var md2 : MD2 = new MD2();
 
var material : MaterialObject3D = new WireframeMaterial();
 
md2.addEventListener(FileLoadEvent.LOAD_COMPLETE, onMD2Complete);
 
md2.load("/path/to/md2", material);
 
scene.addChild(md2);
 
private function onMD2Complete(event : FileLoadEvent) : void
{
       var md2 : MD2 = event.target as MD2;
 
       md2.play();
       // or play some clip :
       // md2.play( "run" )
}

Animation:

Click the image below to show an example of the new animation controls.

Download the source of this example here.

The DAE and MD2 class implement IAnimatable, IAnimationProvider and IControllerProvider, which can be found in the org.papervision3d.core.animation.* package.

• IAnimatable provides the play(), stop(), pause() and resume() methods.
• IAnimationProvider provides a getter / setter #animation for low level access to org.papervision3d.core.controllers.AnimationController
• IControllerProvider provides a getter / setter #controllers for low level access to the different controllers:
  • org.papervision3d.core.controllers.MorphController
  • org.papervision3d.core.controllers.SkinController
  • org.papervision3d.core.controllers.AnimationController

As so much has changed I’m sure some bugs are introduced. Please let me know!

PS: I’ll be on vacation until june 8th, so its unlikely I’ll be able to fix any bugs before that time.
PS2: Many people have helped by submitting code-snips, reporting bugs etc. I’ll credit you all when I’m back

I just updated Papervision3D to allow the CameraObject3D#unproject method to work when CameraObject3D#useProjectionMatrix = true.

Papervision3D has two methods of ‘projecting’ vectors onto the screen:

1. useProjectionMatrix = false, this is the default ‘fast’ method
2. useProjectionMatrix = true, this is the method where projection is done by a matrix

Note that Papervision3D switches to method #2 in ‘ortho mode’.

So what is the difference exactly between these two kinds of projection? First we need to define what projection is. Projection is what adds ‘perspective’ to a scene and makes sure your scene fits to the viewport. Its the last step in the so-called ‘render pipeline’.

We can derive the ‘fore shortening’ effect of perspective in several ways. Method #1 is simple, effective and fast. It uses the camera’s focus and zoom values. Method #2 uses a more classic way of projection : it uses a dedicated matrix. Tech buffs: very much like OpenGL’s gluPerspective.

Andy Zupko has a great post on unproject using method #1.

We will discuss unproject using method #2, which works slightly different.

The difference is that when #useProjectionMatrix is set to true, then CameraObject3D#unproject returns a point in world-space, rather then a ray. Let me explain with some code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

// we want to use a projection matrix
camera.useProjectionMatrix = true;
 
// '0' indicates we want a point on the near plane
var pointOnNearPlane : Number3D = camera.unproject( screenX, screenY, 0 );
 
// '1' indicates we want a point on the far plane
var pointOnFarPlane : Number3D = camera.unproject( screenX, screenY, 1 );
 
// Construct the ray's direction
var dir : Number3D = Number3D.sub( pointOnFarPlane, pointOnNearPlane );
 
// Normalize
dir.normalize();
 
// So, now you have a ray with its origin at 'pointOnNearPlane',
// and direction 'dir'

Optimization: could save some cycles by doing (*not* in ortho mode!, see below):

1
2
3
4
5
6
7

var camPosition : Number3D = new Number3D( camera.x, camera.y, camera.z );
// Construct the ray's direction
var dir : Number3D = Number3D.sub( pointOnFarPlane, camPosition );
// Normalize
dir.normalize();
// So, now you have a ray with its origin at the camera's position
// and direction 'dir'

Why can’t we use above optimization in ortho mode?
In perspective mode all ‘rays’ originate from the camera’s position: the ‘view cone’ has shape of a pyramid. Hence we can assume ‘pointOnNearPlane’ to coincide with the camera’s position.
In ortho mode however the ‘view cone’ has the shape of a cube, all rays are parallel. Hence we need to unproject *two* points to construct our ray.

Now simply follow Andy Zupko ’s post to drag objects around.
Or perform ‘picking’, or…

Update:
Created a little demo to visualize what I’m ranting about, check it out here

This week, request-log-analyzer obtained its 100th watcher on GitHub!

Bart and I have worked hard to make r-l-a a useful product for many people in various situations. The fact that more than 100 people are following the project’s progress and that at this moment, the gem has been download almost 200 times, shows that we are somewhat successful in this regard. Numbers like these, in combination with the e-mail messages we have received, motivate us to keep spending time on the project and keep improving it, even if these improvements are not directly useful for our own projects. It has grown beyond scratching our own itch. 

On a related note, my Flickr photostream recently welcomed its 10,000th visitor.

What started as a convenient utility to backup and share my holiday photos with my family and friends, now has become somewhat of a showcase of what I am about and what I am up to. I would not consider myself a “professional” photographer and I am not an active member of the Flickr community, but still I get feedback on my photographs by visitors, because their visits increase the view counters of my photos.

These counters have really motivated me to make more of an effort when I put photos on Flickr. I started by adding titles, descriptions and tags, so that my photos are easier to find. I also became much more critical of the pictures I upload to Flickr: new uploads have to add something significant to my collection. Analyzing why some pictures got more attention than others made me a better photographer, although there still is a lot of room for improvement.  

Ignite the lazyweb, kick-start a quality improvement loop

What interests me in these examples is that simple counters like watchers on GitHub or views on Flickr can be valuable feedback and can motivate people to put in effort. The end result is quality improvement: write better software, make better pictures, etc.. Even an inherently lazy person like me can get motivated to keep putting in effort and to keep improving myself, because of such a simple feedback loop!

Additionally, it creates a dependence on the website in question. I look at my Flickr stats page every day, and I am subscribed to my activity feed on GitHub to get notified when new people start watching my projects. I now simply have to publish tools I write on GitHub to boost its quality, immediately and in the long run. And I have to upload pictures to Flickr as it is vital for my photography learning curve.

I guess I finally figured out what Web 2.0 is all about! Can we use a similar technique on Floorplanner to boost the quality of the designs our visitors make?

Ted Patrick (Senior Manager Developer Communities at Adobe Systems) has posted an interesting article about the first milestone of the Eclipse E4 project.

It seems that the SWT project has added compilation support for SWF from JAVA. Write your app in JAVA and publish as SWF to Flash Player. The cool part is that you get full JAVA development in Eclipse with all debugging and tooling but you get a SWF file on publish. 

We’re using Flex Builder for our Flex/AS3 stuff, but this could become a very interesting option since the JAVA development in Eclipse is a lot more sophisticated than Flex Builder at the moment.

Because Jaap finally convinced Gert-Jan, we have moved to the Git version control system for the main Floorplanner repository. Now we can use branches for different functionality far more easily. As an easy reminder to some common Git tasks I will need regularly, I have written down some Git recipes. This is basically meant for me and my fellow developers, but maybe it can help you as well. Suggestions to improve my workflow are welcome!

Displaying the current branch in your prompt

Because I will be using branches more regularly now, it is nice to know what branch I am currently working in. git branch will provide this information, but it can be more direct by including the current branch in your terminal prompt.

To display the current git branch in my terminal prompt, I have added the following to my Bash configuration file ~/.profile:

parse_git_branch() {
  git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/ \[\1\]/'
}
 
PS1='\[\033[01;37m\]\w\[\033[00;35m\]$(parse_git_branch)\[\033[00m\] \$ '

I use a black, semi-transparent terminal with white text, and pink branch names. Change the colors to your own liking!

Working on a remote branch that is not available locally yet

If you want to help out on a branch that somebody else started and has pushed it to the remote repository, you can checkout this branch and make it “track” the remote branch.

$ git checkout -b newbranch origin/newbranch

In this example, origin/newbranch is the branch in the remote repository. Locally, this branch will be called newbranch. A simple git pull will update my branch with the latest changes from the remote branch later on, git push will send my changes to the remote server.

Creating a new branch and pushing it to the remote server

Sometimes I want to start a new branch myself if I want to work on a new feature or on disruptive refactoring of the main codebase.

$ git checkout -b feature

The feature branch is now available locally. After some time, I want to share my current changes in this branch with other developers. I should make the branch available in the remote repository so other can access it like I described above.

$ git commit -m "Added initial version of %feature%"
$ git push origin feature

The feature branch will now be available to other developers as well. Note that the local feature-branch is not tracking the remote branch of the same name. This can be enabled by changing the configuration of the repository.

Merging a branch

After work is completed on my feature-branch and it is tested thoroughly, I want to merge the branch with the master branch of the project. To make sure the merge with the master branch is seamless and all possible merge conflicts are handled beforehand, we first run git rebase. This ensures that the changes in the feature branch are relative to the latest commit to the master branch and can therefore be applied by a “fast forward”. It is best to run git rebase from time time to time while you are developing in the feature branch to make sure your work does not divert to much from the main development in the master branch.

  # make sure that the current branch is seamlessly
  # mergeable with the master branch
$ git rebase master
  # switch to the master branch
$ git checkout master
  # now, merge the feature branch
$ git merge feature
  # publish the merge to the remote server
$ git push origin master

I can now delete the local and remote feature branch, as the changes have been incorporated in the master branch:

$ git branch -d feature
$ git branch -d -r origin/feature

I just released version 0.2.0 of request-log-analyzer, our tool to analyze request log files that are generated by Rails and Merb for performance tweaking. This new version supports the new log format of Rails 2.2, which has changed slightly.

An updated gem should be available any minute now. Run sudo gem update to upgrade the newest version.

Papervision3D has won the INNOVATION OF THE YEAR in this year’s .net Awards!
Other nominees included Google Android and App Engine, Microsoft Telescope, Open Social and Silverlight 2. I’m very proud to be part of the Papervision3D team!

Read more on the Papervision3D blog.

Last week I started working on a new version of the print functionality in Floorplanner. The problem with the previous version was that it didn’t print shadows, it was impossible to rotate the plan to landscape and textures didn’t look very well. Overall it produced a very mediocre image compared to the visual quality of a plan on the screen. As can be seen below:

Old print output

For printing we use the AS2 PrintJob class, which works really easy, with a small disadvantage: it doesn’t print Floorplans properly. I tried all kinds of settings for this class, with as input just a MovieClip with a drawing of the plan on it. I tried setting the printAsBitmap to true and false, both giving the exact same result. So although being stuck with it, the printAsBitmap boolean inspired me to investigate the Bitmap thing. We already had an image export utility, which produces images of a Floorplan with the same quality as it is shown in the FlashPlayer. The funny thing is that when I printed this exported image, the print quality was really good, at least a lot better than our own printing functionality. So for some reason, even if I used the printAsBitmap functionality, it still didn’t produce the same quality as using a real bitmap.
Of course we could have stopped here and just tell the users of Floorplanner to print the exported image. But since we are Floorplanner, we didn’t want to introduce another step requiring user action in the printing process, after all, we are the “easiest way” .
So I decided to use the same BitmapData created for the image export as basis for the printing. The first results where rather disappointing, I attached the created BitmapData to a MovieClip which in turn was given to the PrintJob. The produced print was of really low quality, you could easily see pixels all over the place. I tried using more pixels but now I bumped into the pixel limit of BitmapData (2880×2880 pixels). To work around this I invented the BigBitmapData class, this is a wrapper class that mimics the functionality of the BitmapData class, but which doesn’t have an upper limit of the number of pixels which can be used. When an instance of BitBitmapData is created which is bigger than 2880 by 2880, it creates additional BitmapDatas to store the additional pixels needed. So in fact the BigBitmapData internally stores an two dimensional array of BitmapDatas to bypass the pixel limit.
So with this new class I was able to scale up the created image to unprecedented sizes. I did a lot of printing experiments with scaling of the input, I tried scaling settings from 2 times to more than 10 times! This huge bitmap was attached to a MovieClip which in turn had to be scaled to fit on one single page of the printer. These values did give a better result, but still not good enough. It turned out that I had to take the DPI of the printer into account. The optimal scale factor turned out to be (1 / 72) * 300. The 72 is the DPI of the screen, and the 300 is the DPI of the printer. Furthermore a ‘point’ (print unit of measurement) is 1/72 inch, while the size of a pixel (screen unit of measurement) is dependent on the resolution of the screen.
So what this method actually does is creating a big image (which is resized to the paper contents) with enough quality to be printable on a 300 DPI printer.

New print output

So as you can see in the second image above, the new print looks a lot better, and since the new print uses bitmaps it is really easy to rotate it to always produce a print that tries to fill the entire paper.

A disadvantage of this method is that it takes quite some computational power to compute all the pixels for the bitmap that is created in memory, so as a result the FlashPlayer and sometimes even the browser freezes for a moment. Although this isn’t very good, we thought this behavior is acceptable since the quality of the prints is much better than the previous ones and the freeze time is actually really short. If you are not convinced by the images (which is understandable since the quality difference can only be seen on paper), just check it out!