Tech Blog

Running an online service like Floorplanner is not without risks. There are a lot of things that can go wrong. One of them is downtime of your service due to a crashing system. It doesn’t have to happen of course, but when it does, it can have some nasty consequences. A good way to reduce the risk of a crashing system is to set up a redundant system.

In short, a redundant system is an exact copy of your main system. When your main system goes down (for whatever reason), the redundant system can take over. One of the most important parts of any online service is the database. A way to maintain an exact copy of your database is setting up database replication. This post gives you the basics on how to set up database replication using a MySQL database.

Database replication isn’t probably something you’d setup right from the start. This means that you have to work with a running system when you decide to do so. From past experience we know that it’s not a good idea to replicate your database to the same machine, let alone to the same disk. To make this work, you need a separate machine with MySQL installed. This is your slave machine and database.

Let’s get down to business. These are the steps for setting up database replication:

1. Enable binary logging on master database
2. Create a backup of master database
3. Transfer backup to slave machine
4. Import backup to slave database
5. Start slave database

1. Enable binary logging on master database

The binary log contains all statements that update data [..]. Statements are stored in the form of “events” that describe the modifications. The binary log also contains information about how long each statement took that updated data.

Every write action that’s performed on your database, like an insert or an update, is stored in these binary logs (bin logs). The master dumps all actions in these logs and the slave database can read them and perform the same actions. This way your master and slave databases will always have the same data.

To enable binary logging you have to create a config file, for example master.cnf. An important thing here is the server-id, which is needed for replication to work. Then there is log-bin which specifies base name of binary logs, and finally binlog-do-db, which specifies which databases should be bin logged.

[mysqld]
datadir=/home/yourname/mysql/myisam/data
basedir=/usr/local/mysql
port=3306
server-id=1
log-bin=mysql-bin
binlog-do-db=my_fat_db1
binlog-do-db=ma_other_fat_db

Then start (or restart) the master database using the config file:

~ $ mysqld --defaults-file=master.cnf&

The slave database needs a userid/password in order to access the master machine. It’s best practice create a dedicated user with just the required privileges. This can be done with the following SQL statement.

GRANT replication slave ON *.* TO 'repl'@'slave-ip-here'

2. Create a backup of master database

The next step it to create a backup of your master database. The type of your database is very important for this action. If you use MyISAM, you need to schedule some downtime to dump your database. Otherwise the dumped data isn’t consistent and therefor useless. With InnoDB it’s possible to dump a consistent backup of your database while keeping it up and running. We stumbled on this feature by accident, but we haven’t been able to in the official documentation.

MyISAM

~ $ mysqldump --lock-all-tables

InnoDB

~ $ mysqldump --single-transaction

3. Transfer backup to slave machine

Now that the backup is created on the master machine, it needs to be transferred to the slave machine.

4. Import backup to slave database

Once the backup is present on the slave machine, it can be imported into the slave database:

~ $ mysql < filename

5. Start slave database

The last step is to tell the database that it’s a slave database. First we have to give it an server id. We use a config file for this, slave.cnf:

[mysqld]
datadir=/home/yourname/mysql/myisam/data
basedir=/usr/local/mysql
port=3306
server-id=2
replicate-do-db=my_fat_db1
replicate-do-db=my_other_fat_db

Then we start the database with this config file:

~ $ mysqld --defaults-file=slave.cnf&

and we issue the following SQL statement:

CHANGE MASTER TO
-> MASTER_HOST='master-ip-here',
-> MASTER_PORT=3306,
-> MASTER_USER='repl',
-> MASTER_PASSWORD='repl',
-> MASTER_LOG_FILE='bin-log-filename-here',
-> MASTER_LOG_POS=4;

It’s hard to predict the MASTER_LOG_POS. You can find this number by issuing the following SQL statement on the master database:

SHOW MASTER STATUS\G;

Once the slave is configured correctly it will get the bin logs from the master machine, parse them and execute each action. It takes some time to have an exact copy of the master database, depending on the size and number of bin logs. To get an idea about the delay, issue the following query:

SHOW SLAVE STATUS\G;

With this post we hope to contribute to a less riskier world and more peaceful state of mind for the sys admins out there

The text editor of choice of our Rails team is TextMate. Our Flash team is a bit divided between Eclipse+FDT and FlexBuilder. I wanted to see if TextMate is a good alternative for building Flash/ActionScript projects.

There are a couple of good sources on using TexMate for ActionScript projects, but it’s a little fragmented and sometimes outdated. I found pixelate’s blog post and Simon’s blog very useful.

1. TextMate

First of all, you need have MacroMates’ TexMate of course. If you don’t already own a copy, you can download a 30-day-trial.

2. ActionScript3 bundle

Simon Gregory did some fine work by creating a TextMate bundle for ActionScript3. The most important things that the bundle handles are auto-completion and auto-import. You can download it directly from his blog, or fetch it with Git or SVN.

To install via Subversion:

export LC_CTYPE=en_US.UTF-8
cd ~/"Library/Application Support/TextMate/Bundles/"
svn co http://svn.textmate.org/trunk/Review/Bundles/ActionScript\ 3.tmbundle
osascript -e 'tell app "TextMate" to reload bundles'

To install via Git:

cd ~/"Library/Application Support/TextMate/Bundles/"
git clone git://github.com/simongregory/actionscript3-tmbundle.git "ActionScript 3.tmbundle"
osascript -e 'tell app "TextMate" to reload bundles'

3. Flex SDK

The actual building of your project is done by the free Flex 3 SDK. Download it and move the extracted folder into your /Developer/SDKs/ folder. The SDK has to be accessible throughout your whole system. You can do this by adding it to the PATH variable in the /etc/profile file. Open terminal and type:

sudo mate /etc/profile

Then add the folder “/Developers/SDKs/flex_sdk_3/bin” to the file and save it. It should look something like this:

PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/mysql/bin:/Developer/SDKs/flex_sdk_3/bin"

4. TextMate settings

Before we can start the ActionScript fun we have to setup a few things in TextMate. Open TextMate and select File→New Project. Click on the info button located in the bottom of the Project Drawer. Add two shell variables so that the ActionScript Bundle knows where to look for your files:

TM_FLEX_FILE_SPECS    src/Main.as
TM_FLEX_OUTPUT        bin/Main.swf

We also need to let TextMate know where the Flex SDK is located. Go to TextMate→Preferences→Advanced→Shell Variables and add a new global variable:

TM_FLEX_PATH    Developer/SDKs/flex_sdk_3

5. Hello World

With all that out of the way, we can finally start working on a ActionScript project: Hello World. Create a folder on your system that will hold this project. Drag this folder to TextMate’s Project Drawer. Create two new folders named bin and src in your project folder. Then create a new file in the src folder and name it Main.as. It should look something like this:

package {        
    import flash.display.Sprite;
    import flash.text.TextField;
 
    [SWF( backgroundColor='0xFFFFFF', frameRate='30', width='200', height='200')]
 
    public class Main extends Sprite {
        private var textField: TextField;
 
        public function Main() {
            textField = new TextField();
            textField.text = "Hello World.";
 
            addChild(textField);
        }        
    }
}

Choose Bundles→ActionScript 3→Build to build your project. A terminal window pops up which runs the Flex SDK to build your project. You can find the resulting Main.swf file in your bin folder. Yay!

Notes

If you want to build Flash Player 10 specific stuff (like me) you have to let the Flex SDK know. Open /Developer/SDKs/flex_sdk_3/frameworks/flex-config.xml and set the target-player tag to 10.0.0

<target-player>10.0.0</target-player>

The easiest way to work with libraries, the so called SWC files, in this setup is to drop them directly into /Developer/SDKs/flex_sdk_3/frameworks/libs folder. That way the Flex SDK has no problem finding them.

The last couple of weeks we’ve been working on a big integration project. The largest real estate portal of The Netherlands, Funda, uses Floorplanner to deliver interactive floor plans to their clients. They use our API’s to seamlessly integrate the Floorplanner into their back end system and front end website.

One of our API’s is our RESTful web service. With it one can manage users, project, floors, designs etc. We set up a whole testing suite around it, but every now and then I wanted to test a single method by hand. For this I used the command line tool cURL which looks something like this (getting all users):

curl -H "Content-Type: application/xml" 
-u "username:password" 
-X GET https://floorplanner.com/users.xml

It works, but it’s kind of a hassle. One day Michel showed me HTTP Client:

A Mac OS X Leopard developer tool for debugging HTTP services by graphically creating & inspecting complex HTTP messages.

HTTP Client makes testing a web service by hand much easier. You can select any REST method you need from a pull down menu. Setting up your header is done by a few mouse clicks and the result is nicely formatted.

The only thing that didn’t seem to work out of the box was Basic HTTP Authentication. I found a workaround for this: add the authorization to the header myself. To do this use “Authorization” as header name. For the header value you have to encode the username and password with Base64 with a “:” as separator. For example in PHP: base64_encode(username:password). There are also a couple of websites around where you can encode a string to Base64. When you have the encoded string, you can add “Basic encoded string here” as a header value and you’re ready to roll.

Publish Flash Player 10 content

To test this, I first needed a way to make my Flex Builder compile Flash Player 10 content. I found a rather old post by Andrei Ionescu about building Flash Player 10 applications that told me exactly what I was looking for. I downloaded the latest Flex SDK (3.3) from Adobe’s Flex Download page, followed the tutorial and I was ready to go.

Load image from local file system

Reading a local file was no biggie, just copied and pasted the code from Mike’s sample. What I needed now was a way to convert the loaded data into an image. As usual Tim helped me out by sending me a snippet:

var data:ByteArray = fileRef['data'];
var loader:Loader = new Loader();
loader.loadBytes(data);
addChild(loader);

Modify image

Now that I could load the image, I wanted to modify it before having it rendered. I extended the snippet to use BitmapData and a Matrix to resize the image to a thumb image (240×180px).

var loader:Loader = new Loader();
loader.contentLoaderInfo.addEventListener(Event.COMPLETE, onDataLoadComplete);
loader.loadBytes(loadFileRef.data);
 
private function onDataLoadComplete(e:Event):void {
  var bitmapData:BitmapData = Bitmap(e.target.content).bitmapData;
  var matrix:Matrix = new Matrix();
  matrix.scale(THUMB_WIDTH/bitmapData.width, THUMB_HEIGHT/bitmapData.height);
 
  imageView.graphics.clear();
  imageView.graphics.beginBitmapFill(bitmapData, matrix, false);
  imageView.graphics.drawRect(0, 0, THUMB_WIDTH, THUMB_HEIGHT);
  imageView.graphics.endFill();
}

Save image to local file system

Now that it’s possible to load and modify an image, saving it is the last step. To save an image, it has to be encoded to a ByteArray. I used the open source as3corelib to help me out. Saving the ByteArray to a file is rather straight forward.

Update 1: You don’t need to use the as3corelib anymore, Flex 3.3 has it own JPEGEncoder.

var encoder:JPEGEncoder = new JPEGEncoder();
var rawBytes:ByteArray = encoder.encode(bitmapData);
 
var saveFileRef:FileReference = new FileReference();
saveFileRef.save(rawBytes);

Update 2: Thibault Imbert figured out a way to speed up JPEG encoding using the new FP10 Vector class. Good stuff!

Update 3: You can now also use Alchemy for the encoding. It’s much, much faster then the other options! See this post from Jens Krause: Speed up JPEG encoding using Alchemy

This sample shows that it’s not only possible to load, modify and save images directly from the local file system, but that it’s actually very simple to do so. Add some PixelBender image processing power to the game and you almost have a Photoshop killer running completly client side!

Code sample

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" backgroundGradientColors="[0xFFFFFF,0xFFFFFF]" backgroundColor="0xFFFFFF">
 
	<mx:Script>
		<![CDATA[
			import mx.graphics.codec.JPEGEncoder;
			import mx.binding.utils.BindingUtils;
			import mx.core.UIComponent;
 
			import flash.net.FileReference;
			import flash.net.FileFilter;
			import flash.events.IOErrorEvent;
			import flash.events.Event;
			import flash.utils.ByteArray;
 
			private var loadFileRef:FileReference;
 
			private static const FILE_TYPES:Array = [new FileFilter("Image Files", "*.jpg;*.jpeg;*.gif;*.png;*.JPG;*.JPEG;*.GIF;*.PNG")];
			private static const THUMB_WIDTH:uint = 240;
			private static const THUMB_HEIGHT:uint = 180;
 
			private function loadFile():void {
				loadFileRef = new FileReference();
				loadFileRef.addEventListener(Event.SELECT, onFileSelect);
				loadFileRef.browse();
			}
 
			private function saveFile():void {		
				var bitmapData:BitmapData = new BitmapData(THUMB_WIDTH, THUMB_HEIGHT);
				bitmapData.draw(imageView);
 
				var encoder:JPEGEncoder = new JPEGEncoder();
				var rawBytes:ByteArray = encoder.encode(bitmapData);
 
 				var saveFileRef:FileReference = new FileReference();
				saveFileRef.save(rawBytes);
			}
 
			private function onFileSelect(e:Event):void {
				loadFileRef.addEventListener(Event.COMPLETE, onFileLoadComplete);
				loadFileRef.load();
			}
 
			private function onFileLoadComplete(e:Event):void {
			   	var loader:Loader = new Loader();
			   	loader.contentLoaderInfo.addEventListener(Event.COMPLETE, onDataLoadComplete);
			   	loader.loadBytes(loadFileRef.data);
 
				loadFileRef = null;
			}
 
			private function onDataLoadComplete(e:Event):void {
				var bitmapData:BitmapData = Bitmap(e.target.content).bitmapData;
 
                var matrix:Matrix = new Matrix();
                matrix.scale(THUMB_WIDTH/bitmapData.width, THUMB_HEIGHT/bitmapData.height);
 
				imageView.graphics.clear();
				imageView.graphics.beginBitmapFill(bitmapData, matrix, false); 
				imageView.graphics.drawRect(0, 0, THUMB_WIDTH, THUMB_HEIGHT);
				imageView.graphics.endFill(); 
 
				saveButton.enabled = true;
			}
 
		]]>
	</mx:Script>
	<mx:Panel title="Create thumb image" paddingBottom="5" paddingLeft="5" paddingRight="5" paddingTop="5">
		<mx:VBox>
			<mx:Canvas id="imageView" width="240" height="180" borderThickness="1" borderColor="#CCCCCC" borderStyle="solid"/>
			<mx:HBox paddingTop="5" borderThickness="1">
				<mx:Button label="Load image" click="loadFile()"/>
				<mx:Button label="Save image" click="saveFile()" id="saveButton" enabled="false"/>
			</mx:HBox>
		</mx:VBox>
	</mx:Panel>
</mx:Application>

This might be a little off topic, but I found it interesting enough to share it with you all. It’s about the future of capitalism: Constructive Capitalism through the principles of Renewal, Peace, Equity, Meaning and Democracy. 

It’s a presentation Umair Haque gave at Daytona Sessions last month. Umair Haque is Director of the Havas Media Lab, a new kind of strategic advisor that helps investors, entrepreneurs, and firms experiment with, craft, and drive radical management, business model, and strategic innovation. Daytona Sessions is a recurring conference about the future of business and Internet held in Stockholm, Sweden. 

Enjoy!

Ted Patrick (Senior Manager Developer Communities at Adobe Systems) has posted an interesting article about the first milestone of the Eclipse E4 project.

It seems that the SWT project has added compilation support for SWF from JAVA. Write your app in JAVA and publish as SWF to Flash Player. The cool part is that you get full JAVA development in Eclipse with all debugging and tooling but you get a SWF file on publish. 

We’re using Flex Builder for our Flex/AS3 stuff, but this could become a very interesting option since the JAVA development in Eclipse is a lot more sophisticated than Flex Builder at the moment.

Gert-Jan is on the left side and Jaap on the right.

Nico is on the left side and Willem on the right.

There is a lot of hype these day about being in “the cloud”. However, “the cloud” seems to mean a lot of different things. Tim O’Reilly sees three types of cloud computing.  His first type, utility computing, is the type I’m talking about here. 

Utility computing. Amazon’s success in providing virtual machine instances, storage, and computation at pay-as-you-go utility pricing was the breakthrough in this category, and now everyone wants to play. Developers, not end-users, are the target of this kind of cloud computing.

Launched in July 2002, Amazon Web Services provide online services for other web sites or client-side applications. Each individual product is interesting by itself, but all the services together made it just the solution we were looking for. In this and coming posts I will talk about our experiences with Amazon’s version of “the cloud”. The first service we started to use was Elastic Cloud Computing (EC2).

Each day more than 2000 users from all over the world register at Floorplanner.com. At this moment we have over 800.000 registered users in our database. A lot of these users visit the site on a regular basis, which generates a lot of traffic. To meet this demand, we had a couple of dedicated servers running. When the demand rised above a certain level, we’d add a new one to spread the load. As logical as it might sound, it is far from ideal.

It takes a lot of time to install, configure and maintain a server. Precious time you can’t spend on the development of your product. It doesn’t scale fast. For example: when you are featured on a big site that generates a lot of additional traffic, your servers will probably have a very hard time (or crash) and there is nothing you can do about it. Buying additional servers takes too much time. Another thing about buying servers is that it’s an investment you have to finance up front. You have to pay the full price for it, but in the beginning you’ll only use a small part of it’s capacities. Once you’ve spend the money on a server, you can’t spend it on anything else. And maybe the biggest issue we had with dedicated servers was that it is hardware, and hardware gets old very fast. After a while you’re stuck with an outdated server, that you (a) won’t replace because “it’s still doing something” or (b) costs time (= money) to migrate to another server. This means a higher risk for failure, or just more work.   

Now with the EC2 service life has gotten better. Sure, we still had to install everything on the virtual instance, but only the first time. Once you’ve done this, you can add additional instances with a click of a button (based on an image of the first instance). When traffic spikes you can add instances within minutes instead of days. The pay-as-you-go model keeps us from financing every server up front. And because it’s a virtual server, we don’t have to think about the hardware anymore. Just deploy it and get on with the fun stuff!

On February 22-24th the Flash In The Can event is back in town! Design, Technology. Cool shit. I will be just a visitor, but Tim is going to help Ralph with his workshop Papervision3D from the Core. If you want to meet up, give us a ping.

At this moment, all the language translations of the Floorplanner 2D app are stored in a database table. Today we discovered that a couple of these translations didn’t align properly in the interface. After some investigation we discovered that they all contained a hidden tab character at the end of  each string. This was probably caused by importing a malformed CSV file.

I thought a simple REPLACE query would fix this problem, but (as usual) it was a little more complicated than that. First I had to find the fields with the tab character… Willem pointed me to the right direction with his favorite weapon of choice REGEXP. According to the MySQL docs I could find tab characters with something like this:

SELECT * FROM table WHERE field REGEXP '[[.LF.]]'

The next step was to remove the tab characters. My first thought was to do this by replacing them with an empty string. It turns out you can’t combine a REPLACE with a REGEXP in a query. So I used good ol’ PHP for the job. A nice advantage was that I didn’t have to do any replacing, I could just use the trim() function.

$res = mysql_query("SELECT id, field FROM table WHERE field REGEXP '[[.LF.]]'");
if($res) {
	while($row = mysql_fetch_assoc($res)) {
		$id = $row['id'];
		$field = trim($row['field']);
		mysql_query("UPDATE table SET field = '$field' WHERE id = $id");
	}
}

Rather simple, when you know what to do… Another bug bites the dust!