Adam Howitt's Blog

Amazon S3: 503 error Http/1.1 Service Unavailable

To say I'm disappointed is an understatement but Amazon S3 is currently reporting Http/1.1 Service Unavailable since 6.02 AM.  Boooo.  I've added a cfhttp call to the top of WalkJogRun.net to detect the 503 error so I can at least offer an explanation to my visitors.

 

I know all sites suffer from failures periodically but Amazon S3?  I thought the purpose was massive redundancy so that your site never goes down or your content is always available.  Hmph.  At least they are on it.

UPDATE: It's 9.04am - 3 hours after the outtage and it's back again.  Big burp appears to have taken place and cleared the airways.

Moving your static assets to Amazon S3

If you need a cheap host for your podcast, pdf files, site images or any of your static site assets, take a look at Amazon S3.  I've been using it since just after it launched and it's a great way to reduce bandwidth to your ColdFusion/ Application server.  For WalkJogRun my average monthly bill is just under $6, not bad for a site that has seen 27 million hits this last year.  I'll give you a quickstart guide to for using it with your site and at a later date I'll describe how to use S3.cfc to integrate S3 into your ColdFusion application to allow you to push files from your site visitors up there too.  You should be up and running in about 10 minutes or less.

To use it for your site you will need to setup an Amazon S3 account (http://www.amazonaws.com).  Once you create an account and join the Amazon S3 service, you will be shown an access key and a secret key.  Copy these down.  Before we get into copying files, you might want to add a CNAME host "s3" to your DNS records to point to S3.amazonaws.com.  This means when you are done copying files the DNS records may have been refreshed and you'll be ready to rock.  Adding a CNAME as described will make it possible for you to refer to the files you put up on Amazon S3 with http://s3.yourdomain.com/myfile.jpg for example.

To start uploading your files in the meantime:

1. Download S3Fox for firefox https://addons.mozilla.org/en-US/firefox/addon/3247
2. Restart firefox when it is done
3. Go to Tools>S3 Organizer
4. Where it says Manage account enter an account name
5. Access Key is the access key you got when you registered
6. Secret Key is the other part from when you registered
7. Click close and you should see an empty directory on the right and your local filesystem on the left
8. Right click on the right panel and click "create directory"
9. The directory should be the CNAME you created in the DNS earlier.  So if your site is walkjogrun.net and the CNAME is s3, you should create a directory called "s3.walkjogrun.net".  Note that at the root of S3 your directory names have to be unique across the whole of Amazon's S3 userbase.  This is how they are able to do the CNAME virtual hosting.  As long as you used your own domain name and not mine, you should see a folder created when you hit OK
10. Double click the new directory to go one level down.  This is the root used for your new subdomain so if I create a directory called images I can access them with http://s3.walkjogrun.net/images

A note on terminology is important here.  I have used the term "directory" because that is the level of abstraction provided by s3fox to make it look like an FTP service.  Amazon calls any directories in the root "buckets", and buckets are the things that have to be unique.  Anything lower than the root in a sub-directory doesn't have to have a unique name.  The technical reason for this is that, again, S3fox abstracts the complexity.  In reality, every file you upload becomes a row in a big database as a key and a value.  The key to each file you upload is the unique bucket name concatenated with a string, so by this construction each filename becomes unique too.  It's important to know this if you are going to go one step further and use S3.cfc but for now it's just a bit of extra info.

To add files to the images directory:

1. Double click on the folder to go into it
2. Browse in the left panel to the files you are adding
3. Highlight the file you are adding and either right click and say upload or select multiple files and folders and hit the right arrow in the middle of the two panels
4. Once you see the files have all uploaded below, go up to a directory to the /s3.yourdomain.com/ directory
5. Right click on the directory and hit "edit acl"
6. Ensure that "everyone" has read permissions and check "apply to subfolders"

Any file uploaded to the images directory would then be available under s3.yourdomain.com/images/myimage.jpg etc.  You can right click any file in the right panel and select "Copy URL to clipboard" then paste it into your browser to see if it worked.  

That's it!  Other uses for the service are that you can backup files off your laptop or use it to host large design proofs that would take forever on your local machine.  Did I tell you how fast it is?  WalkJogRun images load 10X faster from Amazon under load than off my humble 10Mb/sec cable connection!

AdamHowitt.com launched

I know, two launch posts in one month but this one is THE most significant for me in that it marks the launch of my solo career as a consultant.  The beautiful design of Adam Howitt Consulting is the handywork of my former colleague and good buddy Jeff Kenny. He always amazes me when he pulls this stuff together.

The site features

• Hcard integration on the contact page through technorati's hcard to vcard converter
• CSS layouts including a print style sheet for the printable scoop
• Google Analytics with Goal Configuration to track conversions, referers, vcard downloads and page effectiveness

My elevator pitch is that I'm offering website management, not website development.  I feel like the market for web developers is saturated and the price of a truly skilled developer has become commoditized.  Instead I'll be focussing on two separate angles targetted at marketers and developers:

1. Campaign management - tracking and optimizing both online and offline ad campaigns
2. Content management - tracking, optimizing and tuning your website content
3. Traffic management - load testing, application troubleshooting and performance tuning

So if you have a problem, if no one else can help and if you can find him, maybe you can hire the a-team.

Bad application variables and the site redesign launch

I spent most of monday ripping my hair out trying to work out why the redesign of the Chicago Park District website was crashing each server systematically.  The application is over 4 years old and the latest round of changes were to reskin the pages and add new content to the home page. After 4 hours of troubleshootizing :-) and load testing the application in a pre-production environment I found the issue.

Somehow, the application variables were being recreated with each request, in this case that included the rssHandler CFC responsible for pulling in the RSS feeds driving parts of the page.  What I spotted was that the cfapplication tag typically set in the application.cfm was not set there but in fact later on the fbx_settings.cfm file after some environment specific code had been processed (am I in dev, stage, preprod or live).  As soon as I changed the code around my load testing results pivoted sharply from 20 concurrent users killing the server to maxing out our load testing setup. 

Some of the limitations you encounter when attempting your first load test:

1. Client machine pipe size.  Regardless of your load testing tool you are limited by the bandwidth connecting your client to the internet so you may struggle to simulate over 180 users remotely
2. Client IP stack and socket limitations.  A standard windows XP install typically only supports about 200 users before you see socket errors.
3. Client software.  Trial licenses of load testing software can limit the number of users you can simulate so make sure you understand the limitations of the tool.
   
4. Web server pipesize.  Even if your client has a fat pipe to connect to the internet, if your webserver is on a 10 mbps switch or is on a T1 it will restrict the number of users it supports.  With 10 mbps it has to support inbound traffic, outbound traffic and if you have a database server on a separate machine (which you should) it will also have inbound and outbound requests there too.  My suggestion would be to use a good SNMP graphing tool like Cacti to monitor each port on the webserver and your infrastructure so you can easily spot the point where your webserver has maxed out the pipe.

The load testing tools need not cost a small fortune and once you have a set of load scripts for a client, it helps with regression testing to make sure your funky new piece of code on the home page hasn't degraded the performance of your application to a crawl.

Microsoft has a free tool called Web Application Stress Tool (WAST) which we have used on a occasion to generate large loads.  High end load testing tools typically have a master machine and then clients installed on multiple machines to distribute load over several IP addresses.  For WAST, simply have some of your team members work at home or log on one evening for an hour, co-ordinate the tests over IM and all push the big red button when the test leader says go.  This means 8 people can generate a load of 200x8= 1600 concurrent users. 

WAST doesn't do very well at ramping up the number of users so we have licenses for Paessler's Load Test tool at $249 per seat which allow us to simulate slightly more users and create a better ramp up profile to warm up the servers before nailing them.  It also has a better interface and, if your infrastructure supports it, the potential to generate up to 10,000 concurrent users per machine.

The other benefit of this low budget load testing approach is that your team are accessing the site from geographically diverse locations similar to the way your site will be accessed in the wild.  This introduces elements like increased request latency and also means your load balancer tool (if you have one) is more likely to distribute the load more evenly than it would coming from the same IP address or local network, depending on your configuration.

I hope this was useful since I don't see much on the CF lists in the way of practical load testing on a budget.  If anyone would like to see more on this topic let me know since I have done quite a bit over the last year while working with the Chicago Park District site.  Once per quarter we run tests to ensure that the registration application section can support 3000 registrants in under 3 minutes.