Adam Howitt's Blog

Run a Linux process on a remote server and disconnect terminal session

I've been writing some nifty python scripts lately as we (WalkJogRun) migrate our route data from XML files to Google Maps encoded polylines. I finally whipped the script into shape and started pulling our route data from Amazon S3, encoding as SQL update statements and then rinse and repeat for nearly 500,000 legacy XML files. Trouble is, 500,000 files on my local machine would take about 20 hours.

Amazon AWS to the rescue! I span up a new Amazon EC2 medium size instance (high CPU) and pushed my script up to the cloud. The beauty of this is that the bandwidth between EC2 and S3 is free whereas my local machine relied on a crappy network connection and incurs the bandwidth charge on our S3 account. Ordinarily S3 bandwidth is pretty cheap but the script running on our EC2 machine is pulling down 5gb of data per minute and processing 1,000 files per minute. In around 8 hours we should have every legacy file processed and converted to a SQL statement we can run against our MySQL database.

But wait. My terminal timed out. Doh. If I login to the server over terminal and start the python script it is connected to my shell so if the connection breaks (on my crappy internet connection) the process is terminated on the server. The batch isn't as smart as it could be so it doesn't gracefully restart where it left off so I called on my friendly linux whiz Scott Frazer for a solution.

Scott directed me to the NOHUP command and after a few minutes of reading, a little typing and restarting my first batch I was able to disconnect my session thanks to this helpful command:

Roughly, nohup says don't hangup when I do but because my script outputs to the terminal as it runs the rest of the output needs to be redirected to files instead of the console.

If you run this command, exit and log back in you'll see the PTS/1 replaced by ? when you run the ps -fe command. I'm able to track the progress of my script because each iteration/file writes a line to the sql file it generates and the handy wc -l command tells me how many lines are in that file.

So thanks to Scott I'll have all my new shiny encoded polylines for our database and we can start to eliminate the need for the legacy XML files. The background there is that Google Maps on the iPhone and in the browser both support polyline encoding such that instead of a large xml file of latitude and longitude coordinates (some as big as 40kb!) I get an encoded string I can insert into the database each no more than 1000 characters.

/dev/hdd1 -- converted during upgrade to edgy

I recently upgraded one of my Ubuntu servers from Dapper to Hardy Heron one upgrade at a time as is recommended all over the place.  In the course of doing the first upgrade, I missed an important note in the fstab file to say that my hard disk had been converted during the the upgrade.   I only found out when Apache failed to start due o a missing log file location. 

Upon further inspection, the log file located on a mountpoint /media/bigdrive wasn't there.  I looked at the output of

to see if it was still there and noticed that the volume was now described as lrm.  I ran fdisk list: and it reported /dev/sdb was available and still 23% full so I knew my data was intact.

 

I tried to mount it but got the message /dev/sdb1 already mounted or /media/bigdrive busy.  After some Googling I found a thread on the Ubuntu forms where someone noted that /dev/sdb needs to be changed to /dev/mapper/sdb to get it to work:

 

...and there was much rejoicing. 

VMWare Workstation on Hardy Heron

On a whim I upgraded my Ubuntu Gutsy install to Hardy Heron yesterday and while most things are working, a key component to allow me to run my billing was not.  Since Quickbooks Online only supports IE and iPhone browsers (don't get me started about that genius decision), I run VMWare Workstation with XP as a guest OS.

The upgrade went smoothly but when I tried to open VMWare it kept crashing without loading.  A quick scout around and I found this great thread with DirkCoopman's solution.  Basically you edit one file in vmmon.tar, tar it back up and re-run vmware-config.pl.  I'm running Hardy Heron 8.04 and VMWare Workstation 6.0.3 build-80004.

Google Sitemaps made easy with Linux

I just discovered that Google Webmaster now allows you to register Google Sitemaps in a variety of formats: RSS, Simple text or sitemap XML format.

This may not be exciting news to most but the simple text format makes life drastically simpler in terms of an entry point to creating a sitemap. 

I was about to fire up a text editor and some tunes to rip through a site to manually collect the page names when I realized that the find command in linux will spit out a carriage return separated list.  A quick

yielded the foundation of what I needed.  Note that backslashes used to escape special characters.  This gave me output as follows:

The last step was to use the substitute sed command to replace the ./ at the start of the string with the site name and pipe it into the sitemap.txt file:

AMD 64 bit Troubles with Ubuntu 7.10

Just a teaser of a post to see if anyone else has run into issues with Ubuntu 7.10 and 64 bit architectures. 

I built a machine from scratch on Sunday after months of piece by piece purchases and readily installed Ubuntu 7.10 64 bit server version.  4 gig of Corsair RAM, 250 GB hard disk, AMD X2 6400+ chip and a Gigabyte motherboard to support it all in a Rosewill case.  Connected up to my LAN and SSH'd into the box no problems.

My first issue came when I tried to install ColdFusion 8 with the same old CF7 approach for Ubuntu thanks to Dave Shuck's guide. The first step appeared to go well of installing zlib, but Apache 2.2 from source build failed at the zlib part.  It turns out that the mod_deflate piece of Apache that relies on zlib's gzip was struggling to install.  Eventually I worked around this by installing Apache with --disable-deflate option to work around it.  Ultimately Dave told me the installers are now better at working with the apt-get installed Apache 2 config.

Issue number 2 came when I went to install ColdFusion 8 and the installer chokes.  Not in the same replace the OS detection in the installer way - no. Somehow the java executable doesn't work and I found comments from an Adobe person suggesting a co-flounderer join the prerelease program for the CF8.1 update featuring a linux 64 bit installer.  Hmph.  So I joined but haven't received anything yet.

Lastly today I ran into trouble installing subversion on the box so I'm getting stuck all over the show.  If you have any experience with any of these on linux 64 bit hardware, please post some links in the comments for me to dive into.

Build a scalable architecture with Amazon's EC2

I've just wrapped up my CF Online Meetup live presentation of using Amazon EC2 to build a scalable architecture to run ColdFusion.  In the presentation you will learn what you need to get started with Amazon EC2 and I demonstrate a machine image I built running ColdFusion 8 and Apache.  I talk about the pricing and some of the challenges you may face.  

Over 30 people turned up to see the talk but if you missed it you can view the recording here. and Charlie has added the presentation to his UGTV site as he does with every CF Online Meetup.

Due to licensing concerns I haven't made the machine image running CF8 publicly available yet.  I'll give Vince at NewAtlanta a nudge and if anyone can put me in contact with the right people to ask at Adobe I may well make it public.  This will mean you that as long as you have the tools I mention in the presentation (all free), you can fire up my ubuntu704 machine instance which is pre-installed with Apache, go to the downloads directly and choose whether you want to install CF8 or BD7 and then follow the prompts to the end!  You can try the 30 day trial and evaluate your applications on dedicated hardware.  One of the tricks with trying this yourself is that Adobe and NewAtlanta require you to login to download the software so you can't just ftp the file into Amazon from their site.  I had to pull it locally after logging in then push it up to Amazon with SCP/FTP.

I think this could be a huge selling point for both Adobe and NewAtlanta - the ability to test the server without committing hardware or having to install anything.  Who knows - maybe one of them will even create their own pre-installed time limited demo image people can use where you just fire it up, copy up your files with WinSCP and point your browser!

As always - any questions let me know below.

How Amazon Saved My Marriage

I moved all my websites from a hosting company in Georgia to hosting them myself after learning Ubuntu nearly two years ago.  Little did I know that the huge P3s with whining fans would take up so much closet space and cause so much trouble.

I'm talking at CFUnited Express on September 30th in Chicago about how Amazon's webservices, specifically S3 and EC2, have enabled me to migrate my applications onto dedicated high end hardware for a low monthly rate.  I'll talk about the benefits, the drawbacks and some of the challenges you will face getting up and running including some tools you may want to use.

WalkJogRun on Fox News

I'm delighted to report that WalkJogRun was featured on Fox News two days ago for the Cool Websites section after I tipped off Nancy Loo about the site.  I was surprised in her coverage when she said she had already featured the site at one point but I missed it I guess.  This time I was prepared and have my first YouTube video up now.  I know, I'm probably the last person in the tech world to do this but this is the first thing I felt was worth it.

For anyone who hasn't read anything about WalkJogRun, it's over 3 years old and is a community for runners, cyclists, walkers and anyone else to create and share routes.  It is based on Google Maps and uses custom AJAX, ColdFusion and is built on BlueDragon 6 with MySQL on an Ubuntu server. 

 

BlueDragon 7.0 working with Apache 2.2 on Dapper Drake

I've just got this working. (to be fair i can display the default bd test page and my own hello the time is #now()# page but I am running 2.2 on Ubuntu with BD 7.0 beta 2.  Beta 2 isn't on the site yet but is available from the ftp servers.  Check the bd-interest list archives for how to find it.

The problem I came across may have been related to the order of my installation but it went thusly:
1. install apache2
2. install bd
3. test bd port 8080 - happiness
4. try to follow the guidelines from 6.21 docs for making symbolic links to the apache 2 install.  On ubuntu they are:

5. Confirm that you have the file /usr/lib/apache2/modules/mod_servletexec2.so
6. go to /etc/apache2/mods-enabled directory and see if you have 2 files servletexec.load and servletexec.conf.  If not create servletexec.load with the following line:
next create servletexec.conf as follows:

7. Restart apache:
8. Create a simple index.cfm in your apache webroot (e.g. /var/www)
The time is <cfoutput>#now()#</cfoutput>
9. as long as you are not using virtual hosts you should see the current time instead of the plain text

Hope this helps others!  For reference, I have been working with VMWare's free virtual machine player this week for the purpose of testing BD7beta2 and have an Ubuntu Dapper drake vm running apache 2.2 and BlueDragon 7 beta 2. You can build your own Virtual machines with a free site called EasyVMX (see the super simple edition) to allow you to create a virtual machine and install an OS from your boot disks.

I installed Dapper Drake on one of these images to get the configuration working.  I'm done now and
I'm not sure whether BD will let me distribute this or would care to distribute it themselves but it's a good starting point for evaluating a web server without compromising any of your existing setup.

CFUnited and BlueDragon 7

I'm excited to be heading to my first CF conference since Macromedia's DevCon 2002 in Florida.  After that conference the company I was working for hit hard times and then I moved back to Chicago.  I focussed my training budget on classes last year but this year, it was time for a conference.  The training environment is completely different than classroom training at a conference.  Not only do they offer the opportunity to cover many topics but they also offer different levels from beginner topics to advanced including areas not traditionally covered in classes.

One presentation I won't be missing is Vince Bonafonti's BlueDragon 7 keynote  where the latest release is introduced.  He has been offering sneak peeks over at his blog this week.  I've been using BlueDragon 6 to run WalkJogRun for 3 months now on Apache and Ubuntu with MySQL and it is doing great. BlueDragon has happily handled the 30,000 page views per day on a simple P3 box with 256Mb memory!  Yes, it's an old converted desktop but that hasn't caused any problems so far.  Not bad for free!  Now if I can just work out a way to make money from the site without resorting to adwords I'll be golden.