Thursday, May 29, 2008

Deploying Rails on Windows

Why would you want to deploy Rails on Windows? With tools like Capistrano, surely deploying on Linux is the obvious choice. Well there are (at least) two valid reasons to deploy Rails on Windows:

  1. You don't have a choice. Many companies are Windows shops and maintaining a Linux/OS X/other server is simply not an option. This situation is becoming less common as Linux gains popularity in the server room. However, depending on the scale of your Rails app, you may have to share a server with other services, and this reason still holds.
  2. You use Microsoft's SQL Server as your Rails database. This is probably not common for new Rails apps in completely new systems (where you have your choice of SQL servers), although SQL Server is still a good product and sensible choice. Rails connects to SQL Server through ADO (at least, this is the method suggested by AWDwR), which is only available on Windows.

I know there are many guides on running Rails, but I didn't find one that covered this particular configuration in detail: LightTPD <- SCGI -> Rails <- ADO -> SQL Server on Windows, running as services. This guide covers vanilla Windows XP/Windows Server 2003 to working web server. It assumes you have a Rails app working in a development environment (such as RadRails), connecting to an existing SQL Server, and committed to an existing SVN server.


Gather and Install Necessary Software

  1. Ruby (v 1.8.6) - Download the Ruby One-click Installer for Windows. Install to C:\Ruby\ and include Gems.
  2. LightTPD (v 1.4.18-1) - Download the pre-compiled LightTPD web server (.exe), part of the WLMP project. Install to C:\LightTPD\.
  3. Rails (v 2.0.2) - Install Rails via RubyGems. Open the command line and type:
    gem update --system
    gem install rails
  4. Zed Shaw's SCGI Rails Runner (v 0.4.3) - Download the gem from Shaw's blog (click the link labeled "gem"). Install .gem file using RubyGems; again, open the command line and type:
    gem install C:\path\to\scgi_rails-0.x.x.gem
    Replace the path in the above command with the location/file you downloaded the .gem to.
  5. Subversion (v 1.4.5) - Download the SVN Windows Installer. Install to the default location (C:\Program Files\Subversion); this will install the SVN command-line tools.
  6. Windows Server 2003 Resource Kit Tools - Download rktools.exe. Install to the default location (C:\Program Files\Windows Resource Kits\Tools).
  7. Ruby/DBI ADO (v 0.1.1) - Download the ruby-dbi package from RubyForge. Extract archive somewhere using your favorite tool (such as 7-zip). Install the package from the command line:
    cd C:\path\to\ruby-dbi\
    ruby setup.rb config --with=dbd_ado
    ruby setup.rb setup
    ruby setup.rb install
    Replace the path in the above command with the location where you extracted ruby-dbi.
  8. Rails SQL Server adapter gem. This step is optional if you freeze this gem into your application as described below.

I had to fix the SCGI Rails gem to work with Rails 2.0; you may not for later versions of the SCGI Rails Runner. Open the file C:\ruby\lib\ruby\gems\1.8\gems\scgi_rails-0.4.3\bin\scgi_service using your favorite Ruby editor (or just WordPad), and comment out line 36,

    ActiveRecord::Base.threaded_connections = false

so that it looks like this:

    #ActiveRecord::Base.threaded_connections = false

Apparently this method is no longer available in Rails 2.0.  I have not run into any issues removing this line, but if anyone has any suggestions for fixing it for Rails 2.0, let me know.

Note: Since we're using the command line a lot in this guide, it makes sense to leave the window open. At this point however, you should close all command line windows, because some of the installers add locations to your PATH variable, which only affects new command line windows.

Prepare Rails Application

This guide suggests a configuration to host multiple Rails applications from a single Lighty server. We create a main folder to contain all Rails apps; inside, we put a log folder for the Lighty logs, and a folder for each Rails application. In my particular setup, I use this as a way to run the same intranet Rails app in production and development modes for testing purposes before rolling out changes.

Open a command line and type:

cd C:\
mkdir webapps
cd webapps
mkdir log
svn export svn://your-svn-server/railsapp1/trunk railsapp1
cd railsapp1
scgi_ctrl config -S

Replace "your-svn-server" with the name of your svn server, and "railsapp1" with the name/svn path of your particular Rails application. Note: the last command will ask you for a password. I have no clue what this password is for, so I don't think it's too important. If you know what the password is for let me know.

The scgi_ctrl command creates a new file in your rails app at C:\webapps\railsapp1\config\scgi.yaml. For additional apps, you will need to alter this file. For example, to run the app in development mode, you would change the file to look like this:

--- 
:disable_signals: true
:env: development
:control_url: druby://127.0.0.1:8998
:config: config/scgi.yaml
:host: 127.0.0.1
:port: 9998
:password: LeaveThePasswordAlone
:logfile: log/scgi.log

The useful options are highlighted above. You must specify a different port for each Rails app to run on (LightTPD redirects requests to these ports). You may also change the env setting to a different environment, based on your needs.

Configure Lighty

Copy the folder C:\LightTPD\conf to C:\LightTPD\conf-backup so that you have the original configuration files for future reference. Open C:\LightTPD\conf\lighttpd-srv.conf in a text editor (again, Wordpad works). Edit the file to match the following:

# LightTPD Configuration file (RUN AS A SERVICE)
#
# Use it as a base for LightTPD 1.0.0 and above.
# This version is built for WLMP Project - http://wlmp.dtech.hu/
#
# $Id: lighttpd-srv.conf,v 1.0 2006/11/03 23:35:28 weigon Exp $

## where to send error-messages to
server.errorlog             = "C:/webapps/log/lighttpd-srv.error.log"

#### accesslog module
accesslog.filename          = "C:/webapps/log/lighttpd-srv.access.log"

## to help the rc.scripts
#server.pid-file            = "C:/LightTPD/logs/lighttpd-srv.pid"

#### include other configfiles
include "C:/LightTPD/conf/lighttpd-tag.conf"
include "C:/LightTPD/conf/lighttpd-inc.conf"

The important changes are highlighted above. Providing absolute paths becomes necessary when running Lighty as a service.

Next, open C:\LightTPD\conf\lighttpd-srv.conf in your text editor. Replace it with the following, and edit to suit your specific paths:

# LightTPD Configuration file (INCLUDE)
#
# Use it as a base for LightTPD 1.0.0 and above.
#
# This version is a stripped down version for use with SCGI, Rails, and 
# multiple hosts on ports.
#
# $Id: lighttpd-inc.conf,v 1.7 2004/11/03 22:26:05 weigon Exp $

############ Options you really have to take care of ####################

## Always has to be a default root
var.home = "C:/webapps"
server.document-root = var.home + "/railsapp1/public" 

## modules to load
server.modules              = (
                                "mod_access",
                                "mod_accesslog",
                                "mod_alias",
                                "mod_redirect",
                                "mod_rewrite",
                                "mod_scgi",
                                "mod_ssi",
                                "mod_status",
                               )
## App1 Configuration

var.app1 = var.home + "/railsapp1"

$SERVER["socket"] == ":80" {
    server.port                     = 80
    server.document-root            = var.app1 + "/public"
    server.upload-dirs              = ( var.app1 + "/tmp" ) 
    server.errorlog                 = var.home + "/log/app1-errors.log" 
    accesslog.filename              = var.home + "/log/app1-access.log" 
    static-file.exclude-extensions  = ( ".cgi", ".fcgi", ".scgi" )
    server.error-handler-404        = "/dispatch.scgi" 
    scgi.server = ( "dispatch.scgi" => ((
        "host" => "127.0.0.1",
        "port" => 9999,
        "check-local" => "disable" 
    )) )
}

## App2 Configuration

var.app2 = var.home + "/railsapp2" 

$SERVER["socket"] == ":3000" {
    server.port                     = 3000
    server.document-root            = var.app2 + "/public"
    server.upload-dirs              = ( var.app2 + "/tmp" ) 
    server.errorlog                 = var.home + "/log/app2-errors.log" 
    accesslog.filename              = var.home + "/log/app2-access.log" 
    static-file.exclude-extensions  = ( ".cgi", ".fcgi", ".scgi" )
    server.error-handler-404        = "/dispatch.scgi" 
    scgi.server = ( "dispatch.scgi" => ((
        "host" => "127.0.0.1",
        "port" => 9998,
        "check-local" => "disable" 
    )) )
}

## Applies to all Apps

## files to check for if .../ is requested
# index-file.names            = ( "index.html", "index.htm", "default.htm" )

# mimetype mapping
mimetype.assign             = (
  ".pdf"          =>      "application/pdf",
  ".sig"          =>      "application/pgp-signature",
  ".spl"          =>      "application/futuresplash",
  ".class"        =>      "application/octet-stream",
  ".ps"           =>      "application/postscript",
  ".torrent"      =>      "application/x-bittorrent",
  ".dvi"          =>      "application/x-dvi",
  ".gz"           =>      "application/x-gzip",
  ".pac"          =>      "application/x-ns-proxy-autoconfig",
  ".swf"          =>      "application/x-shockwave-flash",
  ".tar.gz"       =>      "application/x-tgz",
  ".tgz"          =>      "application/x-tgz",
  ".tar"          =>      "application/x-tar",
  ".zip"          =>      "application/zip",
  ".mp3"          =>      "audio/mpeg",
  ".m3u"          =>      "audio/x-mpegurl",
  ".wma"          =>      "audio/x-ms-wma",
  ".wax"          =>      "audio/x-ms-wax",
  ".ogg"          =>      "application/ogg",
  ".wav"          =>      "audio/x-wav",
  ".gif"          =>      "image/gif",
  ".jpg"          =>      "image/jpeg",
  ".jpeg"         =>      "image/jpeg",
  ".png"          =>      "image/png",
  ".xbm"          =>      "image/x-xbitmap",
  ".xpm"          =>      "image/x-xpixmap",
  ".xwd"          =>      "image/x-xwindowdump",
  ".css"          =>      "text/css",
  ".html"         =>      "text/html",
  ".htm"          =>      "text/html",
  ".js"           =>      "text/javascript",
  ".asc"          =>      "text/plain",
  ".c"            =>      "text/plain",
  ".cpp"          =>      "text/plain",
  ".log"          =>      "text/plain",
  ".conf"         =>      "text/plain",
  ".text"         =>      "text/plain",
  ".txt"          =>      "text/plain",
  ".dtd"          =>      "text/xml",
  ".xml"          =>      "text/xml",
  ".mpeg"         =>      "video/mpeg",
  ".mpg"          =>      "video/mpeg",
  ".mov"          =>      "video/quicktime",
  ".qt"           =>      "video/quicktime",
  ".avi"          =>      "video/x-msvideo",
  ".asf"          =>      "video/x-ms-asf",
  ".asx"          =>      "video/x-ms-asf",
  ".wmv"          =>      "video/x-ms-wmv",
  ".bz2"          =>      "application/x-bzip",
  ".tbz"          =>      "application/x-bzip-compressed-tar",
  ".tar.bz2"      =>      "application/x-bzip-compressed-tar"
 )

# Use the "Content-Type" extended attribute to obtain mime type if possible
mimetype.use-xattr          = "enable"

######### Options that are good to be but not neccesary to be changed #######

## enable debugging
#debug.log-request-header   = "enable"
#debug.log-response-header  = "enable"
#debug.log-request-handling = "enable"
#debug.log-file-not-found   = "enable"

#### SCGI module
scgi.debug                  = 0

#### status module
status.status-url           = "/server-status"
status.config-url           = "/server-config"

#### url handling modules (rewrite, redirect, access)
#url.rewrite                = ( "^/$"             => "/server-status" )
#url.redirect               = ( "^/wishlist/(.+)" => "http://www.123.org/$1" )
#### rewrite to pick up page cache
url.rewrite = ( "^([^.]+)$" => "$1.html" ) 

Again, portions that you will most likely need to change are highlighted above. Be sure to provide absolute paths.

Finally, with Lighty configured, we want to run that Rails app! Copy the following into a text editor and save it to C:\LightTPD\Start_LightTPD.bat:

@echo off
c:
cd C:\lighttpd\
echo Starting lighty...
START /B lighttpd.exe -f conf\lighttpd-srv.conf -m lib -D
PAUSE >NUL && EXIT

And save the following to C:\LightTPD\Start_RailsApp1.bat:

@echo off
c:
cd C:\webapps\railsapp1
echo Starting RailsApp1 SCGI_service...
START /B scgi_service
PAUSE >NUL && EXIT

As always, adjust to suit your particular Rails app. You will want to create a runner batch file for each Rails app you wish to run. (unless you run as services as described below)

Now just run each batch file. A command window should appear for each, print the "Starting x..." line, and stay open. Pop open a web browser, and type http://localhost/ in the address bar. If your Rails app comes up, congratulations! If not, you've got some troubleshooting to do. Check the command windows for any errors; you can also look at the logs in C:\webapps\log and C:\webapps\railsapp1\log for errors and additional info.

Setup LightTPD and Rails as Services

Once you have Lighty and Rails running from batch files, you may want to run them as services. Services have two major advantages: they keep running when you logout, and they can run with restricted security privileges.

We are going to use the srvany program that comes with the Windows Resource Kit to run both Lighty and Rails as services. srvany acts as a wrapper that allows regular applications to run as a Windows service. Open a command line and type the following:

instsrv lighttpd "C:\Program Files\Windows Resource Kits\Tools\srvany.exe"
instsrv railsapp1 "C:\Program Files\Windows Resource Kits\Tools\srvany.exe"

Create a single service for LightTPD, and one service for every Rails app you wish to run. Just change the name of the service (highlighted above).

This creates two new services, named "lighttpd" and "railsapp1" that both run srvany. Now we edit the registry to configure what srvany runs. You can do this manually (using the values listed below), or use a text editor to save the following to lighttpd.reg (doesn't matter where you put it):

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lighttpd\Parameters]
"Application"="C:\\LightTPD\\LightTPD.exe"
"AppParameters"="-f \"C:\\LightTPD\\conf\\lighttpd-srv.conf\" -m \"C:\\LightTPD\\lib\" -D"

Save the following to railsapp1.reg (or name of your Rails app):

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\railsapp1\Parameters]
"Application"="C:\\ruby\\bin\\ruby.exe"
"AppParameters"="C:\\ruby\\bin\\scgi_service"
"AppDirectory"="C:\\webapps\\railsapp1"

Save a copy of the .reg file above for each Rails service, changing the registry path and AppDirectory value to point to your service name and Rails app location.

Finally, double-click each of these .reg files to add their contents to the registry.

By default, new services run under (or "log on as") the Local System account. This should be changed to a low-privilege account, in case the Rails environment or LightTPD is compromised. We want to run all of the services in this Guide under the Network Service account.

Open the Services dialog at Start > Control Panel > Admistrative Tools > Services. Find the service "lighttpd" that we created, and double-click it. Click the "Log On" tab at the top of the properties dialog, and click "This account:" radio. Type "NT AUTHORITY\NetworkService" into the account field, and clear the password and confirm fields. Click "OK" in the properties dialog. Repeat these steps for each Rails service you have created.

One last step is needed to get the services to run. Both LightTPD and the SCGI Rails Runner will crash if they don't have write permissions to their respective log directories. (As a bonus, when running as services under srvany, they crash silently, not even appearing to have stopped, and having no way to report the error) Using Explorer, navigate to C:\webapps\, right-click the log folder, and select "Properties". Click the "Security" tab, and click "Add...". Type "Network Service" into the object names field, and click "OK". Back in the Properties dialog, make sure "NETWORK SERVICE" is selected in the upper user name list, and check the box in the lower permissions list to allow writes (row "Write", column "Allow"). Click "OK" on the properties dialog. Now navigate to C:\webapps\railsapp1\, etc. and repeat for the log folder inside each Rails app.

Now to test the services. Make sure to close any open command windows that are running LightTPD and/or your Rails apps (such as the batch files created above). Start the services using the Services dialog (Start > Control Panel > Admistrative Tools > Services) or from the command line:

NET START lighttpd
NET START railsapp1

At this point, I'd recommend starting Lighty and just one Rails app, then checking if it is running by pointing a browser at http://localhost/. If it works, start up and test the rest of your Rails apps and call it a day! If you get a blank page, you will have to troubleshoot your configuration. The only good way to do that is to run cmd using srvany. One gotcha: Microsoft's solution uses Local System with the "interact with desktop" feature; however, this only works when you are logged directly into the machine, not over a Remote Desktop connection. Once you have cmd running as a service, try running your batch files from before and troubleshoot any errors reported.

Automating deployment

A couple tips to automate deployment:

  • Commit config\scgi.yaml to SVN, so that you don't have to run scgi_ctrl config when exporting future versions of your webapp. If you need multiple versions of scgi.yaml (say, for different environments), create a deploy folder in your Rails app, then create deploy\scgi-prod.yaml and deploy\scgi-dev.yaml; copy one of these to config\scgi.yaml at deployment.
  • Create a batch file to automate the process, and create a shortcut to said batch on your desktop. This gets you fairly close to one-click deployment (SVN commit, RDC to server, run batch). A sample deploy script might look like this (save in C:\webapps\):
    @echo off
    echo Stopping Service
    NET STOP railsapp1
    sleep 1
    echo Deleting old copy
    rd /s /q railsapp1
    echo Exporting new copy from SVN
    svn export svn://your-svn-server/railsapp1/trunk railsapp1
    echo Configuring deployment
    copy railsapp1\deploy\scgi-prod.yaml railsapp1\config\scgi.yaml
    echo Starting Service
    NET START railsapp1

Friday, April 11, 2008

Advanced sum() usage in Rails

Active Support adds a nice sum() method to Enumerable, which is mixed into Array. Rails also has a nice extension to Ruby symbols that lets you call a method that normally takes a block like this:

total_price = items.sum(&:price)

You may be thinking that this only works for numbers. However, because Ruby is dynamically typed, we can actually take the sum of any type that implements the + method. (In Ruby, even mathematical operators are methods, just as numbers are objects) That means we can use it to concatenate arrays (array.+ is concatenation). This can come in handy when working with multiple Active Record has_many relationships:

firm_invoices = @firm.clients.to_a.sum(&:invoices)

To be fair, this probably isn't the most efficient way to do this. The example above is equivalent to the following usage of has_many :through from the Rails API:

@firm.clients.collect { |c| c.invoices }.flatten
@firm.invoices # defined by has_many :through the Client join model

Sometimes life isn't as easy as examples. In my case, we have customers that can belong to multiple offices (they are mobile and do work from each). So a customer has_and_belongs_to_many (habtm) offices (in SQL terms, this is a many-to-many relationship via a join table). In our Rails project, the join table does not have a model object, as it contains only the data to define the relationship. An office, in turn, has_many projects (our main unit of work; a single request from a customer). My goal was to report all of the projects for the offices of the logged-in customer. With sum(), it's easy:

projects = @customer.offices.to_a.sum(&:active_projects)

One thing I don't like about this is that it is hard to read. Summing projects doesn't make verbal sense. On the other hand

projects = @customer.office_active_projects

smells funny. Why should Customer have specific knowledge of how to get Projects for Offices?

In the end, summing arrays is a useful method to have in your tool belt. It can be used to quickly drill through relationships (without altering model objects) to make sure you get the results you are looking for, before making your code more efficient/elegant/permanent.

Thursday, March 20, 2008

DD-WRT Firmware on Buffalo WHR-G54S

I've had the Buffalo WHR-G54S router for about a year, but tonight I decided to install the popular third-party firmware DD-WRT.  I knew the router was compatible with DD-WRT when I purchased it (that was part of the reason I selected it).  I highly recommend the WHR-G54S; it has worked well for as long as I've had it.

Actually, it hasn't performed flawlessly.  In fact, I've been having problems since my new roommate moved in.  He likes to P2P when he's not here (which is usually when I'm here and want to use the bandwidth).  Tonight when I got home, my computer was having major connectivity issues.  Even after stopping his downloads, the problems continued.  I had to power-cycle my cable modem and router just to get a simple page to load.  Before rebooting it, even the web-configuration tools on the router couldn't load (so the issues were in the router?!?)  That's why after a year, I decided to give DD-WRT a shot.  Hopefully it will improve my connection (or contain tools to shape the P2P traffic/give me priority)!

I just wanted to give some pointers that helped me through the process:

  • As this article points out, dd-wrt.[version]_mini_generic.bin is the firmware file you want to use when first flashing from Buffalo's factory firmware.  That means you want to download dd-wrt.[version]_mini.zip from the DD-WRT Downloads Area.  At the time of writing, the latest stable version was 2.3_SP2.
  • The DD-WRT Wiki Installation Page was very helpful, especially the Precautions and Buffalo > TFTP Flashing Buffalo Routers under Windows sections.  I followed the instructions from the latter and successfully flashed the router on the second try. (the first try, I jumped the gun when the red light came on; when you see the first ping as the router comes back up hit enter immediately, I only got one successful ping during the TFTP window the first time around)
  • My reset button is labeled "init" and sits on butt end of the router, next to the "bridge" switch and directly opposite the antenna jack.  You will need something thin like a paperclip to press it.
  • When unplugging and re-plugging my router, I used the router (DC power) end of the power cable, since this was easier to do with the router sitting on the desk next to my keyboard.  I unplugged everything else except a singe Ethernet cable from my computer.
  • My Ethernet port (nForce 4 built-in) is auto-sensing, but came back well before the TFTP ping.

The installation went so smoothly, I'm wondering why I waited this long to try DD-WRT.  The answer is, "If it ain't broke, don't fix it!"  Hopefully I'll have more to say when I've had a chance to play with some of the firmware's more advanced settings.  I can say that even the "mini" version has all of the features I used in the original Buffalo firmware.  Unfortunately I forgot to write down the port-forwarding settings I had for serving HL2:Deathmatch and Synergy internet games.  Oh well, I'm sure I can find the ports again.

The dream app?  Setting up a Cron job that pipes fortune to net send on the hour.  Imagine - every computer on the network (or maybe just your roommate's) gets a dialog box with a fortune every hour, courtesy of your local router and DD-WRT.  Priceless.

Monday, March 17, 2008

Email2Trac on Windows

See the official Email2Trac documentation for an up to date version of these instructions.


If you're not familiar with Trac: Trac is an integrated issue tracking system, wiki, and SVN repository browser. It is open source software (under the very liberal BSD licence), and is popular as a public bug tracker for many open source projects. Email2Trac is a plugin (actually, more of a script) for Trac that enables users to submit bugs via email.

Email2Trac was clearly created by *nix users, but since it is written in Python, can be made to work in Windows without too much hassle. This post is a summary of a couple messages I posted to the Trac Users Group detailing our Email2Trac setup.


Platform: Windows Server 2003, Python 2.4, Trac 0.10.3, email2trac 0.10 (year+ old install)

There are 3 pieces you need to effectively turn emails into Trac tickets:

  1. A way to receive emails on your Trac server
  2. A way to process or convert emails into Trac tickets (this would be the email2trac python script)
  3. A way to automate part 2

Here's what we use on Windows:

Receiving Emails via SMTP

We use Windows' built-in SMTP server to deliver mail to our Trac server. When I setup email2trac, I thought that this step was going to be the most difficult (mostly because I have no experience with email servers/services). Surprisingly, it is actually the easiest step. The trick is to install the SMTP server, but not the POP server, so that emails are received and left as files in the "drop" folder. (This is the way an email server hands off email between the two services) In a way, email2trac performs the distribution functions normally handled by POP.

Install the SMTP service via Windows Components.

  1. Open Add/Remove Programs (Start > Control Panel > Add or Remove Programs)
  2. Click "Add/Remove Windows Components" in the left-hand bar.
  3. Use the "Details..." button to drill-in to Application Server > Internet Information Services (IIS) > SMTP Service. In Windows XP, it's just Internet Information Services (IIS) > SMTP Service.
  4. Check the box next to SMTP Service, click "OK", "OK", "Next", "Finish".

At this point, you should ensure that the SMTP Service is running (on my workstation it didn't start after installation; this may be a policy issue).

  1. Open up the Services dialog at Start > Control Panel > Admistrative Tools >Services.
  2. Scroll down to "Simple Mail Transfer Protocol (SMTP).
  3. If startup type is not "Automatic", double-click the service. Change "Startup Type" to "Automatic". Click "OK".
  4. If status is not "started", click play button in services dialog toolbar to start the service.

The SMTP service should be configured and ready to receive emails out of the box. You can find SMTP settings at Start > Control Panel > Admistrative Tools >Internet Information Services (IIS) Manager. The installation automatically creates the default domain "Local (Default)" and the corresponding drop folder at C:\Inetpub\mailroot\Drop. You can see that my email2trac batch script picks up .eml files from that location.

To actually receive emails on the server, you have to send to an address of the form "anything@trac-server.domain.com". Your Exchange or other mail server should automatically forward the emails to your Trac server. If you want to use a different address, you will likely have to configure your mail server to forward the mail. The name before the @ can be anything; POP service usually uses this to distribute mail to inboxes, SMTP and email2trac ignore it. A possible enhancement to email2trac would be to use the local-part of the address to identify the destination environment for the ticket. You will probably want to give out a single address, just to keep things consistent; since it is our Help Desk's Trac, we use "helpdesk@trac-server.domain.com".

Alternative Way to Receive Emails

If the server that hosts Trac already handles emails (thus the SMTP domain "trac-server.domain.com" is taken), you will need a different method to receive emails. Instead of using SMTP, you can setup an email box/address for Trac on your existing email system, and use a POP client to retrieve and save emails to a folder. Said POP client could be run in the same batch file used to automate Email2Trac as described below (or replace the Email2Trac call If the POP client can run scripts directly).

Fetchmail is an email client that can be setup in this way with Exchange. See this post for details (Thanks, Nicole). This article details installing Cygwin/Fetchmail on Windows.

Setting Up Email2Trac

The Email2Trac script isn't very Windows-friendly out of the box. I didn't so much install it, as I ripped the main Python files out of the source tarball. Python code doesn't need to be compiled, so this works fine.

  1. Extract email2trac.tar.gz (I use 7zip)
  2. Navigate into the resulting email2trac-0.x folder and rename:
    email2trac.py.in   to email2trac.py   and
    delete_spam.py.in to delete_spam.py
  3. Copy email2trac.py, delete_spam.py, and email2trac.conf to the location where you want to run the script (I put mine in C:\python24\scripts)

For Email2Trac 0.10, I had to alter the script slightly to run. Edit email2trac.py with your favorite editor (I use IDLE, which comes with the Windows Python package).

Comment out the syslog import (line 95 in 0.10); change:

import syslog

to

#import syslog

Note: It looks like this is fixed in Email2Trac 0.13

Next change the name of the default config file (line 984 in 0.10); change:

configfile = '@email2trac_conf@'

to

configfile = 'email2trac.conf'

Depending on how you call the Email2Trac script, you might need to specify a fully-qualified path to your config file. (Thanks Nicole) Use something like this instead:

configfile = 'C:\python24\scripts\email2trac.conf'

Note: You can also specify the config file when calling the script using this switch: -f [config file] or --file=[config file] This may be an easier/cleaner solution than changing the default in the script. I haven't tested this option.

Next, you'll want to edit email2trac.conf. Be sure to configure your environment with Windows-style paths, and specify the temp directory, which is used to extract attachments. Also, make sure the temp directory exists. The top of mine looks like this:

[DEFAULT]
project: C:\trac\project1
tmpdir: C:\temp
...

Note: We only use email2trac with one environment on our server (despite the fact we run several). There is no easy way (that I know of) to use multiple environments with the current version of the Email2Trac script in this setup.

If you are running Trac 0.11, you will also need to add the following under your [DEFAULT] section: (Thanks Nicole)

trac_version: 0.11

You will likely want to make more changes to the config file to suit your needs. See the Email2Trac documentation for details.

At this point, you should be able to push a single email into a ticket from the command-line. Type:

cd C:\python24\Scripts
python email2trac.py < C:\path\to\email.eml

Note: Change paths above based where you saved email2trac and where your emails are dropped. Also depends on Python being in your PATH; add C:\python24 (or your python root) to your PATH, or replace python with C:\python24\python.exe in the above.

Automating Email2Trac

We use Windows' Scheduled Tasks to automate email2trac. This works in two parts: a batch file to run the script on a set of emails and a scheduled task to run said batch file at regular intervals.

Create a new batch file in the same folder as your email2trac.py, something like C:\python24\Scripts\trac-email.bat. Edit it with notepad and insert the following:

@echo off

for %%f in (C:\Inetpub\mailroot\Drop\*.eml) do python email2trac.py < %%f
del C:\Inetpub\mailroot\Drop\*.eml

Note: Change the path C:\Inetpub\mailroot\Drop\ to the location where your emails are dropped, and change *.eml, if they have different extensions. Same PATH caveats as above.

Now create the Scheduled Task:

  1. Start the Scheduled Task Wizard(Start > Control Panel > Scheduled Tasks > Add Scheduled Task). Click "Next >".
  2. Use "Browse..." to select the batch file you just created as the program you wish to run.
  3. Give the task a name, select "Daily". Click "Next >".
  4. Set "Start time:" to 12:00 AM, "Perform this task:" to "Every Day", "Start date:" to current date. Click "Next >".
  5. Enter a user. I had problems running the task as the unprivileged user that runs tracd, so I set myself as the user (definitely not as secure, but our Trac is on our local intranet). Click "Next >".
  6. Check "Open advanced properties..." and click "Finish". The edit task dialog will pop up.
  7. Click "Schedule" tab, then "Advanced...".
  8. Check "Repeat task", set "Every:" to 10 minutes (or your desired interval). Set "Duration:" to 24 hours.
  9. Click "OK", "OK" to close the dialogs.

You can now test your task by right-clicking it and selecting "Run". You should see a command-line window briefly appear. Check that the emails are gone and new tickets have been created in your Trac.


Disclaimer: I am not an expert on email, nor do I have much Python experience. This documentation was written up months after I setup Email2Trac, but should be fairly complete.

Wednesday, March 5, 2008

"Rounders+ Green"

I just finished making the majority of template changes to this blog.  After working with the Blogger template system, I'm satisfied that it will continue to meet my needs, because it is so flexible.

I call my new template "Rounders+ Green".  It is a mutation (or intelligent design change, if you prefer) of the Rounders 2 theme here on Blogger.

  • I expanded the width by 200px from around 800px to around 1000px.  (I would call this "optimized for 1024x768")
  • This broke the rounded corners, so I came up with a (arguably better) way to do rounded corners, using 6 stacked <div>s.  The bottom div provides the properly sized rectangle of the chosen theme color.  The next four <div>s each add a corner "mask" image, rounding off the bottom <div>'s shape.  The final <div> sets up padding for the internal layout of that particular section.  All 6 <div>s occupy the same space; after the first, each fills the entirety of the parent.  All of the layout is done in CSS, using background-color and background:url.  This approach could probably be reduced to 4 <div>s by placing two of the corners with the bottom and top <div>s.
  • I added icons for RSS and tags, using CSS of course.
  • I added formatting to the blog footer, to better match the header.
  • I aligned all of the left-side text (at least on the front page).  (This was mostly done already)
  • I fixed the post-header-line-1 section, which lacked any formatting.  I decided to float it right, and moved it before the post title, so that it would occupy space within the title block.
  • I added <pre> tag formatting for any code blocks that I insert.  This may need to be expanded, depending on what I post.  (A command-line style might be nice)

I'd like to publish the template, but first I have to figure out if I can "attach" items (like the xml file) to a post.  If not, I'll likely use my Google Pages site, or Sky Drive.


There are a few things left to be done with the template.  This list is mostly just a reminder for me.

  • Look at cleaning up the comments section on the individual post pages.  Create icons for the comments section header, and the add comment link.  (Word-bubble and plus, 12x12 on rounded orange)
  • Add tags icon to listing on left.  Maybe re-think use of orange for this icon.
  • Add icon for archive section, if seems missing after adding tags icon.
  • Move all images to my google pages site, to make things more uniform.
  • Use new corner technique for all rounded sections.
  • I'd still really like to attempt a stretch version of the template for bigger/smaller screens.
  • Try to add syntaxhighlighter to the mix.  (See this post for details)

Saturday, March 1, 2008

Sticking With Blogger (Update)

Moving Away From Blogger


Less than 24 hours in, and I'm already not liking Blogger. I'm sure it's useful for most people, but the lack of code support made my first "real" post a pain. Also, I couldn't find any themes I really liked. After posting that code, I realized how skinny that main column is. I figure I should find the right tool NOW before I have too many posts that may be hard to migrate.

I'd like to try WordPress, since I have some experience with the software from a web languages course I took in college. I notice they offer free hosting as well.


Update: I tried Wordpress. Not useful. I'm a bit of a control freak; I like to be able to control the layout of my blog. With Wordpress, you have to pay to customize.

Looking again at the customization options for Blogger, I think I'll give it another go.  And I mostly like the Rounders template I picked.  The fact that it is optimized for 800x600 was it's major drawback, but now that I see that I can easily customize it, not a problem.  Same goes with code formatting; I used a Blogger code-format tool to generate ugly chunks of HTML/CSS around each piece of code.  I should be able to roll that formatting into my template CSS, and get by with a simple <pre><code></code></pre>

I've already done some tweaking to the template; with just a little modification, it's now "optimized" for 1024x768.  I'd really like to change it to "stretch" or fill up the horizontal space available.  I think with a little work, that should be possible.  I'm hoping to recycle the original Rounders corner images, so I don't have to host my own... we'll see how that goes.

Friday, February 29, 2008

Counting Records in Ruby on Rails

I was looking through our RoR code and I came across

  def estimate_remaining
    documents.inject(0.0) {sum, d sum + d.estimate_remaining}
  end
It turns out that this can be changed to
  def estimate_remaining
    documents.to_a.sum(&:estimate_remaining)
  end
thanks to some extensions to Enumerable in Active Support. I find this much easier to read; the former requires you to know some serious Ruby, though the latter takes advantage of a symbol trick that I don't really understand (&:estimate_remaining). (Thanks to this blog for the .to_a trick)

I was hoping to do something similar to another method,

  def documents_completed
    i = 0; documents.each {d i += 1 if d.completed}; i #.nitems
  end
which counts the number of documents under the parent project that are completed. A quick look in the Rails API documentation shows that sum() is the only aggregate method they added to Enumerable. So I decided to add count() myself. I shamelessly borrowed the sum() code and modified it to look like so:
module Enumerable

  # Calculates a count by evaluating elements. Examples:
  #
  #  payments.count { p p.price > 50 }
  #  payments.count(&:overdue)
  #
  # This is instead of payments.inject { sum, p p.overdue ? sum + 1 : sum  }
  #
  # The default identity (sum of an empty list) is zero.
  # However, you can override this default:
  #
  # [].count(Payment.new(0)) { p p.price > 50 } # => Payment.new(0)
  #
  def count(identity = 0, &block)
    return identity unless size > 0

    if block_given?
      map(&block).count
    else
      inject(0) { sum, element element ? sum + 1 : sum }
      #i = 0; map { element i += 1 if element}; i #This may be faster
    end
  end


end
I saved this in [PathToRailsApp]/config/initializers/count.rb (Rails 2 environment). Now my code to count completed documents looks like this:
  def documents_completed
    documents.to_a.count(&:completed)
  end

Much cleaner! Obviously, this method is only useful if you need to count records that satisfy some sort of criteria (otherwise .length would suffice). Also, this might be more efficient done via SQL, but in my case Document.completed is a Ruby method (or "virtual column"), not an SQL column.

Things to Come

A quick note on the kinds of things I'm into, and what I hope to put on this blog. At work, I use:

Most of the posts will probably relate to these in some way or another. At home, I use more software for different activities. I'm hoping that I'll get some posts on topics like home/computer audio, and making music on the cheap, watching HDTV on your computer, etc. I'd also like to throw in reviews/promos for stuff that I use and think is great. Obviously I'm not planning on spending my life comparing products, so these kinds of posts are likely to be biased. I'm the type of person who will put up with annoyances if utility is there.

Thursday, February 28, 2008

New Blog

This is the start of my technology/informational blog. This is something that I've wanted to do for a while, but just got around to starting. I see that just getting it set up has taken me way too long. I'm hoping to set aside a bit of time each week to do some blogging. We'll see how that goes.