Monday, February 23, 2009

Windows Media Center FTW

I recently built a new computer, and repurposed the older machine as a guest computer for LAN gaming. In addition, I am also using it as a TV/DVR thanks to Windows Media Center and my ATI HDTV Wonder card. This is a write-up of the configuration and some specific problems and solutions I encountered putting it together. I assume the reader has installed Windows Vista Home Premium or Ultimate, or as in my case, Windows 7 Beta, and can use the Media Center interface to configure their guide/tuner/various settings.

But first, the bad news:

ATI HDTV Wonder and Windows x64

At first I tried to install my TV tuner in my shiny new rig, on which I had already installed 64-bit Windows to make full use of its 4GB of memory. From what I tried, this configuration does not work. Vista and Windows 7 recognize the card and install 64-bit drivers automatically from Windows Update. Media Center recognizes the tuner and goes through the configuration wizard just fine. Even the channel strength indicators appear to work correctly – I got a mix of full and weak signals which were accurate based on past reception. However, once you actually try to watch a channel, you get nothing but a black screen. Eventually Media Center claims there’s no signal. Some web searching only revealed a single forum comment indicating that the problem is not exactly 64-bit Windows, but actually a compatibility problem with having 4GB or more of RAM. I wasn’t about to ditch my RAM just because of the tuner, so I moved it back to the older computer.

Installing the HDTV Wonder in Windows Vista / Windows 7

Installing the card is really easy. All you have to do is put in a PCI slot and boot. Windows should recognize the card and, if you’re connected to the internet, download the drivers from Windows Update automatically. If it doesn’t (which happened for me in 7 Beta), you can force Windows to look for drivers from the Device Manager. Just type “Device Manager” in Start Menu search. Look for a category called “Unknown Devices”. Look for any items under that category with names like “multimedia controller” or “audio/video controller”. Right-click those items and select “Update Driver Software…”. Select the automatic search option; Windows should install the driver from Windows Update. Repeat for all three items.  After installing the third, Windows should pick up two more. Either way, you need to have the following five items listed under “Sound, video and game controllers” for the tuner to work correctly:

  • ATI DTV Wonder Analog Audio Capture Device 
  • ATI DTV Wonder Analog AV Capture Device
  • ATI DTV Wonder Analog AV Crossbar Device
  • ATI DTV Wonder Digital and Analog Tuner Device
  • ATI DTV Wonder Digital BDA Capture Device

If you only want/need to watch OTA digital ATSC channels, you’re pretty much done. You should be able to fire up Media Center, step through the TV setup wizard, and start watching and recording live TV. If you want to also use the analog NTSC tuner in Vista, you will need to do additional work, see http://www.hdtvtunerinfo.com/vistahdtvinstall.html for details.

Installing the Remote Wonder in Windows Vista / Windows 7

Windows will automatically detect the Remote Wonder, but unfortunately the provided driver doesn’t seem to offer any functionality. There are two options: coerce the Windows XP driver into working on Vista, or use the Remote Wonder Plus driver for Vista. I prefer the latter when using Media Center.

  1. Using the Windows XP driver – see http://www.hdtvtunerinfo.com/vistaremotewonderplus.html. Substitute the Windows XP Professional/Home – Remote Wonder – Remote Wonder driver in the instructions, if that is the version of the remote that you have.
  2. Using the Vista driver – download the driver from ATI. Install. Plug in the remote receiver. Because the driver is designed for the Plus version of the remote, some of the buttons on the vanilla version do not work as expected: mouse controls do not work, volume +/- are reversed, letter buttons have different functions, as do the top five function buttons. The good news: channel +/-, mute, numbers, arrows, ok, play, pause, stop, record, ff, rev buttons work as expected. Other buttons I have deciphered:
    • TV - Media Center / Live TV
    • DVD - Media Center / Movies
    • Web – Media Center / Pictures
    • [Book] – Media Center / Music
    • [Check box] – Media Center / Recorded TV
    • [Timer] – Close Menu
    • [Full Screen] – Back
    • A – Mute
    • C – Ok or Enter
    • D – Menu
    • E – Skip Backwards
    • F – Skip Forwards

    Note: I happen to have the original Remote Wonder. If you have the Plus model, obviously the second option would work the best.

    Power Options

    One of the really nice features that Windows 7 offers is the ability to wake your computer back up from sleep or hibernate using timers. This works nicely with Media Center once you have your tuner set up and schedule television recordings.  You leave your computer on and let it go to sleep; a few minutes before your show comes on, the wake timer turns the computer back on and Media Center records the show. When the recording is done, the computer goes back to sleep. Fantastic. My computer has problems waking from “normal” sleep mode, so I chose to configure it to go directly to hibernate.

    Open the Power Options control panel by typing “Power Options” in Start Menu search.  Click “Change Plan Settings” on the “Balanced” plan (or which ever is active).  Click “Change advanced power settings”.  Click the plus next to “Sleep” and each option under it.  If you want to hibernate directly set “Sleep after” to “Never” and “Allow hybrid sleep” to “Off”.  I set “Hibernate after” to “20 Minutes”.  Be sure “Allow wake timers” is enabled.

    Next expand the “Power buttons and lid” category. In order to make sure that it’s hard to shut the computer off, I set both button actions to “Hibernate”. Now click “OK” on the Power Options dialog and close the control panel. You may also want to change the default start menu power button. Right-click the Start button (aka Windows “orb”) and select “Properties”. Change the “Power button action” to “Hibernate”.

    If your Media Center PC is on a network with other Vista or Windows 7 computers, you may notice that it won’t hibernate while other computers are on, or won’t stay in hibernation mode. It may also turn on automatically when other computers boot up. This is due to a feature called “Wake on LAN” that allows special network packets to wake the computer. Fortunately, if only want to turn the computer on manually and for scheduled recordings, you can shut this feature off.

    Open the Network and Sharing Center (search or click the network icon in the tray). Click “Change adapter settings”. Right-click the active network adapter (most likely “Local Area Connection”) and choose “Properties”. Click the “Configure…” button. Click the “Advanced” tab. Scroll the Property list until you see “Wake on …”. Disable any options that start with “Wake on”. Click “OK”. Close the control panel.

    Other Tips

    Here are some other useful tidbits I found while configuring my Windows Media Center setup:

    Use a separate hard drive for TV recording. This will reduce stutter when you are watching live tv, and help when background recording. On systems with low RAM, this will help prevent paging from interfering with recording.

    Windows 7 uses a new .wtv format for recording TV, which can’t be watched in Windows Vista with either Media Player or Media Center (actually not quite true, this change came in a non-distributed update codenamed “Fiji” which shipped on newer Vista media center PCs).

    You can share your Recorded TV folder over a network, and add that folder into the Recorded TV library on another Windows 7 machine. This is a great way to quickly find and watch recorded shows anywhere on your network.

    Thursday, May 29, 2008

    Deploying Rails on Windows

    Why would you want to deploy Rails on Windows? With tools like Capistrano, surely deploying on Linux is the obvious choice. Well there are (at least) two valid reasons to deploy Rails on Windows:

    1. You don't have a choice. Many companies are Windows shops and maintaining a Linux/OS X/other server is simply not an option. This situation is becoming less common as Linux gains popularity in the server room. However, depending on the scale of your Rails app, you may have to share a server with other services, and this reason still holds.
    2. You use Microsoft's SQL Server as your Rails database. This is probably not common for new Rails apps in completely new systems (where you have your choice of SQL servers), although SQL Server is still a good product and sensible choice. Rails connects to SQL Server through ADO (at least, this is the method suggested by AWDwR), which is only available on Windows.

    I know there are many guides on running Rails, but I didn't find one that covered this particular configuration in detail: LightTPD <- SCGI -> Rails <- ADO -> SQL Server on Windows, running as services. This guide covers vanilla Windows XP/Windows Server 2003 to working web server. It assumes you have a Rails app working in a development environment (such as RadRails), connecting to an existing SQL Server, and committed to an existing SVN server.


    Gather and Install Necessary Software

    1. Ruby (v 1.8.6) - Download the Ruby One-click Installer for Windows. Install to C:\Ruby\ and include Gems.
    2. LightTPD (v 1.4.18-1) - Download the pre-compiled LightTPD web server (.exe), part of the WLMP project. Install to C:\LightTPD\.
    3. Rails (v 2.0.2) - Install Rails via RubyGems. Open the command line and type:
      gem update --system
      gem install rails
    4. Zed Shaw's SCGI Rails Runner (v 0.4.3) - Download the gem from Shaw's blog (click the link labeled "gem"). Install .gem file using RubyGems; again, open the command line and type:
      gem install C:\path\to\scgi_rails-0.x.x.gem
      Replace the path in the above command with the location/file you downloaded the .gem to.
    5. Subversion (v 1.4.5) - Download the SVN Windows Installer. Install to the default location (C:\Program Files\Subversion); this will install the SVN command-line tools.
    6. Windows Server 2003 Resource Kit Tools - Download rktools.exe. Install to the default location (C:\Program Files\Windows Resource Kits\Tools).
    7. Ruby/DBI ADO (v 0.1.1) - Download the ruby-dbi package from RubyForge. Extract archive somewhere using your favorite tool (such as 7-zip). Install the package from the command line:
      cd C:\path\to\ruby-dbi\
      ruby setup.rb config --with=dbd_ado
      ruby setup.rb setup
      ruby setup.rb install
      Replace the path in the above command with the location where you extracted ruby-dbi.
    8. Rails SQL Server adapter gem. This step is optional if you freeze this gem into your application as described below.

    I had to fix the SCGI Rails gem to work with Rails 2.0; you may not for later versions of the SCGI Rails Runner. Open the file C:\ruby\lib\ruby\gems\1.8\gems\scgi_rails-0.4.3\bin\scgi_service using your favorite Ruby editor (or just WordPad), and comment out line 36,

        ActiveRecord::Base.threaded_connections = false

    so that it looks like this:

        #ActiveRecord::Base.threaded_connections = false

    Apparently this method is no longer available in Rails 2.0.  I have not run into any issues removing this line, but if anyone has any suggestions for fixing it for Rails 2.0, let me know.

    Note: Since we're using the command line a lot in this guide, it makes sense to leave the window open. At this point however, you should close all command line windows, because some of the installers add locations to your PATH variable, which only affects new command line windows.

    Prepare Rails Application

    This guide suggests a configuration to host multiple Rails applications from a single Lighty server. We create a main folder to contain all Rails apps; inside, we put a log folder for the Lighty logs, and a folder for each Rails application. In my particular setup, I use this as a way to run the same intranet Rails app in production and development modes for testing purposes before rolling out changes.

    Open a command line and type:

    cd C:\
    mkdir webapps
    cd webapps
    mkdir log
    svn export svn://your-svn-server/railsapp1/trunk railsapp1
    cd railsapp1
    scgi_ctrl config -S

    Replace "your-svn-server" with the name of your svn server, and "railsapp1" with the name/svn path of your particular Rails application. Note: the last command will ask you for a password. I have no clue what this password is for, so I don't think it's too important. If you know what the password is for let me know.

    The scgi_ctrl command creates a new file in your rails app at C:\webapps\railsapp1\config\scgi.yaml. For additional apps, you will need to alter this file. For example, to run the app in development mode, you would change the file to look like this:

    --- 
    :disable_signals: true
    :env: development
    :control_url: druby://127.0.0.1:8998
    :config: config/scgi.yaml
    :host: 127.0.0.1
    :port: 9998
    :password: LeaveThePasswordAlone
    :logfile: log/scgi.log

    The useful options are highlighted above. You must specify a different port for each Rails app to run on (LightTPD redirects requests to these ports). You may also change the env setting to a different environment, based on your needs.

    Configure Lighty

    Copy the folder C:\LightTPD\conf to C:\LightTPD\conf-backup so that you have the original configuration files for future reference. Open C:\LightTPD\conf\lighttpd-srv.conf in a text editor (again, Wordpad works). Edit the file to match the following:

    # LightTPD Configuration file (RUN AS A SERVICE)
    #
    # Use it as a base for LightTPD 1.0.0 and above.
    # This version is built for WLMP Project - http://wlmp.dtech.hu/
    #
    # $Id: lighttpd-srv.conf,v 1.0 2006/11/03 23:35:28 weigon Exp $
    
    ## where to send error-messages to
    server.errorlog             = "C:/webapps/log/lighttpd-srv.error.log"
    
    #### accesslog module
    accesslog.filename          = "C:/webapps/log/lighttpd-srv.access.log"
    
    ## to help the rc.scripts
    #server.pid-file            = "C:/LightTPD/logs/lighttpd-srv.pid"
    
    #### include other configfiles
    include "C:/LightTPD/conf/lighttpd-tag.conf"
    include "C:/LightTPD/conf/lighttpd-inc.conf"

    The important changes are highlighted above. Providing absolute paths becomes necessary when running Lighty as a service.

    Next, open C:\LightTPD\conf\lighttpd-srv.conf in your text editor. Replace it with the following, and edit to suit your specific paths:

    # LightTPD Configuration file (INCLUDE)
    #
    # Use it as a base for LightTPD 1.0.0 and above.
    #
    # This version is a stripped down version for use with SCGI, Rails, and 
    # multiple hosts on ports.
    #
    # $Id: lighttpd-inc.conf,v 1.7 2004/11/03 22:26:05 weigon Exp $
    
    ############ Options you really have to take care of ####################
    
    ## Always has to be a default root
    var.home = "C:/webapps"
    server.document-root = var.home + "/railsapp1/public" 
    
    ## modules to load
    server.modules              = (
                                    "mod_access",
                                    "mod_accesslog",
                                    "mod_alias",
                                    "mod_redirect",
                                    "mod_rewrite",
                                    "mod_scgi",
                                    "mod_ssi",
                                    "mod_status",
                                   )
    ## App1 Configuration
    
    var.app1 = var.home + "/railsapp1"
    
    $SERVER["socket"] == ":80" {
        server.port                     = 80
        server.document-root            = var.app1 + "/public"
        server.upload-dirs              = ( var.app1 + "/tmp" ) 
        server.errorlog                 = var.home + "/log/app1-errors.log" 
        accesslog.filename              = var.home + "/log/app1-access.log" 
        static-file.exclude-extensions  = ( ".cgi", ".fcgi", ".scgi" )
        server.error-handler-404        = "/dispatch.scgi" 
        scgi.server = ( "dispatch.scgi" => ((
            "host" => "127.0.0.1",
            "port" => 9999,
            "check-local" => "disable" 
        )) )
    }
    
    ## App2 Configuration
    
    var.app2 = var.home + "/railsapp2" 
    
    $SERVER["socket"] == ":3000" {
        server.port                     = 3000
        server.document-root            = var.app2 + "/public"
        server.upload-dirs              = ( var.app2 + "/tmp" ) 
        server.errorlog                 = var.home + "/log/app2-errors.log" 
        accesslog.filename              = var.home + "/log/app2-access.log" 
        static-file.exclude-extensions  = ( ".cgi", ".fcgi", ".scgi" )
        server.error-handler-404        = "/dispatch.scgi" 
        scgi.server = ( "dispatch.scgi" => ((
            "host" => "127.0.0.1",
            "port" => 9998,
            "check-local" => "disable" 
        )) )
    }
    
    ## Applies to all Apps
    
    ## files to check for if .../ is requested
    # index-file.names            = ( "index.html", "index.htm", "default.htm" )
    
    # mimetype mapping
    mimetype.assign             = (
      ".pdf"          =>      "application/pdf",
      ".sig"          =>      "application/pgp-signature",
      ".spl"          =>      "application/futuresplash",
      ".class"        =>      "application/octet-stream",
      ".ps"           =>      "application/postscript",
      ".torrent"      =>      "application/x-bittorrent",
      ".dvi"          =>      "application/x-dvi",
      ".gz"           =>      "application/x-gzip",
      ".pac"          =>      "application/x-ns-proxy-autoconfig",
      ".swf"          =>      "application/x-shockwave-flash",
      ".tar.gz"       =>      "application/x-tgz",
      ".tgz"          =>      "application/x-tgz",
      ".tar"          =>      "application/x-tar",
      ".zip"          =>      "application/zip",
      ".mp3"          =>      "audio/mpeg",
      ".m3u"          =>      "audio/x-mpegurl",
      ".wma"          =>      "audio/x-ms-wma",
      ".wax"          =>      "audio/x-ms-wax",
      ".ogg"          =>      "application/ogg",
      ".wav"          =>      "audio/x-wav",
      ".gif"          =>      "image/gif",
      ".jpg"          =>      "image/jpeg",
      ".jpeg"         =>      "image/jpeg",
      ".png"          =>      "image/png",
      ".xbm"          =>      "image/x-xbitmap",
      ".xpm"          =>      "image/x-xpixmap",
      ".xwd"          =>      "image/x-xwindowdump",
      ".css"          =>      "text/css",
      ".html"         =>      "text/html",
      ".htm"          =>      "text/html",
      ".js"           =>      "text/javascript",
      ".asc"          =>      "text/plain",
      ".c"            =>      "text/plain",
      ".cpp"          =>      "text/plain",
      ".log"          =>      "text/plain",
      ".conf"         =>      "text/plain",
      ".text"         =>      "text/plain",
      ".txt"          =>      "text/plain",
      ".dtd"          =>      "text/xml",
      ".xml"          =>      "text/xml",
      ".mpeg"         =>      "video/mpeg",
      ".mpg"          =>      "video/mpeg",
      ".mov"          =>      "video/quicktime",
      ".qt"           =>      "video/quicktime",
      ".avi"          =>      "video/x-msvideo",
      ".asf"          =>      "video/x-ms-asf",
      ".asx"          =>      "video/x-ms-asf",
      ".wmv"          =>      "video/x-ms-wmv",
      ".bz2"          =>      "application/x-bzip",
      ".tbz"          =>      "application/x-bzip-compressed-tar",
      ".tar.bz2"      =>      "application/x-bzip-compressed-tar"
     )
    
    # Use the "Content-Type" extended attribute to obtain mime type if possible
    mimetype.use-xattr          = "enable"
    
    ######### Options that are good to be but not neccesary to be changed #######
    
    ## enable debugging
    #debug.log-request-header   = "enable"
    #debug.log-response-header  = "enable"
    #debug.log-request-handling = "enable"
    #debug.log-file-not-found   = "enable"
    
    #### SCGI module
    scgi.debug                  = 0
    
    #### status module
    status.status-url           = "/server-status"
    status.config-url           = "/server-config"
    
    #### url handling modules (rewrite, redirect, access)
    #url.rewrite                = ( "^/$"             => "/server-status" )
    #url.redirect               = ( "^/wishlist/(.+)" => "http://www.123.org/$1" )
    #### rewrite to pick up page cache
    url.rewrite = ( "^([^.]+)$" => "$1.html" ) 
    

    Again, portions that you will most likely need to change are highlighted above. Be sure to provide absolute paths.

    Finally, with Lighty configured, we want to run that Rails app! Copy the following into a text editor and save it to C:\LightTPD\Start_LightTPD.bat:

    @echo off
    c:
    cd C:\lighttpd\
    echo Starting lighty...
    START /B lighttpd.exe -f conf\lighttpd-srv.conf -m lib -D
    PAUSE >NUL && EXIT

    And save the following to C:\LightTPD\Start_RailsApp1.bat:

    @echo off
    c:
    cd C:\webapps\railsapp1
    echo Starting RailsApp1 SCGI_service...
    START /B scgi_service
    PAUSE >NUL && EXIT

    As always, adjust to suit your particular Rails app. You will want to create a runner batch file for each Rails app you wish to run. (unless you run as services as described below)

    Now just run each batch file. A command window should appear for each, print the "Starting x..." line, and stay open. Pop open a web browser, and type http://localhost/ in the address bar. If your Rails app comes up, congratulations! If not, you've got some troubleshooting to do. Check the command windows for any errors; you can also look at the logs in C:\webapps\log and C:\webapps\railsapp1\log for errors and additional info.

    Setup LightTPD and Rails as Services

    Once you have Lighty and Rails running from batch files, you may want to run them as services. Services have two major advantages: they keep running when you logout, and they can run with restricted security privileges.

    We are going to use the srvany program that comes with the Windows Resource Kit to run both Lighty and Rails as services. srvany acts as a wrapper that allows regular applications to run as a Windows service. Open a command line and type the following:

    instsrv lighttpd "C:\Program Files\Windows Resource Kits\Tools\srvany.exe"
    instsrv railsapp1 "C:\Program Files\Windows Resource Kits\Tools\srvany.exe"

    Create a single service for LightTPD, and one service for every Rails app you wish to run. Just change the name of the service (highlighted above).

    This creates two new services, named "lighttpd" and "railsapp1" that both run srvany. Now we edit the registry to configure what srvany runs. You can do this manually (using the values listed below), or use a text editor to save the following to lighttpd.reg (doesn't matter where you put it):

    Windows Registry Editor Version 5.00
    
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lighttpd\Parameters]
    "Application"="C:\\LightTPD\\LightTPD.exe"
    "AppParameters"="-f \"C:\\LightTPD\\conf\\lighttpd-srv.conf\" -m \"C:\\LightTPD\\lib\" -D"
    

    Save the following to railsapp1.reg (or name of your Rails app):

    Windows Registry Editor Version 5.00
    
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\railsapp1\Parameters]
    "Application"="C:\\ruby\\bin\\ruby.exe"
    "AppParameters"="C:\\ruby\\bin\\scgi_service"
    "AppDirectory"="C:\\webapps\\railsapp1"
    

    Save a copy of the .reg file above for each Rails service, changing the registry path and AppDirectory value to point to your service name and Rails app location.

    Finally, double-click each of these .reg files to add their contents to the registry.

    By default, new services run under (or "log on as") the Local System account. This should be changed to a low-privilege account, in case the Rails environment or LightTPD is compromised. We want to run all of the services in this Guide under the Network Service account.

    Open the Services dialog at Start > Control Panel > Admistrative Tools > Services. Find the service "lighttpd" that we created, and double-click it. Click the "Log On" tab at the top of the properties dialog, and click "This account:" radio. Type "NT AUTHORITY\NetworkService" into the account field, and clear the password and confirm fields. Click "OK" in the properties dialog. Repeat these steps for each Rails service you have created.

    One last step is needed to get the services to run. Both LightTPD and the SCGI Rails Runner will crash if they don't have write permissions to their respective log directories. (As a bonus, when running as services under srvany, they crash silently, not even appearing to have stopped, and having no way to report the error) Using Explorer, navigate to C:\webapps\, right-click the log folder, and select "Properties". Click the "Security" tab, and click "Add...". Type "Network Service" into the object names field, and click "OK". Back in the Properties dialog, make sure "NETWORK SERVICE" is selected in the upper user name list, and check the box in the lower permissions list to allow writes (row "Write", column "Allow"). Click "OK" on the properties dialog. Now navigate to C:\webapps\railsapp1\, etc. and repeat for the log folder inside each Rails app.

    Now to test the services. Make sure to close any open command windows that are running LightTPD and/or your Rails apps (such as the batch files created above). Start the services using the Services dialog (Start > Control Panel > Admistrative Tools > Services) or from the command line:

    NET START lighttpd
    NET START railsapp1

    At this point, I'd recommend starting Lighty and just one Rails app, then checking if it is running by pointing a browser at http://localhost/. If it works, start up and test the rest of your Rails apps and call it a day! If you get a blank page, you will have to troubleshoot your configuration. The only good way to do that is to run cmd using srvany. One gotcha: Microsoft's solution uses Local System with the "interact with desktop" feature; however, this only works when you are logged directly into the machine, not over a Remote Desktop connection. Once you have cmd running as a service, try running your batch files from before and troubleshoot any errors reported.

    Automating deployment

    A couple tips to automate deployment:

    • Commit config\scgi.yaml to SVN, so that you don't have to run scgi_ctrl config when exporting future versions of your webapp. If you need multiple versions of scgi.yaml (say, for different environments), create a deploy folder in your Rails app, then create deploy\scgi-prod.yaml and deploy\scgi-dev.yaml; copy one of these to config\scgi.yaml at deployment.
    • Create a batch file to automate the process, and create a shortcut to said batch on your desktop. This gets you fairly close to one-click deployment (SVN commit, RDC to server, run batch). A sample deploy script might look like this (save in C:\webapps\):
      @echo off
      echo Stopping Service
      NET STOP railsapp1
      sleep 1
      echo Deleting old copy
      rd /s /q railsapp1
      echo Exporting new copy from SVN
      svn export svn://your-svn-server/railsapp1/trunk railsapp1
      echo Configuring deployment
      copy railsapp1\deploy\scgi-prod.yaml railsapp1\config\scgi.yaml
      echo Starting Service
      NET START railsapp1

    Friday, April 11, 2008

    Advanced sum() usage in Rails

    Active Support adds a nice sum() method to Enumerable, which is mixed into Array. Rails also has a nice extension to Ruby symbols that lets you call a method that normally takes a block like this:

    total_price = items.sum(&:price)

    You may be thinking that this only works for numbers. However, because Ruby is dynamically typed, we can actually take the sum of any type that implements the + method. (In Ruby, even mathematical operators are methods, just as numbers are objects) That means we can use it to concatenate arrays (array.+ is concatenation). This can come in handy when working with multiple Active Record has_many relationships:

    firm_invoices = @firm.clients.to_a.sum(&:invoices)

    To be fair, this probably isn't the most efficient way to do this. The example above is equivalent to the following usage of has_many :through from the Rails API:

    @firm.clients.collect { |c| c.invoices }.flatten
    @firm.invoices # defined by has_many :through the Client join model

    Sometimes life isn't as easy as examples. In my case, we have customers that can belong to multiple offices (they are mobile and do work from each). So a customer has_and_belongs_to_many (habtm) offices (in SQL terms, this is a many-to-many relationship via a join table). In our Rails project, the join table does not have a model object, as it contains only the data to define the relationship. An office, in turn, has_many projects (our main unit of work; a single request from a customer). My goal was to report all of the projects for the offices of the logged-in customer. With sum(), it's easy:

    projects = @customer.offices.to_a.sum(&:active_projects)

    One thing I don't like about this is that it is hard to read. Summing projects doesn't make verbal sense. On the other hand

    projects = @customer.office_active_projects

    smells funny. Why should Customer have specific knowledge of how to get Projects for Offices?

    In the end, summing arrays is a useful method to have in your tool belt. It can be used to quickly drill through relationships (without altering model objects) to make sure you get the results you are looking for, before making your code more efficient/elegant/permanent.

    Thursday, March 20, 2008

    DD-WRT Firmware on Buffalo WHR-G54S

    I've had the Buffalo WHR-G54S router for about a year, but tonight I decided to install the popular third-party firmware DD-WRT.  I knew the router was compatible with DD-WRT when I purchased it (that was part of the reason I selected it).  I highly recommend the WHR-G54S; it has worked well for as long as I've had it.

    Actually, it hasn't performed flawlessly.  In fact, I've been having problems since my new roommate moved in.  He likes to P2P when he's not here (which is usually when I'm here and want to use the bandwidth).  Tonight when I got home, my computer was having major connectivity issues.  Even after stopping his downloads, the problems continued.  I had to power-cycle my cable modem and router just to get a simple page to load.  Before rebooting it, even the web-configuration tools on the router couldn't load (so the issues were in the router?!?)  That's why after a year, I decided to give DD-WRT a shot.  Hopefully it will improve my connection (or contain tools to shape the P2P traffic/give me priority)!

    I just wanted to give some pointers that helped me through the process:

    • As this article points out, dd-wrt.[version]_mini_generic.bin is the firmware file you want to use when first flashing from Buffalo's factory firmware.  That means you want to download dd-wrt.[version]_mini.zip from the DD-WRT Downloads Area.  At the time of writing, the latest stable version was 2.3_SP2.
    • The DD-WRT Wiki Installation Page was very helpful, especially the Precautions and Buffalo > TFTP Flashing Buffalo Routers under Windows sections.  I followed the instructions from the latter and successfully flashed the router on the second try. (the first try, I jumped the gun when the red light came on; when you see the first ping as the router comes back up hit enter immediately, I only got one successful ping during the TFTP window the first time around)
    • My reset button is labeled "init" and sits on butt end of the router, next to the "bridge" switch and directly opposite the antenna jack.  You will need something thin like a paperclip to press it.
    • When unplugging and re-plugging my router, I used the router (DC power) end of the power cable, since this was easier to do with the router sitting on the desk next to my keyboard.  I unplugged everything else except a singe Ethernet cable from my computer.
    • My Ethernet port (nForce 4 built-in) is auto-sensing, but came back well before the TFTP ping.

    The installation went so smoothly, I'm wondering why I waited this long to try DD-WRT.  The answer is, "If it ain't broke, don't fix it!"  Hopefully I'll have more to say when I've had a chance to play with some of the firmware's more advanced settings.  I can say that even the "mini" version has all of the features I used in the original Buffalo firmware.  Unfortunately I forgot to write down the port-forwarding settings I had for serving HL2:Deathmatch and Synergy internet games.  Oh well, I'm sure I can find the ports again.

    The dream app?  Setting up a Cron job that pipes fortune to net send on the hour.  Imagine - every computer on the network (or maybe just your roommate's) gets a dialog box with a fortune every hour, courtesy of your local router and DD-WRT.  Priceless.

    Monday, March 17, 2008

    Email2Trac on Windows

    See the official Email2Trac documentation for an up to date version of these instructions.


    If you're not familiar with Trac: Trac is an integrated issue tracking system, wiki, and SVN repository browser. It is open source software (under the very liberal BSD licence), and is popular as a public bug tracker for many open source projects. Email2Trac is a plugin (actually, more of a script) for Trac that enables users to submit bugs via email.

    Email2Trac was clearly created by *nix users, but since it is written in Python, can be made to work in Windows without too much hassle. This post is a summary of a couple messages I posted to the Trac Users Group detailing our Email2Trac setup.


    Platform: Windows Server 2003, Python 2.4, Trac 0.10.3, email2trac 0.10 (year+ old install)

    There are 3 pieces you need to effectively turn emails into Trac tickets:

    1. A way to receive emails on your Trac server
    2. A way to process or convert emails into Trac tickets (this would be the email2trac python script)
    3. A way to automate part 2

    Here's what we use on Windows:

    Receiving Emails via SMTP

    We use Windows' built-in SMTP server to deliver mail to our Trac server. When I setup email2trac, I thought that this step was going to be the most difficult (mostly because I have no experience with email servers/services). Surprisingly, it is actually the easiest step. The trick is to install the SMTP server, but not the POP server, so that emails are received and left as files in the "drop" folder. (This is the way an email server hands off email between the two services) In a way, email2trac performs the distribution functions normally handled by POP.

    Install the SMTP service via Windows Components.

    1. Open Add/Remove Programs (Start > Control Panel > Add or Remove Programs)
    2. Click "Add/Remove Windows Components" in the left-hand bar.
    3. Use the "Details..." button to drill-in to Application Server > Internet Information Services (IIS) > SMTP Service. In Windows XP, it's just Internet Information Services (IIS) > SMTP Service.
    4. Check the box next to SMTP Service, click "OK", "OK", "Next", "Finish".

    At this point, you should ensure that the SMTP Service is running (on my workstation it didn't start after installation; this may be a policy issue).

    1. Open up the Services dialog at Start > Control Panel > Admistrative Tools >Services.
    2. Scroll down to "Simple Mail Transfer Protocol (SMTP).
    3. If startup type is not "Automatic", double-click the service. Change "Startup Type" to "Automatic". Click "OK".
    4. If status is not "started", click play button in services dialog toolbar to start the service.

    The SMTP service should be configured and ready to receive emails out of the box. You can find SMTP settings at Start > Control Panel > Admistrative Tools >Internet Information Services (IIS) Manager. The installation automatically creates the default domain "Local (Default)" and the corresponding drop folder at C:\Inetpub\mailroot\Drop. You can see that my email2trac batch script picks up .eml files from that location.

    To actually receive emails on the server, you have to send to an address of the form "anything@trac-server.domain.com". Your Exchange or other mail server should automatically forward the emails to your Trac server. If you want to use a different address, you will likely have to configure your mail server to forward the mail. The name before the @ can be anything; POP service usually uses this to distribute mail to inboxes, SMTP and email2trac ignore it. A possible enhancement to email2trac would be to use the local-part of the address to identify the destination environment for the ticket. You will probably want to give out a single address, just to keep things consistent; since it is our Help Desk's Trac, we use "helpdesk@trac-server.domain.com".

    Alternative Way to Receive Emails

    If the server that hosts Trac already handles emails (thus the SMTP domain "trac-server.domain.com" is taken), you will need a different method to receive emails. Instead of using SMTP, you can setup an email box/address for Trac on your existing email system, and use a POP client to retrieve and save emails to a folder. Said POP client could be run in the same batch file used to automate Email2Trac as described below (or replace the Email2Trac call If the POP client can run scripts directly).

    Fetchmail is an email client that can be setup in this way with Exchange. See this post for details (Thanks, Nicole). This article details installing Cygwin/Fetchmail on Windows.

    Setting Up Email2Trac

    The Email2Trac script isn't very Windows-friendly out of the box. I didn't so much install it, as I ripped the main Python files out of the source tarball. Python code doesn't need to be compiled, so this works fine.

    1. Extract email2trac.tar.gz (I use 7zip)
    2. Navigate into the resulting email2trac-0.x folder and rename:
      email2trac.py.in   to email2trac.py   and
      delete_spam.py.in to delete_spam.py
    3. Copy email2trac.py, delete_spam.py, and email2trac.conf to the location where you want to run the script (I put mine in C:\python24\scripts)

    For Email2Trac 0.10, I had to alter the script slightly to run. Edit email2trac.py with your favorite editor (I use IDLE, which comes with the Windows Python package).

    Comment out the syslog import (line 95 in 0.10); change:

    import syslog

    to

    #import syslog

    Note: It looks like this is fixed in Email2Trac 0.13

    Next change the name of the default config file (line 984 in 0.10); change:

    configfile = '@email2trac_conf@'

    to

    configfile = 'email2trac.conf'

    Depending on how you call the Email2Trac script, you might need to specify a fully-qualified path to your config file. (Thanks Nicole) Use something like this instead:

    configfile = 'C:\python24\scripts\email2trac.conf'

    Note: You can also specify the config file when calling the script using this switch: -f [config file] or --file=[config file] This may be an easier/cleaner solution than changing the default in the script. I haven't tested this option.

    Next, you'll want to edit email2trac.conf. Be sure to configure your environment with Windows-style paths, and specify the temp directory, which is used to extract attachments. Also, make sure the temp directory exists. The top of mine looks like this:

    [DEFAULT]
    project: C:\trac\project1
    tmpdir: C:\temp
    ...

    Note: We only use email2trac with one environment on our server (despite the fact we run several). There is no easy way (that I know of) to use multiple environments with the current version of the Email2Trac script in this setup.

    If you are running Trac 0.11, you will also need to add the following under your [DEFAULT] section: (Thanks Nicole)

    trac_version: 0.11

    You will likely want to make more changes to the config file to suit your needs. See the Email2Trac documentation for details.

    At this point, you should be able to push a single email into a ticket from the command-line. Type:

    cd C:\python24\Scripts
    python email2trac.py < C:\path\to\email.eml

    Note: Change paths above based where you saved email2trac and where your emails are dropped. Also depends on Python being in your PATH; add C:\python24 (or your python root) to your PATH, or replace python with C:\python24\python.exe in the above.

    Automating Email2Trac

    We use Windows' Scheduled Tasks to automate email2trac. This works in two parts: a batch file to run the script on a set of emails and a scheduled task to run said batch file at regular intervals.

    Create a new batch file in the same folder as your email2trac.py, something like C:\python24\Scripts\trac-email.bat. Edit it with notepad and insert the following:

    @echo off
    
    for %%f in (C:\Inetpub\mailroot\Drop\*.eml) do python email2trac.py < %%f
    del C:\Inetpub\mailroot\Drop\*.eml

    Note: Change the path C:\Inetpub\mailroot\Drop\ to the location where your emails are dropped, and change *.eml, if they have different extensions. Same PATH caveats as above.

    Now create the Scheduled Task:

    1. Start the Scheduled Task Wizard(Start > Control Panel > Scheduled Tasks > Add Scheduled Task). Click "Next >".
    2. Use "Browse..." to select the batch file you just created as the program you wish to run.
    3. Give the task a name, select "Daily". Click "Next >".
    4. Set "Start time:" to 12:00 AM, "Perform this task:" to "Every Day", "Start date:" to current date. Click "Next >".
    5. Enter a user. I had problems running the task as the unprivileged user that runs tracd, so I set myself as the user (definitely not as secure, but our Trac is on our local intranet). Click "Next >".
    6. Check "Open advanced properties..." and click "Finish". The edit task dialog will pop up.
    7. Click "Schedule" tab, then "Advanced...".
    8. Check "Repeat task", set "Every:" to 10 minutes (or your desired interval). Set "Duration:" to 24 hours.
    9. Click "OK", "OK" to close the dialogs.

    You can now test your task by right-clicking it and selecting "Run". You should see a command-line window briefly appear. Check that the emails are gone and new tickets have been created in your Trac.


    Disclaimer: I am not an expert on email, nor do I have much Python experience. This documentation was written up months after I setup Email2Trac, but should be fairly complete.