Common Media, Inc.

When a server crashes, Mongrel (or a Mongrel cluster) obviously doesn’t get a chance to shut down cleanly. This means it leaves behind the files it uses to store its process IDs. When the server restarts, the Mongrel startup script attempts to start the daemon(s), but on finding these PID files are already present, assumes (incorrectly) that Mongrel is already running, and cancels startup, saying, “PID file log/mongrel.pid already exists. Mongrel could be running already. Check your log/mongrel.log for errors.”

This is technically correct behavior–after all, what if Mongrel really is already running?–but it makes it nearly impossible to bring Mongrel back automatically after a server crash; one would have to manually delete the PID file(s) and then start the daemon(s).

If you don’t have systems administrators tending your websites 24/7, you need a better solution. We considered hacking the init script (found at /etc/rc.d/init.d/mongrel_cluster on our Fedora Core server) but found that the necessary logic made the script too complicated. Instead, we created a new startup script, filed at /etc/rc.d/init.d/mongrel_cleanup to solve the problem.

The mongrel_cleanup script is set to run at the same run-levels as mongrel_cluster. On shutdown or restart, it does nothing, but on start, it checks for the presence of the PID files and deletes them if they’re found. It therefore has to run before mongrel_cluster, which is why the priority number is 84 for startup and 16 for shutdown: mongrel_cluster is 85 and 15.

To use this script, save it in /etc/rc.d/init.d/mongrel_cleanup (or whatever the appropriate script directory is) and then put it in the startup queue with these commands:

# chkconfig --add mongrel_cleanup
# chkconfig --level 345 mongrel_cleanup on

Also, edit this script. I’ve hardwired the paths and names of our Mongrel cluster PID files; you will want to change your paths, or let me know if you come up with a more elegant method!

#!/bin/bash
#
# Parker Morse for Common Media, Inc., 9 November, 2007
#
# mongrel_cleanup      Startup script to recover from crashes.
#
# chkconfig: - 84 16
# description: A hack to clear PID files left behind by Mongrel clusters
#              after an unscheduled server crash. Checks for the presence
#              of these files and deletes them if found.
#              

RETVAL=0
PIDFILE_DIR=/path/to/app/current/log

# Gracefully exit if the controller is missing.
#which mongrel_cluster_ctl >/dev/null || exit 0

# Go no further if config directory is missing.
#[ -d "$CONF_DIR" ] || exit 0

case "$1" in
    start)
      if test -s $PIDFILE_DIR/mongrel.8000.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8000.pid;
      fi
      if test -s $PIDFILE_DIR/mongrel.8001.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8001.pid;
      fi
      if test -s $PIDFILE_DIR/mongrel.8002.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8002.pid;
      fi
      if test -s $PIDFILE_DIR/mongrel.8003.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8003.pid;
      fi
      RETVAL=$?
  ;;
    stop)
      exit 0
  ;;
    restart)
      exit 0
  ;;
    *)
      echo "Usage: mongrel_crash_cleanup {start|stop|restart}"
      exit 1
  ;;
esac      

exit $RETVAL

filed under: Ruby on Rails, System Administration | comments (2) | read more...

If you get that perplexing error, just run:

rake db:migrate

The problem is that the database is out of synch with the models, and your schema.rb needs to be updated.

Followup 8/14/2007: having these methods auto-generated is inconsistent at best, the code is available here if you’d rather eliminate the cat and mouse game and move on to larger problems (written by Alan Marcero).

filed under: Ruby on Rails | comments (0) | read more...

When I described our Capistrano implementation issues the other day, I was still missing one piece of the puzzle. Our deployment script would run cleanly, but there would be some errors at the end when Capistrano tried to “reap” the running server process and restart it. Because Dreamhost runs the server itself, the process ID of the server process isn’t found in the standard location, and Capistrano was unable to kill it and restart it.

Fortunately, nearly every problem on the ‘net has been found and solved before. I went back to the “Nuby on Rails” script which gave us the “Let Capistrano create current” tip, and found some code in their deploy.rb which handles the problem. With that covered, we’re really back to one-command deployments.

As an aside, something I’ve noticed as I climb the Rails learning curve is a startling number of weblogs professing to be “newbies learning Rails.” Many of them are giving good advice (anybody who can learn from their experience and pass it on is giving usable advice) but few of them are at the “newbie” level any longer, and I could argue that some of them never were. All of us are or were “new” to Rails and Ruby, but like us, I expect most of the authors of these “newbie” sites came to Rails with previous experience in LAMP, .Net, or some other framework. A true “newbie” blog, written by someone with no prior web programming experience (or perhaps no programming experience) would probably be painful to read, though.

I mostly mention this because I’m a little worried that experienced web developers feel a need to be self-deprecating about their skills with Rails. It’s fine if the culture of the framework is for everyone to adopt this “only-an-egg” approach, leaving their minds open for continual learning, but if everyone is disclaiming advanced knowledge in order to avoid responsibility for their suggestions, that’s a little creepy.

filed under: Ruby on Rails | comments (1) | read more...

I fantasize about the day when we’ll be running on a co-located server where I’ll have full control of configuration, software installation, etc., but for the time being, we have shared hosting. (Actually, I fantasize about the day when we’ll be running a rack of servers behind a load-balancer. I’m not sure if the co-located box of our own is “starting small” or a case of even my fantasies having fantasies.)

Also, new as it is, the Rails community has already embraced a number of conventional-wisdom processes handed on as best practices; one of them is using Capistrano to automate deployment from the trunk of our Subversion repository to the production server. Or, as one of the tutorials I read started,

Yeah, we know. If you’re just getting started with Rails, and you’ve been reading all these great articles by all these experienced Rails developers, it’s pretty much inevitable. One day, you’re going to see this:

You ARE using Capistrano, aren’t you?

And you will say, well, no I’m not. I must be a complete idiot.

But it’s not really that simple. Maybe you’re not using Capistrano because getting it set up initially was a real pain.

When you first “Capistranize” a Rails app, you get a hefty deploy.rb file which is almost completely non-useful to you and your application. The basic tutorial suggests a basic file which is useful but still has problems on your actual deployment server. You go through the generated file, fixing configuration lines to match your installation (you think) and run cap deploy, and… long error messages.

Here are three distinct steps we took to solve some of the error messages and get Capistrano working.

• Remove the :db role: Capistrano attempts to make SSH connections to every host listed with a role in deploy.rb. If you’re with a shared host, odds are you don’t have shell access to your database machine. You’ll be prompted for a password, then get an error which looks like this:
  

  ** [update_code] exception while rolling back: Net::SSH::AuthenticationFailed, commonkitchen
authentication failed for `commonkitchen’


  Commenting out the :db role is sufficient to stop Capistrano from trying to connect to that host, and lets you move on to the next problem. Update: not having that role keeps migrations from being run. Instead, set the :db role to be the same hostname as the other roles.

• Use a “standard” repository layout: I say “standard” because you’re free to lay out your repository however you wish. However, we initially had all our app at the top level of our repository, and Capistrano was trying to find it in the trunk/ directory. Best to use the recommended layout described in the turtle book… or at least keep your application inside a trunk directory, even if you never use tags and branches. The errors are thick, but somewhere will be one that looks like this:
  

  /usr/local/lib/ruby/gems/1.8/gems/capistrano-1.4.1/lib/capistrano/scm/subversion.rb:24:in `latest_revision': Could not determine latest revision (RuntimeError)

  Ultimately, it’s easier to rearrange your repository than teach Capistrano to drop the trunk path.

• Let Capistrano create the web root: When we set up the virtual host on our server, the root directory (~/application/current/public) was automatically created by that setup. Capistrano would then create the rollback directories, but couldn’t create a symlink to the current directory. We got errors like this:
  

  command "~/application/current/script/process/reaper" failed on ourserver.example.com

  Once the current directory and its children were deleted, Capistrano was able to create its own current with a symlink to the most recent version, and everything was fine. (We figured this out from this tip.)

Now that Capistrano is working, it’s great to be able to deploy a revision with a single command. But getting there is less than simple, unfortunately.

Update: There’s a fourth tip I missed.

Update, 22 July 2007: Capistrano 2.0 is out now, so it’s worth noting that all the notes above were based on version 1.4.1. As of today, we haven’t upgraded yet (nor are we hosted on Dreamhost any longer) so use this information at your own risk.

filed under: Ruby on Rails | comments (4) | read more...