Common Media, Inc.



Nov
14
Bringing Mongrel back from a server crash

When a server crashes, Mongrel (or a Mongrel cluster) obviously doesn’t get a chance to shut down cleanly. This means it leaves behind the files it uses to store its process IDs. When the server restarts, the Mongrel startup script attempts to start the daemon(s), but on finding these PID files are already present, assumes (incorrectly) that Mongrel is already running, and cancels startup, saying, “PID file log/mongrel.pid already exists. Mongrel could be running already. Check your log/mongrel.log for errors.”

This is technically correct behavior–after all, what if Mongrel really is already running?–but it makes it nearly impossible to bring Mongrel back automatically after a server crash; one would have to manually delete the PID file(s) and then start the daemon(s).

If you don’t have systems administrators tending your websites 24/7, you need a better solution. We considered hacking the init script (found at /etc/rc.d/init.d/mongrel_cluster on our Fedora Core server) but found that the necessary logic made the script too complicated. Instead, we created a new startup script, filed at /etc/rc.d/init.d/mongrel_cleanup to solve the problem.

The mongrel_cleanup script is set to run at the same run-levels as mongrel_cluster. On shutdown or restart, it does nothing, but on start, it checks for the presence of the PID files and deletes them if they’re found. It therefore has to run before mongrel_cluster, which is why the priority number is 84 for startup and 16 for shutdown: mongrel_cluster is 85 and 15.

To use this script, save it in /etc/rc.d/init.d/mongrel_cleanup (or whatever the appropriate script directory is) and then put it in the startup queue with these commands:

# chkconfig --add mongrel_cleanup
# chkconfig --level 345 mongrel_cleanup on

Also, edit this script. I’ve hardwired the paths and names of our Mongrel cluster PID files; you will want to change your paths, or let me know if you come up with a more elegant method!

#!/bin/bash
#
# Parker Morse for Common Media, Inc., 9 November, 2007
#
# mongrel_cleanup      Startup script to recover from crashes.
#
# chkconfig: - 84 16
# description: A hack to clear PID files left behind by Mongrel clusters
#              after an unscheduled server crash. Checks for the presence
#              of these files and deletes them if found.
#              

RETVAL=0
PIDFILE_DIR=/path/to/app/current/log

# Gracefully exit if the controller is missing.
#which mongrel_cluster_ctl >/dev/null || exit 0

# Go no further if config directory is missing.
#[ -d "$CONF_DIR" ] || exit 0

case "$1" in
    start)
      if test -s $PIDFILE_DIR/mongrel.8000.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8000.pid;
      fi
      if test -s $PIDFILE_DIR/mongrel.8001.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8001.pid;
      fi
      if test -s $PIDFILE_DIR/mongrel.8002.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8002.pid;
      fi
      if test -s $PIDFILE_DIR/mongrel.8003.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8003.pid;
      fi
      RETVAL=$?
  ;;
    stop)
      exit 0
  ;;
    restart)
      exit 0
  ;;
    *)
      echo "Usage: mongrel_crash_cleanup {start|stop|restart}"
      exit 1
  ;;
esac      

exit $RETVAL

filed under: Ruby on Rails, System Administration | comments (2) | read more...

Aug
10
undefined method `conditions_by_like’ for Classname::Class

If you get that perplexing error, just run:

rake db:migrate

The problem is that the database is out of synch with the models, and your schema.rb needs to be updated.

Followup 8/14/2007: having these methods auto-generated is inconsistent at best, the code is available here if you’d rather eliminate the cat and mouse game and move on to larger problems (written by Alan Marcero).

filed under: Ruby on Rails | comments (0) | read more...

Jun
18
Restarting Rails

When I described our Capistrano implementation issues the other day, I was still missing one piece of the puzzle. Our deployment script would run cleanly, but there would be some errors at the end when Capistrano tried to “reap” the running server process and restart it. Because Dreamhost runs the server itself, the process ID of the server process isn’t found in the standard location, and Capistrano was unable to kill it and restart it.

Fortunately, nearly every problem on the ‘net has been found and solved before. I went back to the “Nuby on Rails” script which gave us the “Let Capistrano create current” tip, and found some code in their deploy.rb which handles the problem. With that covered, we’re really back to one-command deployments.

As an aside, something I’ve noticed as I climb the Rails learning curve is a startling number of weblogs professing to be “newbies learning Rails.” Many of them are giving good advice (anybody who can learn from their experience and pass it on is giving usable advice) but few of them are at the “newbie” level any longer, and I could argue that some of them never were. All of us are or were “new” to Rails and Ruby, but like us, I expect most of the authors of these “newbie” sites came to Rails with previous experience in LAMP, .Net, or some other framework. A true “newbie” blog, written by someone with no prior web programming experience (or perhaps no programming experience) would probably be painful to read, though.

I mostly mention this because I’m a little worried that experienced web developers feel a need to be self-deprecating about their skills with Rails. It’s fine if the culture of the framework is for everyone to adopt this “only-an-egg” approach, leaving their minds open for continual learning, but if everyone is disclaiming advanced knowledge in order to avoid responsibility for their suggestions, that’s a little creepy.

filed under: Ruby on Rails | comments (1) | read more...

Jun
16
Deploying to Dreamhost with Capistrano

I fantasize about the day when we’ll be running on a co-located server where I’ll have full control of configuration, software installation, etc., but for the time being, we have shared hosting. (Actually, I fantasize about the day when we’ll be running a rack of servers behind a load-balancer. I’m not sure if the co-located box of our own is “starting small” or a case of even my fantasies having fantasies.)

Also, new as it is, the Rails community has already embraced a number of conventional-wisdom processes handed on as best practices; one of them is using Capistrano to automate deployment from the trunk of our Subversion repository to the production server. Or, as one of the tutorials I read started,

Yeah, we know. If you’re just getting started with Rails, and you’ve been reading all these great articles by all these experienced Rails developers, it’s pretty much inevitable. One day, you’re going to see this:

You ARE using Capistrano, aren’t you?

And you will say, well, no I’m not. I must be a complete idiot.

But it’s not really that simple. Maybe you’re not using Capistrano because getting it set up initially was a real pain.

When you first “Capistranize” a Rails app, you get a hefty deploy.rb file which is almost completely non-useful to you and your application. The basic tutorial suggests a basic file which is useful but still has problems on your actual deployment server. You go through the generated file, fixing configuration lines to match your installation (you think) and run cap deploy, and… long error messages.

Here are three distinct steps we took to solve some of the error messages and get Capistrano working.

Now that Capistrano is working, it’s great to be able to deploy a revision with a single command. But getting there is less than simple, unfortunately.

Update: There’s a fourth tip I missed.

Update, 22 July 2007: Capistrano 2.0 is out now, so it’s worth noting that all the notes above were based on version 1.4.1. As of today, we haven’t upgraded yet (nor are we hosted on Dreamhost any longer) so use this information at your own risk.

filed under: Ruby on Rails | comments (4) | read more...

Newer Entries »
© 2008 Common Media, Inc. | Theme by DemusDesign and Theme Lab | Powered by WordPress