23
The recently-announced vulnerabilities in Ruby have put many of us administering Rails applications in a production space between a rock and a hard place.
To recap, there are three major “lines” of Ruby interpreters, the 1.8.6 line, the 1.8.7 line, and the 1.9 line. All of these show the vulnerability, so nearly everyone needs to update their Ruby. (I’ll get to the exceptions later.) Rails introduces a new line of complication, because the 1.8.7 only works for Rails 2.1 and newer; if the Rails apps in question aren’t ready for Rails 2.1, or already running it (unlikely), the best route would be to continue to update in the 1.8.6 line. So there’s the rock: we need to update our Ruby.
The “hard place” is that the only patches so far released by the Ruby maintainers tend to produce segmentation faults, according to many who have tried them so far. That’s a long way of saying, “They don’t work.”
As a Rails-supporting sysadmin with, you’re left with several options, none of them comfortable. You can, in order of increasing riskiness:
- Wait out the Ruby maintainers and hope they release a working 1.8.6 patchlevel before your app is compromised.
- Do crash-priority Rails upgrades on all your apps and switch to Rails 2.1, then move your Ruby to the 1.8.7 line.
- Switch to an entirely different Ruby interpreter, like Rubinius or JRuby. (This hinges on the idea that Ruby is an interpreted language, and as long as the language is interpreted consistently there is room for any number of interpreters. Apparently JRuby and Rubinius are not affected by these vulnerabilities, only MRI, Matz’s Ruby Interpreter, which is the “standard” implementation.)
- Switch to a different production stack entirely. For example, we’re currently using Apache 2.2 as a proxy for a Mongrel cluster in front of Rails 2.0.2 apps, a not-uncommon configuration. This could be the pressure we’d need to jump to Phusion Passenger, given that their Ruby Enterprise Edition claims to have applied the security patches to MRE 1.8.6 p111, and thus doesn’t introduce the segfault-inducing breaking features that the most-recent patchlevel does.
I don’t know which way we’re going yet, but I’m not interested in waiting too long, and the upgrade to Rails 2.1 is going to happen sometime anyway, so Option #2 seems most likely for us.
Update: Hongli Lai from Phusion assures us (in the comments) that Ruby Enterprise Edition can be used as a drop-in replacement for MRE without replacing the entire stack, moving REE up to the front of the line in the “different interpreter” option. Discussion seems to suggest to me that new official patches from the Ruby maintainers will not be coming in a timely fashion, but we are beginning to see “contributed” patched distributions emerge for e.g. FreeBSD and Debian.
filed under: Ruby on Rails, System Administration | comments (2) | read more...
9
This afternoon we pushed another big revision to the La Cucina Italiana website, which is now sharing a very small fraction of the thousands of recipes that magazine has in its archives. There will be more recipes coming online in the months to come, but most of the puzzles we had to solve are finished now.
One in particular was handling the photos which go with some recipes. Artful photography is a hallmark of the La Cucina Italiana brand, and the editorial team needed to be able to upload their images directly to the site. These photos wouldn’t be stored in the site database, but because they wouldn’t be part of our Subversion repository for the site, either, they had to live outside the normal site root in order to avoid being blown away by any site updates we deployed.
Enter Rails’ asset hosts. Rails allows for assets (e.g. images, CSS files, or Javascript includes) to be served by a host other than that of the main site. Because the asset host definition happens at the host name level, the asset host can actually be on the same box as the main site, or it can be elsewhere in the world; you manage that at a different level of abstraction.
A “free” benefit of asset hosts is that by defining multiple asset hosts (a0 through an), you can fool a user’s browser into downloading your site through a dozen or more different connections, rather than just the two it limits itself two with any single server. Rails will make each asset link use a different asset host, and the browser will open two connections to each server, not caring that they all happen to be on the same box. This gets us extra YSlow points (of course, we lose them again by requiring another DNS lookup for each asset host).
Our hangup, though, was that some assets, specifically CSS and Javascripts, did need to stay inside the Subversion repository and the site’s file tree.
Here’s the solution we came up with:
- Defined an Apache virtual server answering to subdomains
a0througha4. (In other words, all four subdomains are the same real server, but the browser doesn’t know that; we could make them different physical servers someday.) - Set the site root for that server to
/var/www/assets/. - Made symbolic links from
/var/www/assets/javascriptsto/var/www/production/current/public/javascripts, and from/var/www/assets/stylesheetsto/var/www/production/current/public/stylesheets. (N.B. if you’re using Capistrano, “current” in your site directory is itself a symbolic link to a directory with your last site deploy.) - Set the upload code for recipe photos to store the uploaded images in subdirectories of
/var/www/assets/images.
This way, we get all the benefits of asset hosts for assets which are under revision control, and assets which aren’t. And the asset hosts themselves let us have assets which aren’t necessarily under revision control.
The free bonus here was that by using a symbolic link to the javascripts directory, bundle_fu doesn’t have to know or understand our asset host setup; it just stores its bundled files in that same subdirectory as always, and it just works.
filed under: Ruby on Rails, System Administration | comments (0) | read more...
8
There’s (still) a glitch between the mongrel_cluster gem and Capistrano 2 (we’re using mongrel_cluster 1.0.5 and Capistrano 2.1.0, for reference) where the application restart at the end of a cap deploy fails with an error like this:
Couldn't find any pid file in '/var/www/[application]/current/tmp/pids’ matching ‘dispatch.[0-9]*.pid’
I’m not sure what’s causing this, but the solution comes at the end of this post, under the heading “Restart Mongrel.” Due to issues with Cap 2 and sudo, though, the provided script fails for us. We’re running the updates as root (bad idea, but it gets around the cap sudo issues) so I updated the task like this:
# Restart task
set :mongrel_config, "/etc/mongrel_cluster/#{application}.yml"
namespace :deploy do
task :restart do
run "mongrel_rails cluster::restart -C #{mongrel_config}"
end
end
I also commented out the “:mongrel_conf” variable from our previous configuration, which it appears that Capistrano was ignoring anyway.
filed under: Ruby on Rails, System Administration | comments (0) | read more...
14
When a server crashes, Mongrel (or a Mongrel cluster) obviously doesn’t get a chance to shut down cleanly. This means it leaves behind the files it uses to store its process IDs. When the server restarts, the Mongrel startup script attempts to start the daemon(s), but on finding these PID files are already present, assumes (incorrectly) that Mongrel is already running, and cancels startup, saying, “PID file log/mongrel.pid already exists. Mongrel could be running already. Check your log/mongrel.log for errors.”
This is technically correct behavior–after all, what if Mongrel really is already running?–but it makes it nearly impossible to bring Mongrel back automatically after a server crash; one would have to manually delete the PID file(s) and then start the daemon(s).
If you don’t have systems administrators tending your websites 24/7, you need a better solution. We considered hacking the init script (found at /etc/rc.d/init.d/mongrel_cluster on our Fedora Core server) but found that the necessary logic made the script too complicated. Instead, we created a new startup script, filed at /etc/rc.d/init.d/mongrel_cleanup to solve the problem.
The mongrel_cleanup script is set to run at the same run-levels as mongrel_cluster. On shutdown or restart, it does nothing, but on start, it checks for the presence of the PID files and deletes them if they’re found. It therefore has to run before mongrel_cluster, which is why the priority number is 84 for startup and 16 for shutdown: mongrel_cluster is 85 and 15.
To use this script, save it in /etc/rc.d/init.d/mongrel_cleanup (or whatever the appropriate script directory is) and then put it in the startup queue with these commands:
# chkconfig --add mongrel_cleanup
# chkconfig --level 345 mongrel_cleanup on
Also, edit this script. I’ve hardwired the paths and names of our Mongrel cluster PID files; you will want to change your paths, or let me know if you come up with a more elegant method!
#!/bin/bash
#
# Parker Morse for Common Media, Inc., 9 November, 2007
#
# mongrel_cleanup Startup script to recover from crashes.
#
# chkconfig: - 84 16
# description: A hack to clear PID files left behind by Mongrel clusters
# after an unscheduled server crash. Checks for the presence
# of these files and deletes them if found.
#
RETVAL=0
PIDFILE_DIR=/path/to/app/current/log
# Gracefully exit if the controller is missing.
#which mongrel_cluster_ctl >/dev/null || exit 0
# Go no further if config directory is missing.
#[ -d "$CONF_DIR" ] || exit 0
case "$1" in
start)
if test -s $PIDFILE_DIR/mongrel.8000.pid
then
/bin/rm $PIDFILE_DIR/mongrel.8000.pid;
fi
if test -s $PIDFILE_DIR/mongrel.8001.pid
then
/bin/rm $PIDFILE_DIR/mongrel.8001.pid;
fi
if test -s $PIDFILE_DIR/mongrel.8002.pid
then
/bin/rm $PIDFILE_DIR/mongrel.8002.pid;
fi
if test -s $PIDFILE_DIR/mongrel.8003.pid
then
/bin/rm $PIDFILE_DIR/mongrel.8003.pid;
fi
RETVAL=$?
;;
stop)
exit 0
;;
restart)
exit 0
;;
*)
echo "Usage: mongrel_crash_cleanup {start|stop|restart}"
exit 1
;;
esac
exit $RETVAL
filed under: Ruby on Rails, System Administration | comments (2) | read more...