Common Media, Inc.



Feb
19
Rejecting mail for valid local users

Several months ago we mentioned that certain Linux distributions will, if they are running an SMTP server and accepting mail for local users, accumulate spam for “role” users who will never read their mail, e.g. mail, uucp, news, etc.

I finished by suggesting,

The best shortcut here is to bounce any email destined for the role users. This will vary depending on your MTA, so I won’t detail it here.

It’s true that it varies by MTA, but it turns out it’s really hard to find this information. MTAs aren’t set up to reject mail from specific addresses; they want a specific list of valid addresses and they’ll reject everything else.

It turns out that there’s a faster and cleaner method which is MTA-independent: lock the mailboxes for those users.

This could be as simple as putting an empty file in /var/spool/mail/uucp (for example) which is owned by root with 600 permissions, but that’s going to generate a bunch of error messages when your local delivery agent tries to write to a file it doesn’t have permissions for. A more elegant solution is to symlink those paths to /dev/null:

ln -s /dev/null /var/spool/mail/uucp

Now make sure the symlink has the correct ownership…

chown -h uucp:mail /var/spool/mail/uucp

Now all that spam will be silently delivered to the bit-bucket.

filed under: System Administration | comments (0) | read more...

Sep
14
Clean out role-user mailboxes

I suppose the subtitle to this post could be, “why you don’t want to be a mail administrator.” The fact is, spam has made running a host which accepts email a pretty unpleasant task all around, and we strongly suggest clients we host domains for run email for that domain through Google Apps rather than relying on POP/IMAP through our host.

We do still have a box which accepts incoming SMTP, though, and that means putting up with a certain amount of unsolicited commercial overhead. Every Linux box has a certain number of no-shell users set up to run daemons with limited privilege, e.g. the apache user. Oddly, some spammers either run the same username scans as the brute-force ssh hackers, or they think the apache user is actually reading its email. I found 44MB of unread email, about 5,000 messages, waiting in that inbox. uucp had the next biggest collection; mail and news were right up there as well.

That’s a lot of disk space. After a cursory glance to ensure there was nothing actually important in there, I simply used sudo cp /dev/null /var/spool/mail/apache to take out the trash.

If you are root or have appropriate sudo privileges, you can check another user’s mail using mutt or a similar command-line mail client. Just use the -f flag to feed mutt a mailbox path as an argument. In this case, I could check the apache mailbox using sudo mutt -f /var/spool/mail/apache . Naturally, you would want to have a talk with your company ethicist before doing this on a mailbox belonging to an actual user.

The best shortcut here is to bounce any email destined for the role users. This will vary depending on your MTA, so I won’t detail it here.

filed under: System Administration | comments (1) | read more...

Mar
15
Moving a Rails/Mongrel site to Phusion Passenger on Ubuntu

As I’ve noted before, several of our Rails projects are hosted using Phusion Passenger. We’ve been helping HitFix explore this option in recent weeks because it solves several minor deployment and restart frustrations they’ve been grappling with, and in the process I wrote up this walk-through for the installation.

This process is actually really easy and the instructions on the Passenger site and those provided during the installation process are excellent. The glitch we ran in to is that recent versions of Ubuntu and/or Apache 2.2.x use a slightly different method of loading modules and module configuration. (I haven’t run in to this method often enough to have learned whether it’s new in Apache or in Ubuntu.)

Instead of using one monolithic configuration file, sometimes offloading host configuration to included files, these Apaches instead globally include all modules found in the mods_enabled subdirectory of the Apache directory. Each module has two files, a .load file and a .conf file. The .load file contains only the LoadModule configuration directive; the .conf file contains any configuration directives.

Now, the actual files live in the mods_available subdirectory of Apache, and activating a module just means adding a symlink to the relevant files in mods_enabled, then restarting Apache. Deactivating it just means removing the symlink and restarting again.

With that in mind, here’s the Passenger process:

sudo gem install passenger
sudo passenger-install-apache2-module

At this point, the module is built. This is where we have to adapt Phusion’s original installation instructions. Essentially, the next step is to make sure the module is loaded by Apache. Here’s how I did this:

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/ext/apache2/mod_passenger.so

<IfModule passenger_module>
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.0.6
PassengerRuby /usr/bin/ruby1.8
</IfModule>

sudo ln -s /etc/apache2/mods-available/passenger.load /etc/apache2/mods-enabled/
sudo ln -s /etc/apache2/mods-available/passenger.conf /etc/apache2/mods-enabled/

We can restart Apache at this point without disturbing the existing site. Note that we can change which ruby is used (e.g. for Ruby Enterprise Edition) by altering the path at the PassengerRuby directive and restarting Apache.

To move a site to Passenger, you want to back up the file found at /etc/apache2/sites-available/yoursite, then edit it to make these changes:

The final trick is to update the restart task in Capistrano to handle restarting Passenger and not Mongrel. This is actually pretty easy, particularly if you’ve already overloaded the restart task in your config/deploy.rb file. Just replace the line reading run "mongrel_rails cluster::restart -C #{mongrel_config}" with one reading run "touch #{current_path}/tmp/restart.txt" and we’re good to go. The start and stop tasks probably also needs tweaking to remove the mongrel commands–probably pasting the same command in for “start” will do the trick unless we feel like putting Apache commands in.

filed under: Phusion Passenger, System Administration | comments (0) | read more...

Nov
25
Phusion Passenger, permissions, and Rails’ session store

For various reasons, we’re moving three Rails apps hosted on one VPS to use Phusion Passenger rather than the Apache/Mongrel Cluster stack (via Coda Hale) we’ve been using since we got started.

The Passenger installation process really is as easy as it’s been described, particularly if you’re as comfortable with Apache configuration as we are, but by moving responsibility for serving your apps from Mongrel to Apache, file and process ownership becomes an issue in a way it wasn’t when Mongrel and Apache talked to each other and otherwise minded their own stores. Ben Hughes hinted at this in his walk-through of the Passenger installation, and we were not immune.

Our problem turned out to be session files. The first application we moved to Passenger (and it’s worth noting that apps running on Passenger and apps running Mongrel cluster can coexist under the same Apache) was Redmine, which uses file-based session store by default. If ownership of those session files creates a problem, and it did for us, switching to ActiveRecord session storage resolves the problem.

For those wondering about the point of Passenger, it’s memory use. A single Mongrel instance and an app running under Passenger have comparable RAM requirements, but because Passenger is an Apache module, it can spawn new instances as requests arrive; the Mongrel cluster solution requires multiple mongrel instances to be running all the time. This means Mongrel apps essentially require the RAM for peak capacity all the time. Passenger gives that RAM back at non-peak times, which allows more apps to share the same system, allowing us to get more mileage from our VPSes. (It’s worth noting that the Phusion guys claim a 33% memory savings per instance if Ruby Enterprise Edition is also used, so that will probably be our next step when all the apps are Passenger-ized.)

Update, 12/10: Another gotcha we found was file permissions. Again because of the permissions issues, Rails wasn’t writing to the shared/logs/production.log file. We didn’t notice this immediately, because the applications ran, they just didn’t log anything. The solution here was to change the ownership and permissions on the files. Specifically, we changed group ownership to “nobody” (the group Apache runs under) and the mode of the log file to 664.

filed under: Ruby on Rails, System Administration | comments (1) | read more...

Jun
23
Solutions for Ruby vulnerabilities

The recently-announced vulnerabilities in Ruby have put many of us administering Rails applications in a production space between a rock and a hard place.

To recap, there are three major “lines” of Ruby interpreters, the 1.8.6 line, the 1.8.7 line, and the 1.9 line. All of these show the vulnerability, so nearly everyone needs to update their Ruby. (I’ll get to the exceptions later.) Rails introduces a new line of complication, because the 1.8.7 only works for Rails 2.1 and newer; if the Rails apps in question aren’t ready for Rails 2.1, or already running it (unlikely), the best route would be to continue to update in the 1.8.6 line. So there’s the rock: we need to update our Ruby.

The “hard place” is that the only patches so far released by the Ruby maintainers tend to produce segmentation faults, according to many who have tried them so far. That’s a long way of saying, “They don’t work.”

As a Rails-supporting sysadmin with, you’re left with several options, none of them comfortable. You can, in order of increasing riskiness:

I don’t know which way we’re going yet, but I’m not interested in waiting too long, and the upgrade to Rails 2.1 is going to happen sometime anyway, so Option #2 seems most likely for us.

Update: Hongli Lai from Phusion assures us (in the comments) that Ruby Enterprise Edition can be used as a drop-in replacement for MRE without replacing the entire stack, moving REE up to the front of the line in the “different interpreter” option. Discussion seems to suggest to me that new official patches from the Ruby maintainers will not be coming in a timely fashion, but we are beginning to see “contributed” patched distributions emerge for e.g. FreeBSD and Debian.

filed under: Ruby on Rails, System Administration | comments (2) | read more...

Apr
9
Rails asset hosts, SCM, and bundling

This afternoon we pushed another big revision to the La Cucina Italiana website, which is now sharing a very small fraction of the thousands of recipes that magazine has in its archives. There will be more recipes coming online in the months to come, but most of the puzzles we had to solve are finished now.

One in particular was handling the photos which go with some recipes. Artful photography is a hallmark of the La Cucina Italiana brand, and the editorial team needed to be able to upload their images directly to the site. These photos wouldn’t be stored in the site database, but because they wouldn’t be part of our Subversion repository for the site, either, they had to live outside the normal site root in order to avoid being blown away by any site updates we deployed.

Enter Rails’ asset hosts. Rails allows for assets (e.g. images, CSS files, or Javascript includes) to be served by a host other than that of the main site. Because the asset host definition happens at the host name level, the asset host can actually be on the same box as the main site, or it can be elsewhere in the world; you manage that at a different level of abstraction.

A “free” benefit of asset hosts is that by defining multiple asset hosts (a0 through an), you can fool a user’s browser into downloading your site through a dozen or more different connections, rather than just the two it limits itself two with any single server. Rails will make each asset link use a different asset host, and the browser will open two connections to each server, not caring that they all happen to be on the same box. This gets us extra YSlow points (of course, we lose them again by requiring another DNS lookup for each asset host).

Our hangup, though, was that some assets, specifically CSS and Javascripts, did need to stay inside the Subversion repository and the site’s file tree.

Here’s the solution we came up with:

This way, we get all the benefits of asset hosts for assets which are under revision control, and assets which aren’t. And the asset hosts themselves let us have assets which aren’t necessarily under revision control.

The free bonus here was that by using a symbolic link to the javascripts directory, bundle_fu doesn’t have to know or understand our asset host setup; it just stores its bundled files in that same subdirectory as always, and it just works.

filed under: Ruby on Rails, System Administration | comments (0) | read more...

Dec
8
Restarting Mongrel clusters with Capistrano 2

There’s (still) a glitch between the mongrel_cluster gem and Capistrano 2 (we’re using mongrel_cluster 1.0.5 and Capistrano 2.1.0, for reference) where the application restart at the end of a cap deploy fails with an error like this:

Couldn't find any pid file in '/var/www/[application]/current/tmp/pids' matching 'dispatch.[0-9]*.pid'

I’m not sure what’s causing this, but the solution comes at the end of this post, under the heading “Restart Mongrel.” Due to issues with Cap 2 and sudo, though, the provided script fails for us. We’re running the updates as root (bad idea, but it gets around the cap sudo issues) so I updated the task like this:

# Restart task

set :mongrel_config, "/etc/mongrel_cluster/#{application}.yml"

namespace :deploy do

	task :restart do
		run "mongrel_rails cluster::restart -C #{mongrel_config}"
	end

end

I also commented out the “:mongrel_conf” variable from our previous configuration, which it appears that Capistrano was ignoring anyway.

filed under: Ruby on Rails, System Administration | comments (0) | read more...

Nov
14
Bringing Mongrel back from a server crash

When a server crashes, Mongrel (or a Mongrel cluster) obviously doesn’t get a chance to shut down cleanly. This means it leaves behind the files it uses to store its process IDs. When the server restarts, the Mongrel startup script attempts to start the daemon(s), but on finding these PID files are already present, assumes (incorrectly) that Mongrel is already running, and cancels startup, saying, “PID file log/mongrel.pid already exists. Mongrel could be running already. Check your log/mongrel.log for errors.”

This is technically correct behavior–after all, what if Mongrel really is already running?–but it makes it nearly impossible to bring Mongrel back automatically after a server crash; one would have to manually delete the PID file(s) and then start the daemon(s).

If you don’t have systems administrators tending your websites 24/7, you need a better solution. We considered hacking the init script (found at /etc/rc.d/init.d/mongrel_cluster on our Fedora Core server) but found that the necessary logic made the script too complicated. Instead, we created a new startup script, filed at /etc/rc.d/init.d/mongrel_cleanup to solve the problem.

The mongrel_cleanup script is set to run at the same run-levels as mongrel_cluster. On shutdown or restart, it does nothing, but on start, it checks for the presence of the PID files and deletes them if they’re found. It therefore has to run before mongrel_cluster, which is why the priority number is 84 for startup and 16 for shutdown: mongrel_cluster is 85 and 15.

To use this script, save it in /etc/rc.d/init.d/mongrel_cleanup (or whatever the appropriate script directory is) and then put it in the startup queue with these commands:

# chkconfig --add mongrel_cleanup
# chkconfig --level 345 mongrel_cleanup on

Also, edit this script. I’ve hardwired the paths and names of our Mongrel cluster PID files; you will want to change your paths, or let me know if you come up with a more elegant method!

#!/bin/bash
#
# Parker Morse for Common Media, Inc., 9 November, 2007
#
# mongrel_cleanup      Startup script to recover from crashes.
#
# chkconfig: - 84 16
# description: A hack to clear PID files left behind by Mongrel clusters
#              after an unscheduled server crash. Checks for the presence
#              of these files and deletes them if found.
#              

RETVAL=0
PIDFILE_DIR=/path/to/app/current/log

# Gracefully exit if the controller is missing.
#which mongrel_cluster_ctl >/dev/null || exit 0

# Go no further if config directory is missing.
#[ -d "$CONF_DIR" ] || exit 0

case "$1" in
    start)
      if test -s $PIDFILE_DIR/mongrel.8000.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8000.pid;
      fi
      if test -s $PIDFILE_DIR/mongrel.8001.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8001.pid;
      fi
      if test -s $PIDFILE_DIR/mongrel.8002.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8002.pid;
      fi
      if test -s $PIDFILE_DIR/mongrel.8003.pid
          then
          /bin/rm $PIDFILE_DIR/mongrel.8003.pid;
      fi
      RETVAL=$?
  ;;
    stop)
      exit 0
  ;;
    restart)
      exit 0
  ;;
    *)
      echo "Usage: mongrel_crash_cleanup {start|stop|restart}"
      exit 1
  ;;
esac      

exit $RETVAL

filed under: Ruby on Rails, System Administration | comments (2) | read more...

© 2010 Common Media, Inc. | Theme by DemusDesign and Theme Lab | Powered by WordPress