Archive for the ‘Ruby on Rails’ Category

Nginx configuration tweaks for client-side speed optimization

Thursday, May 6th, 2010

As we move more sites to an Nginx/Passenger server stack, we need to translate the server-side optimizations we use for browser-side performance from Apache configuration to Nginx. Here’s how that comes over:

Gzip compression

In Nginx, this is an optional module which should be built in to the software at compile time. If it’s present, you can activate it in site configuration with a block like this:

 gzip on;
 gzip_min_length  1100;
 gzip_buffers     4       8k;
 gzip_proxied     any;
 gzip_types       text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;

Note that Nginx considers text/xml and text/html to be redundant MIME types.

Long expiration dates

We want to have our assets cached in the user’s browser cache as long as possible, with a few exceptions (e.g. advertising assets.) To that end, we set expiration headers:

       location ~* \.(js|css|jpg|jpeg|gif|png)$ {
                if (-f $request_filename) {
                   expires                        max;
                   break;
                }
       }

This uses the filename extension to determine which files get “far future” expiration dates. If you’re used to naming files without extensions, this might not work so well for you.

By default, Nginx doesn’t use Etags, so we can stop worrying about that.

New Client Site: TABBForum

Tuesday, January 26th, 2010

Tabb ForumIf you’re interested in capital markets, check out our newest client site, Tabb Forum.

Passenger, Paperclip, and Permissions (another route)

Tuesday, June 9th, 2009

We had issues with file uploads recently on a site using Paperclip to handle uploads and served with Passenger. The site worked fine in development, but in production the uploads would fail. Checking the production log showed that Paperclip was producing the following error:

[paperclip] An error was received while processing: #<Paperclip::NotIdentifiedByImageMagickError: /tmp/passenger.1358/var/stream.25635.1 is not recognized by the 'identify' command.>

A web search produced this railsforum thread with some helpful advice, specifically to set

Paperclip.options[:command_path] = "/usr/local/bin"

in an initializer for Paperclip. Our problem was environment-specific, though, so we put it in the config/environment/production.rb file instead. No luck.

So I checked the Apache logs to see what error Passenger was producing. It wasn’t throwing an error of its own; instead we were seeing a lot of

[DEPRECATION] S3 support through RightAWS is deprecated. S3 support will be changed to AWS::S3 in a future version.

…in the Apache error log. It turned out that our development boxes were all using the aws-s3 gem, and the production box was still on RightAWS. When we installed aws-s3 on the server, the problem was solved.

So if you’re having Passenger and Paperclip problems, make sure you’re using the right interface with s3 as part of your troubleshooting process.

Rails incorrectly caches RSS feed as HTML

Tuesday, May 26th, 2009

Here’s the scenario: a site with a page which is served in two formats. http://www.example.com/sample returns HTML, and shouldn’t cache it. http://www.example.com/sample.rss returns an RSS feed, which should be cached. In Rails, we should be able to achieve this by including this code at the top of the controller:

caches_page :sample, :if => Proc.new { |c| c.request.format.rss? }

What this is saying is, if the format of the request is rss, cache it; otherwise, don’t. Because this was page cached, it would create a file public/sample.rss which would get returned in response to any requests; deleting this file equates to invalidating the cache.

The symptom we started seeing, irregularly, was that suddenly http://www.example.com/sample would return RSS. This turned out to be because the RSS feed was getting saved in public/sample.html. To solve the problem, we had to figure out how the RSS feed was getting cached as if it was HTML.

The answer turned out to be a poorly-formatted URL. If you request http://www.example.com/sample?format=rss, the response is (correctly) RSS, but the extension of the cached file is (incorrectly) .html.

In order to work around this bug, we removed the caches_page line from the top of the controller, and instead used cache_page within the response block of the action in question. cache_page accepts two parameters, the string to be cached and the key (the path, in this case) to cache it at, so we could force the cache filename.


respond_to do |format|
format.html
format.rss {
render :layout => false
cache_page(@response, "/sample.rss") }
end

The result here is that HTML is never cached (as required), and when RSS is cached, it is always cached with an .rss extension. The drawback is that the funky URL will never get a cache hit; it will always be generated from Rails (and update the cache while it’s at it). However, that’s an acceptable trade-off for not generating bogus cache files in the first place.

The kludgy part of this–aside from needing to do it in the first place–is using @response in the controller, which we’re not supposed to do. I wish there was a more elegant way of getting the content of the response for caching.

I think this is a bug somewhere in Rails’ caching code, but I haven’t yet dug in to see where or how it could be fixed.

Moving a Rails/Mongrel site to Phusion Passenger on Ubuntu

Sunday, March 15th, 2009

As I’ve noted before, several of our Rails projects are hosted using Phusion Passenger. We’ve been helping HitFix explore this option in recent weeks because it solves several minor deployment and restart frustrations they’ve been grappling with, and in the process I wrote up this walk-through for the installation.

This process is actually really easy and the instructions on the Passenger site and those provided during the installation process are excellent. The glitch we ran in to is that recent versions of Ubuntu and/or Apache 2.2.x use a slightly different method of loading modules and module configuration. (I haven’t run in to this method often enough to have learned whether it’s new in Apache or in Ubuntu.)

Instead of using one monolithic configuration file, sometimes offloading host configuration to included files, these Apaches instead globally include all modules found in the mods_enabled subdirectory of the Apache directory. Each module has two files, a .load file and a .conf file. The .load file contains only the LoadModule configuration directive; the .conf file contains any configuration directives.

Now, the actual files live in the mods_available subdirectory of Apache, and activating a module just means adding a symlink to the relevant files in mods_enabled, then restarting Apache. Deactivating it just means removing the symlink and restarting again.

With that in mind, here’s the Passenger process:

  • Install Passenger on the server. This should be as simple as:

sudo gem install passenger
sudo passenger-install-apache2-module

  • On one system, this second command actually told me that the Apache 2 Development Headers were missing. (Passenger needs these to compile the Apache module.) I needed to run sudo apt-get install apache2-prefork-dev to get these headers. This didn’t work on the first attempt, but running sudo apt-get update once or twice finally got me a successful install of that package. Running the passenger-install-apache2-module script then worked successfully. Note that you will need a compiler on the system in order to generate the Apache module. You would think you could take this for granted, but apparently not.

At this point, the module is built. This is where we have to adapt Phusion’s original installation instructions. Essentially, the next step is to make sure the module is loaded by Apache. Here’s how I did this:

  • Create a file at /etc/apache2/mods-available/passenger.load with the following content:

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.0.6/ext/apache2/mod_passenger.so

  • Create a file at /etc/apache2/mods-available/passenger.conf with the following content (I think the IfModule container may be optional, but I played safe):

<IfModule passenger_module>
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.0.6
PassengerRuby /usr/bin/ruby1.8
</IfModule>

  • Create symbolic links to both those files from /etc/apache2/mods-enabled/, i.e.

sudo ln -s /etc/apache2/mods-available/passenger.load /etc/apache2/mods-enabled/
sudo ln -s /etc/apache2/mods-available/passenger.conf /etc/apache2/mods-enabled/

We can restart Apache at this point without disturbing the existing site. Note that we can change which ruby is used (e.g. for Ruby Enterprise Edition) by altering the path at the PassengerRuby directive and restarting Apache.

To move a site to Passenger, you want to back up the file found at /etc/apache2/sites-available/yoursite, then edit it to make these changes:

  • Change the DocumentRoot to the public directory of the app, e.g. DocumentRoot /u/apps/yoursite/current/public
  • Comment out (with a #) all the lines related to the proxy, starting with the Proxy balancer definition and extending through all the ProxyPass, ProxyPassReverse and ProxyPreserveHost directives.
  • Restart Apache (/etc/init.d/apache2 restart). If something goes wrong, restore the backup you made of the configuration file and restart Apache again to restore the previous configuration.

The final trick is to update the restart task in Capistrano to handle restarting Passenger and not Mongrel. This is actually pretty easy, particularly if you’ve already overloaded the restart task in your config/deploy.rb file. Just replace the line reading run "mongrel_rails cluster::restart -C #{mongrel_config}" with one reading run "touch #{current_path}/tmp/restart.txt" and we’re good to go. The start and stop tasks probably also needs tweaking to remove the mongrel commands–probably pasting the same command in for “start” will do the trick unless we feel like putting Apache commands in.

HitFix is in Beta

Wednesday, December 31st, 2008

The site we spent the later part of December working on is now in public beta. HitFix is not the kind of site Noah and I imagined ourselves working on when we got started; it’s a site about entertainment and popular culture with commentary by industry insiders. But they needed more bodies who knew Rails as they pushed to launch, and their designer knew us from his previous job, so we got a call.

We can’t take credit for any individual aspects of the HitFix site; what we did was work as a team with HitFix’s internal Rails developer, Elliott Blatt, to make the site Beta-ready. This meant taking Photoshop demos and making fully-functional pages from them, working at all levels of the Rails stack. (We’ve been joking that our work on this site was in defiance of Brooks’s Law.)

Elliott has been a thrill to work with, having more experience with Rails than either of us, and we were surprised to find him learning from us almost as much as we were learning from him. What we’ve learned from this site is significant as well; it’s impossible to detail all the small conventions we’ve learned from Elliott, but one big example was learning HAML and its CSS sister, SASS.

It has also been an experiment in decentralized development, as some of the HitFix team is in California, Elliott is in Cambridge, and we’re in Amherst.

We’re still in active development with HitFix, but as I mentioned before, the beta is up, so if you’d like to take a look, you may find it interesting.

Phusion Passenger, permissions, and Rails’ session store

Tuesday, November 25th, 2008

For various reasons, we’re moving three Rails apps hosted on one VPS to use Phusion Passenger rather than the Apache/Mongrel Cluster stack (via Coda Hale) we’ve been using since we got started.

The Passenger installation process really is as easy as it’s been described, particularly if you’re as comfortable with Apache configuration as we are, but by moving responsibility for serving your apps from Mongrel to Apache, file and process ownership becomes an issue in a way it wasn’t when Mongrel and Apache talked to each other and otherwise minded their own stores. Ben Hughes hinted at this in his walk-through of the Passenger installation, and we were not immune.

Our problem turned out to be session files. The first application we moved to Passenger (and it’s worth noting that apps running on Passenger and apps running Mongrel cluster can coexist under the same Apache) was Redmine, which uses file-based session store by default. If ownership of those session files creates a problem, and it did for us, switching to ActiveRecord session storage resolves the problem.

For those wondering about the point of Passenger, it’s memory use. A single Mongrel instance and an app running under Passenger have comparable RAM requirements, but because Passenger is an Apache module, it can spawn new instances as requests arrive; the Mongrel cluster solution requires multiple mongrel instances to be running all the time. This means Mongrel apps essentially require the RAM for peak capacity all the time. Passenger gives that RAM back at non-peak times, which allows more apps to share the same system, allowing us to get more mileage from our VPSes. (It’s worth noting that the Phusion guys claim a 33% memory savings per instance if Ruby Enterprise Edition is also used, so that will probably be our next step when all the apps are Passenger-ized.)

Update, 12/10: Another gotcha we found was file permissions. Again because of the permissions issues, Rails wasn’t writing to the shared/logs/production.log file. We didn’t notice this immediately, because the applications ran, they just didn’t log anything. The solution here was to change the ownership and permissions on the files. Specifically, we changed group ownership to “nobody” (the group Apache runs under) and the mode of the log file to 664.

Rails asset hosts and SSL

Tuesday, July 15th, 2008

We’ve previously mentioned the asset host setup we created for La Cucina Italiana. For the most recent revision, which includes user registration, we had to add another layer to the mix: SSL.

The standard operating procedure for Rails applications and SSL is to create the SSL host (which functions, in Apache, as another virtual host) and proxy it directly to the same Ruby httpd you’re using to run the rest of the site, with the added request header to indicate that yes, this one came in via SSL.

The problem is that Rails then generates a page in which all the sub-requests use SSL. This is a problem when you’re using asset hosts, unless you’ve either paid extra for a certificate that covers your asset host subdomains (and why this should cost extra is beyond me, by the way, but nearly nobody does it) or are willing to force Rails to not securely link page assets, which leads to “mixed content” warnings from many browsers (a secure page including insecure components.)

The solution to this problem is in the Rails documentation where the asset hosts are described in the first place: you generate the asset hosts using a proc. There are two gotchas here, though: First, you need to be using Rails 2.1.0 or better. (This, combined with the Ruby problems that came up last month, pushed La Cucina to Rails 2.1.) The example in the documentation, however, only allows for one asset host, and the %d solution for randomizing across four hosts doesn’t work within a proc. Here’s how to manage it, if you weren’t able to figure that out from the documentation:

ActionController::Base.asset_host = Proc.new { |source, request|
  if request.ssl?
    "#{request.protocol}#{request.host_with_port}"
  else
    "#{request.protocol}a#{rand(4)}.lacucinaitalianamagazine.com"
  end
}

Essentially, this turns off asset hosting for requests using SSL, which means those requests will come through the application itself. Fortunately for us, all the SSL-requiring pages don’t use content which isn’t under source control!

Solutions for Ruby vulnerabilities

Monday, June 23rd, 2008

The recently-announced vulnerabilities in Ruby have put many of us administering Rails applications in a production space between a rock and a hard place.

To recap, there are three major “lines” of Ruby interpreters, the 1.8.6 line, the 1.8.7 line, and the 1.9 line. All of these show the vulnerability, so nearly everyone needs to update their Ruby. (I’ll get to the exceptions later.) Rails introduces a new line of complication, because the 1.8.7 only works for Rails 2.1 and newer; if the Rails apps in question aren’t ready for Rails 2.1, or already running it (unlikely), the best route would be to continue to update in the 1.8.6 line. So there’s the rock: we need to update our Ruby.

The “hard place” is that the only patches so far released by the Ruby maintainers tend to produce segmentation faults, according to many who have tried them so far. That’s a long way of saying, “They don’t work.”

As a Rails-supporting sysadmin with, you’re left with several options, none of them comfortable. You can, in order of increasing riskiness:

  • Wait out the Ruby maintainers and hope they release a working 1.8.6 patchlevel before your app is compromised.
  • Do crash-priority Rails upgrades on all your apps and switch to Rails 2.1, then move your Ruby to the 1.8.7 line.
  • Switch to an entirely different Ruby interpreter, like Rubinius or JRuby. (This hinges on the idea that Ruby is an interpreted language, and as long as the language is interpreted consistently there is room for any number of interpreters. Apparently JRuby and Rubinius are not affected by these vulnerabilities, only MRI, Matz’s Ruby Interpreter, which is the “standard” implementation.)
  • Switch to a different production stack entirely. For example, we’re currently using Apache 2.2 as a proxy for a Mongrel cluster in front of Rails 2.0.2 apps, a not-uncommon configuration. This could be the pressure we’d need to jump to Phusion Passenger, given that their Ruby Enterprise Edition claims to have applied the security patches to MRE 1.8.6 p111, and thus doesn’t introduce the segfault-inducing breaking features that the most-recent patchlevel does.

I don’t know which way we’re going yet, but I’m not interested in waiting too long, and the upgrade to Rails 2.1 is going to happen sometime anyway, so Option #2 seems most likely for us.

Update: Hongli Lai from Phusion assures us (in the comments) that Ruby Enterprise Edition can be used as a drop-in replacement for MRE without replacing the entire stack, moving REE up to the front of the line in the “different interpreter” option. Discussion seems to suggest to me that new official patches from the Ruby maintainers will not be coming in a timely fashion, but we are beginning to see “contributed” patched distributions emerge for e.g. FreeBSD and Debian.

Dealing with a “print Slashdotting”

Tuesday, June 17th, 2008

If you read Wired, you may have heard of Tripletz.com. The Providence-based company allows site users to compose three-part messages on postcards, which are then mailed on consecutive days, a sort of postal Burma Shave arrangement.

When they appeared in Wired, the resulting spike in site traffic exposed some weaknesses in the site architecture. The Slashdot effect is generally an entirely-online situation, but the Tripletz idea was compelling enough (and the name memorable enough) that thousands of readers followed up on the print article and visited the site. Tripletz contacted us to put out the fires. We pushed the first revision yesterday, shifting a big data-sifting operation out of the Rails code (where it was taking so long the page requests would time out) back into the database engine, where it happened in milliseconds.

There’s a lot to be said for the rapid production allowed by Rails, but when it comes to high-traffic situations, it’s worthwhile to be able to look at the application and figure out where bottleneck operations should be taken away from the comfortable, hip tools of Rails and given back to the gritty old database, which almost always performs them faster. Being opinionated about code can be lead to better results in some situations, but making the application work for the client means balancing the code’s poetry with pragmatism–a balance we’re happy to take on, even if we can’t take credit for the site’s undeniably cool idea.