Common Media, Inc.

Last night, just before the U.S. Olympic Team Trials–Track and Field got underway in Eugene, Oregon, CMI launched a major overhaul of the website for the Track and Field Writers of America (TAFWA).

TAFWA’s previous website, launched in 2001 or 2002, had languished several years without a visual overhaul. The design broke in several modern browsers, some of the site’s content was visibly obsolete, and all of it was difficult to update without the help of the original developers.

We installed Drupal for TAFWA, and installed and configured several modules to customize the popular free CMS to the organization’s specifications. It is now simple for authorized members or officers of the organization to update pages on the site and post organization news without needing to know HTML. It’s also possible for TAFWA to share particular information with its members and not the general public. We imported the membership rolls of the previous website, thus allowing certain parts of the site to require member login.

With its eyes on the marquee sport of the upcoming Olympic Games, TAFWA doesn’t have time to muck around with HTML tags–but they do need a site which looks as sharp and professional as the words of their journalist members. CMI helped them address both aspects of the problem.

filed under: Uncategorized | comments (0) | read more...

The recently-announced vulnerabilities in Ruby have put many of us administering Rails applications in a production space between a rock and a hard place.

To recap, there are three major “lines” of Ruby interpreters, the 1.8.6 line, the 1.8.7 line, and the 1.9 line. All of these show the vulnerability, so nearly everyone needs to update their Ruby. (I’ll get to the exceptions later.) Rails introduces a new line of complication, because the 1.8.7 only works for Rails 2.1 and newer; if the Rails apps in question aren’t ready for Rails 2.1, or already running it (unlikely), the best route would be to continue to update in the 1.8.6 line. So there’s the rock: we need to update our Ruby.

The “hard place” is that the only patches so far released by the Ruby maintainers tend to produce segmentation faults, according to many who have tried them so far. That’s a long way of saying, “They don’t work.”

As a Rails-supporting sysadmin with, you’re left with several options, none of them comfortable. You can, in order of increasing riskiness:

• Wait out the Ruby maintainers and hope they release a working 1.8.6 patchlevel before your app is compromised.
• Do crash-priority Rails upgrades on all your apps and switch to Rails 2.1, then move your Ruby to the 1.8.7 line.
• Switch to an entirely different Ruby interpreter, like Rubinius or JRuby. (This hinges on the idea that Ruby is an interpreted language, and as long as the language is interpreted consistently there is room for any number of interpreters. Apparently JRuby and Rubinius are not affected by these vulnerabilities, only MRI, Matz’s Ruby Interpreter, which is the “standard” implementation.)
• Switch to a different production stack entirely. For example, we’re currently using Apache 2.2 as a proxy for a Mongrel cluster in front of Rails 2.0.2 apps, a not-uncommon configuration. This could be the pressure we’d need to jump to Phusion Passenger, given that their Ruby Enterprise Edition claims to have applied the security patches to MRE 1.8.6 p111, and thus doesn’t introduce the segfault-inducing breaking features that the most-recent patchlevel does.

I don’t know which way we’re going yet, but I’m not interested in waiting too long, and the upgrade to Rails 2.1 is going to happen sometime anyway, so Option #2 seems most likely for us.

Update: Hongli Lai from Phusion assures us (in the comments) that Ruby Enterprise Edition can be used as a drop-in replacement for MRE without replacing the entire stack, moving REE up to the front of the line in the “different interpreter” option. Discussion seems to suggest to me that new official patches from the Ruby maintainers will not be coming in a timely fashion, but we are beginning to see “contributed” patched distributions emerge for e.g. FreeBSD and Debian.

filed under: Ruby on Rails, System Administration | comments (1) | read more...

If you read Wired, you may have heard of Tripletz.com. The Providence-based company allows site users to compose three-part messages on postcards, which are then mailed on consecutive days, a sort of postal Burma Shave arrangement.

When they appeared in Wired, the resulting spike in site traffic exposed some weaknesses in the site architecture. The Slashdot effect is generally an entirely-online situation, but the Tripletz idea was compelling enough (and the name memorable enough) that thousands of readers followed up on the print article and visited the site. Tripletz contacted us to put out the fires. We pushed the first revision yesterday, shifting a big data-sifting operation out of the Rails code (where it was taking so long the page requests would time out) back into the database engine, where it happened in milliseconds.

There’s a lot to be said for the rapid production allowed by Rails, but when it comes to high-traffic situations, it’s worthwhile to be able to look at the application and figure out where bottleneck operations should be taken away from the comfortable, hip tools of Rails and given back to the gritty old database, which almost always performs them faster. Being opinionated about code can be lead to better results in some situations, but making the application work for the client means balancing the code’s poetry with pragmatism–a balance we’re happy to take on, even if we can’t take credit for the site’s undeniably cool idea.

filed under: Business, Ruby on Rails | comments (0) | read more...

We’ve been a long time away from our unfinished running-shoe-selection project, Common Running. We were reminded this morning of the importance of good shoe selection when we spotted this New York Times article on Hitoshi Mimura, the Asics craftsman who makes racing flats for many Olympic marathon medalists and medal contenders, including both gold medalists in Athens and both American Trials winners.

Mimura, for example, made shoes for the 2000 Sydney Olympics gold medalist, Naoko Takahashi, with soles of slightly different thicknesses to compensate for a leg-length discrepancy of 8mm–less than a centimeter, about a third of an inch, according to reporter Jere Longman.

Longman, a longtime follower of the sport and sometime marathoner himself, concludes the article with a vivid simile from Mimura himself on the importance of running shoes:

“Samurai cannot fight without their swords,” Mimura said. “It is the same for runners and their shoes.”

filed under: Common Running | comments (0) | read more...

There are several dates we could use as milestones in the launch of Common Media: the day we agreed, in Rose’s, that we’d give this startup thing a shot. The first few days of business-plan writing. The day our incorporation paperwork asserts was our start date. But the beginning of June was when we actually sat down and started coding on Common Kitchen, the time we really became full-time employees of ourselves.

So it’s been a year now. It doesn’t seem like that long, and it certainly hasn’t conformed at all to the grandiose numbers we put in the (now long-obsolete) business plan. We even have direct deposit now, so we can’t claim to be signing our own paychecks anymore.

There are plenty of startups in the world which don’t make it to their first birthday, though, and like children, the longer a startup stays in business, the longer it’s likely to stay in business. Our first year made it more likely that we’ll finish a second, and we’re actually pretty confident about that second year right now.

And the business plans in our heads still have some interesting things in them, so stay tuned.

filed under: Business | comments (0) | read more...

OK, a very small way.

Like many small companies doing development on the Web, Common Media depends heavily on free and open-source software. Part of the point of open source is that a programmer using the program (or library, or plug-in) may get in to the code and wrangle it around until it works best for them. The obligation that comes with that freedom is to “give back” any such changes if they may be useful to the wider community. With big projects, that can mean active participation in a coding community; for smaller packages, it may just mean sending code back to the maintainer for consideration.

We mentioned a few weeks ago how we tweaked a plug-in for Common Kitchen. Today, that code became our first checked-in contribution to an open-source project, Netphase’s acts_as_amazon_product. Hopefully it won’t be the last.

(I also took the opportunity to use a topical test case. Check the commit to see which magazine we test magazine searching with.)

filed under: Ruby on Rails | comments (0) | read more...

This afternoon we pushed another big revision to the La Cucina Italiana website, which is now sharing a very small fraction of the thousands of recipes that magazine has in its archives. There will be more recipes coming online in the months to come, but most of the puzzles we had to solve are finished now.

One in particular was handling the photos which go with some recipes. Artful photography is a hallmark of the La Cucina Italiana brand, and the editorial team needed to be able to upload their images directly to the site. These photos wouldn’t be stored in the site database, but because they wouldn’t be part of our Subversion repository for the site, either, they had to live outside the normal site root in order to avoid being blown away by any site updates we deployed.

Enter Rails’ asset hosts. Rails allows for assets (e.g. images, CSS files, or Javascript includes) to be served by a host other than that of the main site. Because the asset host definition happens at the host name level, the asset host can actually be on the same box as the main site, or it can be elsewhere in the world; you manage that at a different level of abstraction.

A “free” benefit of asset hosts is that by defining multiple asset hosts (a0 through an), you can fool a user’s browser into downloading your site through a dozen or more different connections, rather than just the two it limits itself two with any single server. Rails will make each asset link use a different asset host, and the browser will open two connections to each server, not caring that they all happen to be on the same box. This gets us extra YSlow points (of course, we lose them again by requiring another DNS lookup for each asset host).

Our hangup, though, was that some assets, specifically CSS and Javascripts, did need to stay inside the Subversion repository and the site’s file tree.

Here’s the solution we came up with:

• Defined an Apache virtual server answering to subdomains a0 through a4. (In other words, all four subdomains are the same real server, but the browser doesn’t know that; we could make them different physical servers someday.)
• Set the site root for that server to /var/www/assets/.
• Made symbolic links from /var/www/assets/javascripts to /var/www/production/current/public/javascripts, and from /var/www/assets/stylesheets to /var/www/production/current/public/stylesheets. (N.B. if you’re using Capistrano, “current” in your site directory is itself a symbolic link to a directory with your last site deploy.)
• Set the upload code for recipe photos to store the uploaded images in subdirectories of /var/www/assets/images.

This way, we get all the benefits of asset hosts for assets which are under revision control, and assets which aren’t. And the asset hosts themselves let us have assets which aren’t necessarily under revision control.

The free bonus here was that by using a symbolic link to the javascripts directory, bundle_fu doesn’t have to know or understand our asset host setup; it just stores its bundled files in that same subdirectory as always, and it just works.

filed under: Ruby on Rails, System Administration | comments (0) | read more...

When Yahoo! released it’s YSlow application last year as a plugin for the Firebug Firefox extension (because really, what web developers don’t have Firefox installed, even if it isn’t their primary browser?) nearly everyone installed the tool and started going down the list of rules Yahoo! laid down for improving “front-end performance” on websites. Several people wrote up suggestions for using the output to improve Rails apps, including a good summary for Nginx, but we’re using Apache.

“Front-end performance” means attacking speed as a problem between the browser and the server, and not as a problem which exists solely behind the server. We can optimize an application as much as we want, but if the browser makes thirty-five round-trips to the server fetching CSS, image files, and JavaScripts, application optimization isn’t helping much. In addition to grading apps on these fourteen points, YSlow gives a load time (in milliseconds!), and as your grade improves, you can also see the load time improving by perceptible intervals.

This afternoon I ran YSlow on the current development version of the La Cucina Italiana site. The initial grade was 52, an F. When I was done, it was 88, a B, and if I circumvent a dubious aspect of YSlow, it becomes a 98 A. I made only four changes: three Apache configuration tweaks and one Rails change. If you’re a Rails developer with Apache 2.2 in your stack, here are the low-hanging fruit for a better YSlow score. You can make these changes in your site configuration or within an .htaccess file.

ETags: The Yahoo! explanation of how ETags are important is a little confusing. The summary is this: you want to turn ETags off. They’re ineffective with multiple servers (i.e. an asset-server setup) and even if everything is on one host, there are other, just as useful means of avoiding downloads of cached files. So spare yourself that many bytes per connection and turn them off. It’s a one-line fix:

FileETag none

GZip components: The time saved by sending down compressed files is greater than the time spent compressing them, but only for text-based file types. (Images and PDFs are already compressed, so re-compression won’t help and might hurt.) If you’re using Coda Hale’s configuration for Apache and Mongrel (and many of us are) the code for compressing text components before download is already in your Apache configuration. However, Coda’s config misses one file type which Rails seems to use for Javascript, application/x-javascript, so YSlow keeps dinging us for uncompressed JavaScript files. With that added, the configuration for compression looks like this:

AddOutputFilterByType DEFLATE text/html text/plain text/xml application/xml application/xhtml+xml text/javascript text/css application/x-javascript
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

(Note that there should be only four lines there: one AddOutputFilterByType and then three BrowserMatch lines.)

Far-future cache expiration: Yahoo! is looking for really far future expiration dates for cache expiration. The reasoning here is to maximize the odds that your users are arriving at your site with a “primed” cache, i.e. one where most of your components are already loaded. By putting a way-out expiration on components, users keep those bits in their cache longer. The flip side is that filenames need to change in order to prompt the browser to load a changed file. Rails does this by automatically appending a timestamp to the filenames of many components. (Check your page source and see what is actually getting called when you load a CSS file.) Therefore, we can set our default expires header way out in the future.

This is a two-line addition to the Apache configuration:

ExpiresActive On
ExpiresDefault "access plus 10 years"

Reducing connection count: This is the tough one. Default Rails apps load a zillion (OK, seven) Javascript files and at least one but possibly many more CSS files, and that’s before we start with page images. Most browsers will only open two or three connections at a time to a given server, so all the files are waiting in line to get loaded. You can maximize the number of concurrent connections by using an asset server; this fools the browser into opening more connections by giving it more hosts to connect to. (This is like opening more Starbucks’ in the same town.) That doesn’t really save you, though, and it boosts the number of DNS lookups the browser needs to run. To really cut the number of JS and CSS files, you need to bundle assets. It’s possible to bundle images, but that’s not a problem for LCI, which has only four or five images on the home page. Bundling JS and CSS is where the action is.

There are multiple ways to do this, including a nifty program called AssetPackager and the caching code built in to Rails 2.0. We went with what we saw as the simplest route, which was bundle_fu, but we have an eye on AssetPackager for future consideration. bundle_fu is a Rails plug-in which takes all the CSS and JS files for a template and concatenates them into one CSS and one JS file, sometimes even minifying the JS. It’s a quick installation, and not only does it convince YSlow to give your site a good grade on the number of connections, it also gives good grades for “Put JS at the bottom” and “Minify JS” because, hey, there’s only one file, right? This one move improved our YSlow score more than any other step.

The score at this point: 88 points, a high B. We’re still suffering for not having minified our JS (a little, and we’ve decided not to bother, since it’s being gzipped anyway) and for not using a CDN.

Cheating: The CDN point is one of the most hotly-contested on the YSlow report card, because most sites as small as ours see very little return on the investment of putting our site on a Content Delivery Network such as Akamai. It’s possible, though, to get YSlow to turn a blind eye to your CDN-less-ness. Just add your own site’s domain to the list of CDN servers in YSlow’s preferences. Doing that jumped our score up to 98–pretty close to perfect.

Unfixable: One thing which may keep your site from ever having a good YSlow report is using a lot of outside components. Common Kitchen gets dinged because its page loads include calls to two different ad networks, not to mention multiple images from Amazon on some pages and the Google Analytics code. This makes for a lot of un-bundle-able scripts, multiple DNS requests, and a lot of components where we don’t control their ETags (or lack thereof), expiration header, or compression. Of course, we can’t control their server uptime, either, so it may be that the YSlow scores are the least of our problems!

filed under: Ruby on Rails | comments (1) | read more...

We’ve not been doing very well at reporting our activities here, even though things have been happening which are worth reporting. Forthwith, an update.

La Cucina Italiana occupied a significant chunk of our March. Phase Two of that project, which will see the beginnings of that venerable publication’s massive recipe database becoming available on the site, is complete and is only awaiting editorial oversight of the recipes being entered in the database. Phase Three is under discussion, and is the most exciting part from our point of view, as it will allow the site’s readers to engage the brand directly on the site through comments, ratings, and discussion.

We’re drawing closer to the public launch of an interesting project we started last fall with Scott Soloway. We’ll tell you more about this site when it launches, but for the moment we’ll say that it’s a web service of a sort we’d never handled before, and it includes a strong machine-learning component, which we loved.

We learned just a day or two ago that we’ll be building a new iteration of the website for the Track and Field Writers of America (TAFWA), a professional organization Parker has actually belonged to since his days at Runner’s World. Like many professional organizations, TAFWA has a specific audience to address and clear goals and constraints for their site, but their skills tend not to lie in web development or HTML coding. Our goals for this site are to create a site which is easy for non-technical users to maintain but which will help TAFWA itself provide useful services for its members.

With all this contract work going on, you’d worry that our own projects are being neglected. To the contrary, both “common” sites have seen development activity in recent weeks, though not necessarily in visible ways. My post earlier this week sprung from work I’m doing implementing a machine-learning based shoe recommendation system for Common Running, for example, and we’re gearing up to refresh the shoe database with the latest models.

Common Kitchen got a typical update earlier this afternoon, in which we updated the process for adding cookbooks and magazines to the site. This involved ripping out a chunk of code and replacing it with the neater, tighter acts_as_amazon_product code from Netphase. Their code searches books only by default, so we tweaked the plugin a bit ourselves to handle magazines and other product types using an optional parameter. The difference is pretty slight from the user’s end, but it’s typical of the work we’re doing on Common Kitchen nowadays, where we trim, tighten and otherwise refactor the site’s code based on what we’ve learned since we started.

There’s more CMI work waiting in the wings, both some potential outside projects and some ideas we’re brewing for our own sites. We’ll try to keep you posted here!

filed under: Common Kitchen, Common Running, Web development | comments (1) | read more...

Installing Ruby’s linalg linear algebra library on a Mac OS X system is problematic because linalg is built around LAPACK, the Linear Algebra PACKage, and OS X (at least the version I’m working with, 10.4.11) ships with a bastard version of LAPACK which is missing some important symbols for linalg. The way around this is to install a full version of LAPACK.

You’d think this would be easy, but LAPACK is written in FORTRAN, and the version of gcc included with Xcode doesn’t include a FORTRAN compiler by default. So before we can install LAPACK, we need a FORTRAN compiler.

There are at least two gcc-based FORTRAN compilers out there, and they both offer pre-compiled binaries for Mac OS X (Intel or PowerPC). You can build from source if that’s how you want to do it, but I want this over with, so I’m grabbing the Intel binaries and getting on with my life. LAPACK seems to have trouble dealing with the g95 compiler, so we went with gfortran. gfortran has a nice Mac-like .dmg installer, so you can just download that, click through the installation, and gfortran is ready in /usr/local/bin/ (which is hopefully in your $PATH). You can make a link so that the g77 command (the old gcc compiler for the FORTRAN77 standard) points to gfortran with this command:

sudo ln -s /usr/local/bin/gfortran /usr/local/bin/g77

Now you’ll want to start in on LAPACK. Download the tarball from http://www.netlib.org/lapack/lapack.tgz and store it in /usr/local/src/ as before. Unpack with

tar xzvf lapack.tgz

You’ll have created a directory named e.g. lapack-3.1.1. cd into this directory. What’s missing from LAPACK is the standard ./configure step; we’ll have to edit the make.inc file ourselves before running make to build the package.

Fortunately, Robert Hatcher builds LAPACK as part of his CERNLIB build, which means that the shell commands for creating a working LAPACK make.inc are available as part of that script. Here’s the relevant excerpt:

# customize makefile
sed -e 's/_LINUX/_DARWIN/' make.inc.example > make.inc
echo "" >> make.inc
echo ".SUFFIXES : .f .o" >> make.inc
echo "" >> make.inc

# go ahead and build - "all" will perform tests
make blaslib lapacklib tmglib > make.log 2>&1
if [ $? != 0 ] ; then
echo “*** Error in make blaslist lapacklib tmglib ***”
grep -i err make.log
fi

Now: this step will take a while. If you copied this all into a file and ran it as a shell script (the sane thing to do, I think), it will take a good while to run, on the order of ten or fifteen minutes; if you are keying in the commands line by line, it will pause long after the make blaslib lapacklib tmglib line. Don’t panic; this means it’s working. (If you’re paranoid and like seeing stuff stream across your screen to prove you’re compiling something, you may want to background the process and then use tail -f make.log to get the full output.)

Once it’s done, it’s time to put these files where they belong:

sudo cp blas_DARWIN.a /usr/local/lib/libblas.a
sudo cp lapack_DARWIN.a /usr/local/lib/liblapack3.a

(Note that it may be the case that there are faster BLAS libraries out there; if you’re squeezing every cycle out of your app, it may be worth looking into that, but it’s beyond the scope of this post.)

Unfortunately, we’re still not done. linalg still needs several libraries from the f2c package, which is quite hard to dig up. The best route I’ve found is to grab the package available through Fink. The trick is that Fink installs libraries in /sw/lib/ and we need them elsewhere (/usr/local/lib/ should work). Use a link to solve that:

sudo ln -s /sw/lib/libf2c.a /usr/local/lib/

Now it’s (finally) time to install linalg. Unfortunately, there’s no gem available for this that I’m aware of. (This may be because the package has been essentially “done” for four or five years, so it’s older than the widespread use of gem.) Download the tarball from the project page to /usr/local/src/ and un-tar it; you’ll get a folder named linalg-0.3.2. cd into that folder, and you should be able to use

sudo ruby install.rb

…but you can’t, actually. This builds most of the files you need, barring two; it will start the installation, but eventually stall because it’s missing two .so files, ext/linalg/linalg.so and ext/lapack/lapack.so. These are “Shared Object” files, akin to Windows DLLs, but the Makefiles in these directories defines the DLLIB macro as ending with the .bundle extension, and linalg.bundle is what gets built.

So, we brute-force it by breaking the process down into “make” and “install”, and in between we create those .so files.

ruby install.rb make
cp ext/linalg/linalg.bundle ext/linalg/linalg.so
cp ext/lapack/lapack.bundle ext/lapack/lapack.so
sudo ruby install.rb install

If you don’t trust this hack, put in ruby install.rb test before the install task to verify that everything works. I’m not sure why the package tries to install an .so file its own makefiles don’t build; if someone can figure that out and patch it, I’m sure the maintainers would love to know.

If you find any obvious errors in this, or see some steps we can stick, feel free to comment and we’ll make edits. Hopefully this will come in handy for someone.

filed under: Common Running, Ruby on Rails | comments (7) | read more...