Building LAPACK and Ruby’s linalg on Mac OS X

Mar
24

Update, 7 September 2008: Before you actually do anything with this post, make sure to read the update and comments at the bottom.

Installing Ruby’s linalg linear algebra library on a Mac OS X system is problematic because linalg is built around LAPACK, the Linear Algebra PACKage, and OS X (at least the version I’m working with, 10.4.11) ships with a bastard version of LAPACK which is missing some important symbols for linalg. The way around this is to install a full version of LAPACK.

You’d think this would be easy, but LAPACK is written in FORTRAN, and the version of gcc included with Xcode doesn’t include a FORTRAN compiler by default. So before we can install LAPACK, we need a FORTRAN compiler.

There are at least two gcc-based FORTRAN compilers out there, and they both offer pre-compiled binaries for Mac OS X (Intel or PowerPC). You can build from source if that’s how you want to do it, but I want this over with, so I’m grabbing the Intel binaries and getting on with my life. LAPACK seems to have trouble dealing with the g95 compiler, so we went with gfortran. gfortran has a nice Mac-like .dmg installer, so you can just download that, click through the installation, and gfortran is ready in /usr/local/bin/ (which is hopefully in your $PATH). You can make a link so that the g77 command (the old gcc compiler for the FORTRAN77 standard) points to gfortran with this command:

sudo ln -s /usr/local/bin/gfortran /usr/local/bin/g77

Now you’ll want to start in on LAPACK. Download the tarball from http://www.netlib.org/lapack/lapack.tgz and store it in /usr/local/src/ as before. Unpack with

tar xzvf lapack.tgz

You’ll have created a directory named e.g. lapack-3.1.1. cd into this directory. What’s missing from LAPACK is the standard ./configure step; we’ll have to edit the make.inc file ourselves before running make to build the package.

Fortunately, Robert Hatcher builds LAPACK as part of his CERNLIB build, which means that the shell commands for creating a working LAPACK make.inc are available as part of that script. Here’s the relevant excerpt:

# customize makefile sed -e 's/_LINUX/_DARWIN/' make.inc.example > make.inc echo "" >> make.inc echo ".SUFFIXES : .f .o" >> make.inc echo "" >> make.inc

# go ahead and build - "all" will perform tests make blaslib lapacklib tmglib > make.log 2>&1 if [ $? != 0 ] ; then echo “*** Error in make blaslist lapacklib tmglib ***” grep -i err make.log fi

Now: this step will take a while. If you copied this all into a file and ran it as a shell script (the sane thing to do, I think), it will take a good while to run, on the order of ten or fifteen minutes; if you are keying in the commands line by line, it will pause long after the make blaslib lapacklib tmglib line. Don’t panic; this means it’s working. (If you’re paranoid and like seeing stuff stream across your screen to prove you’re compiling something, you may want to background the process and then use tail -f make.log to get the full output.)

Once it’s done, it’s time to put these files where they belong:

sudo cp blas_DARWIN.a /usr/local/lib/libblas.a sudo cp lapack_DARWIN.a /usr/local/lib/liblapack3.a

(Note that it may be the case that there are faster BLAS libraries out there; if you’re squeezing every cycle out of your app, it may be worth looking into that, but it’s beyond the scope of this post.)

Unfortunately, we’re still not done. linalg still needs several libraries from the f2c package, which is quite hard to dig up. The best route I’ve found is to grab the package available through Fink. The trick is that Fink installs libraries in /sw/lib/ and we need them elsewhere (/usr/local/lib/ should work). Use a link to solve that:

sudo ln -s /sw/lib/libf2c.a /usr/local/lib/

Now it’s (finally) time to install linalg. Unfortunately, there’s no gem available for this that I’m aware of. (This may be because the package has been essentially “done” for four or five years, so it’s older than the widespread use of gem.) Download the tarball from the project page to /usr/local/src/ and un-tar it; you’ll get a folder named linalg-0.3.2. cd into that folder, and you should be able to use

sudo ruby install.rb

…but you can’t, actually. This builds most of the files you need, barring two; it will start the installation, but eventually stall because it’s missing two .so files, ext/linalg/linalg.so and ext/lapack/lapack.so. These are “Shared Object” files, akin to Windows DLLs, but the Makefiles in these directories defines the DLLIB macro as ending with the .bundle extension, and linalg.bundle is what gets built.

So, we brute-force it by breaking the process down into “make” and “install”, and in between we create those .so files.

ruby install.rb make cp ext/linalg/linalg.bundle ext/linalg/linalg.so cp ext/lapack/lapack.bundle ext/lapack/lapack.so sudo ruby install.rb install

If you don’t trust this hack, put in ruby install.rb test before the install task to verify that everything works. I’m not sure why the package tries to install an .so file its own makefiles don’t build; if someone can figure that out and patch it, I’m sure the maintainers would love to know.

If you find any obvious errors in this, or see some steps we can stick, feel free to comment and we’ll make edits. Hopefully this will come in handy for someone.

Update, 7 September 2008: Be sure to read through the comments to where James Lawrence, linalg’s maintainer, points out the new (as of yesterday) 1.0.0 release which resolves most of these problems. If you’re struggling with linalg and aren’t using 1.0.0, try that new version.

filed under: Common Running, Ruby on Rails | permalink

12 Responses to “Building LAPACK and Ruby’s linalg on Mac OS X”

Common Media, Inc. » Blog Archive » Where we are and what we’re doing Says:
March 26th, 2008 at 1:55 pm
[...] Common Media, Inc. Online Communities and Web Development « Building LAPACK and Ruby’s linalg on Mac OS X [...]
Tim Case Says:
April 18th, 2008 at 4:10 pm
Hi thanks for writing this up, you seem to have the only information on the net about getting a full Lapack on macosx.

Does this solution work for leopard?

Running the make.inc you listed above yields:

lapack-3.1.1 $ make blaslib lapacklib tmglib > make.log 2>&1
lapack-3.1.1 $ if [ $? != 0 ] ; then
> echo “*** Error in make blaslist lapacklib tmglib ***”
> grep -i err make.log
> fi
*** Error in make blaslist lapacklib tmglib ***
make[1]: *** [testsecond] Error 1
make: *** [lapack_install] Error 127
lapack-3.1.1 $

Tailing the make.log I see

/bin/sh: ./testsecond: No such file or directory
/bin/sh: ./testdsecnd: No such file or directory
/bin/sh: ./testversion: No such file or directory
make: *** [lapack_install] Error 127

Did you ever run into these problems when building Lapack?

Thanks in advance I appreciate this write up, I don’t know who else to ask on this one.
pjmorse Says:
April 20th, 2008 at 7:13 pm
Tim - I haven’t run through this on Leopard, but it’s not inconceivable that there are sufficient differences between that and Tiger to lead to these errors.

That said, the error messages look familiar. Give me a day or so to get where I can go through it better and I’ll get back to you.
Tim Case Says:
April 22nd, 2008 at 3:06 pm
Hey thanks for even commenting on this, I appreciate it.
pjmorse Says:
April 23rd, 2008 at 8:09 am
Tim, which Fortran compiler are you using? The errors you’re getting are missing files in the INSTALL subdirectory of the LAPACK distribution, which should be built early in the makefile. My make.log shows lines like

g77 -o testsecond second_EXT_ETIME.o secondtst.o

…so if g77 (or whatever is masquerading as g77 on your system) isn’t working properly, those won’t get built, and when the makefile tries anything depending on them, it will fail.
David Bean Says:
May 3rd, 2008 at 4:10 am
Ooops…one more hack I had to make:

My ruby install needed those .so dynamic libs to be named as .bundle. (!) Then I had to hack my linalg.rb file to look like this:

require ‘universal-darwin8.0/lapack.bundle’
require ‘universal-darwin8.0/linalg.bundle’

Now I fire up IRB and do ‘require ‘linalg” => true

whew,,,
Darrell Swenson Says:
June 18th, 2008 at 1:14 pm
I had the same problem as TIm. I found the following link helpful.

http://gcc.gnu.org/ml/fortran/2007-02/msg00400.html

In my case the two files that needed editing were second_EXT_ETIME.f and dsecnd_EXT_ETIME.f
Rob Rambo Says:
July 4th, 2008 at 6:46 pm
Thanks for the posting. I had some issues with f2c.h not being found. I am running OS X 10.5.2 and the f2c.h header is in /sw/include. Creating the following link fixed the problem.

sudo ln -s /sw/include/f2c.h /usr/local/include/
Dimitris Says:
August 1st, 2008 at 9:39 am
Tim, Darrell,

I found that this helps (on OSX 10.4.11):

change lines 22 & 23 in dsecnd_EXT_ETIME.f and second_EXT_ETIME.f to
REAL EXT_ETIME
EXTERNAL EXT_ETIME

Found here: http://www.pauldix.net/2008/06/installing-lina.html
James Kyle Says:
August 23rd, 2008 at 10:37 am
To fix the ld errors for etime, edit the make.inc @ approx line 29.

Comment out TIMER = EXT_ETIME

Uncomment TIMER = INT_CPU_TIME

This worked for me on Leopard 10.5.4, macports gcc43’s gfortran-mp, lapack-3.1.1.

Cheers,

James
Joseph Wilk Says:
September 4th, 2008 at 4:37 am
This worked for me on Leopard 10.5.4, Ruby 1.8.7,
gfortran: 4.3.0, lapack-3.1.1.

I had to follow Rob Rambos advice and run
“sudo ln -s /sw/include/f2c.h /usr/local/include/”

I also had to manually copy over the linalg.bundle and lapack.bundle from my linalg build to my ruby site_ruby folder (/local/lib/ruby/site_ruby/1.8/i686-darwin9).

I found it useful to run the linalg extconf.rb files directly for lapack and linalg to find out why things where not working:

cd ext/lapack/; ruby extconf.rb
cd ext/linalg/; ruby extconf.rb
James M. Lawrence Says:
September 7th, 2008 at 9:51 am
These issues should be fixed in the new release, which I have gratuitously called 1.0.0.

After I applied Luis Parravicini’s patch to use g2c, then replaced all ‘.so’ instances with CONFIG["DLEXT"], it worked immediately. I installed g77 from http://hpc.sourceforge.net .

There is a new Mac binary available for the new release http://rubyforge.org/frs/?group_id=273

Thanks to all who provided help during my extended hiatus.

Common Media, Inc.

12 Responses to “Building LAPACK and Ruby’s linalg on Mac OS X”

Leave a Reply