Freezing Gems
From DreamHost
Contents |
What is Freezing a Gem?
Freezing a gem is when you install a copy of a gem into your application's vendor/gems directory. This directory is checked and any gems found there take precedence over gems installed on your server.
Why Freeze Your Gems?
Freezing gems to Rails applications has become fairly standard practice. In shared hosting environments like DreamHost's, it's also fairly essential! This is the best way to ensure the gems you use in your application are available for it to use. Exactly which gems are installed in the system can vary from one server to another and the version installed can change over time. If you rely on the system-installed gems for your application, you're leaving yourself vulnerable to changes made on the server without your knowledge. If your user account is moved or the gems on the server are updated your application could suddenly start throwing exceptions because a gem you need is no longer installed or a newer version is being used that's incompatible with your application. Freezing gems to your application insulates you from such changes. Not only can you use gems that aren't installed on your server by freezing them, you guarantee that gem changes on the server will not affect your application.
How to Freeze Your Gems
Starting with Rails 2.1, gems have had increased support built in. You can specify your application's gem dependencies in your config/environment.rb file. The syntax is as follows:
In that example, you see an entry for the hpricot gem. The name given there should be the one you found in "gem list --local" for the gem you're using and the version is exact version number you want to use. As you can see in the comments above (which are in the default environment.rb file), there are additional options you can specify. A good example of when you need to use them is when you're using gems from GitHub. Below you see what the entry would look like for the dreamy gem.
As you can see, there are two additional options. The :lib option lets you tell Rails what the actual library name is. GitHub automatically prepends the username of the gem's creator before the actual gem name, which make it impossible for Rails to determine the correct library name from the name of the gem. In this case, the actual name of the library is "dreamy", so that's specified. The :source option lets you specify a non-standard gem source to obtain the gem in question from if it isn't already installed. So, in the example we're using now, where your application is just using the dreamy gem, the finished gem dependency portion of your environment.rb file might look like this:
Once you've got your environment.rb all set with the gems you're using in your application, you want to run the "rake gems:unpack" task. This will cycle through the dependencies you listed and automatically unpack them to the vendor/gems directory of your application. Running this command with the above setup should yield output similar to the below:
$ rake gems:unpack (in /path/to/your/app) Unpacked gem: '/path/to/your/app/vendor/gems/uuid-2.0.1' Unpacked gem: '/path/to/your/app/vendor/gems/sant0sk1-dreamy-0.3.0'
If you're doing this from a Mac OS X machine, you might run into the following message: "WARNING: Installing to ~/.gem since /Library/Ruby/Gems/1.8 and /usr/bin aren't both writable." You can safely ignore that message -- the gems were still unpacked to the correct location.
Building Native Extensions
The Easy Way
In some cases, gems include a binary and need to be compiled so they'll run on the machine your application is being deployed on. Many gems don't require this, but some relatively common gems do such as RedCloth and hpricot along with other less common gems. If you freeze gems that require this, you'll need to build them on your DreamHost server after your deploy your application. The command you need to run to build gems that need it is "rake gems:build". If you're using Capistrano to deploy your application, you can automate the gem building process after deployments by adding the below to your deploy.rb file:
task :after_update_code, :roles => :app do
if ENV['build_gems'] and ENV['build_gems'] == '1'
run "rake -f #{release_path}/Rakefile gems:build"
end
end
That task will make it so if you specify "build_gems=1" when you deploy, gems will be built -- otherwise they won't. So, your deploy might look like this:
cap deploy build_gems=1
The Efficient Way
If you have a lot of gems that require building, doing this every time you deploy might not be the ideal method for you. Fortunately, you can actually include pre-compiled gems with your application for multiple platforms with a little bit of work. Generally when building gems is necessary you're working with at least two different hardware platforms. Ruby stores this information in the RUBY_PLATFORM constant. For instance, if you're developing on one of Apple's newer laptops you can see what Ruby sees your machine as by running this:
$ ruby -e "puts RUBY_PLATFORM" universal-darwin9.0
If you run this command on some of Dreamhost's older machines, you'll get this:
$ ruby -e "puts RUBY_PLATFORM" i386-linux
And if you run it on Dreamhost's newer machines, you'll get this:
$ ruby -e "puts RUBY_PLATFORM" x86_64-linux
The output can vary from one generation of machine to another, so make sure you actually run this on all machines you need to so you're sure it's right. The trick is to deploy your application to your server with all gems frozen, but unbuilt. Once you've done that, you want to run "rake gems:build". Before proceeding, it's helpful to have a little bit of an idea of what's happening here. Assuming you have the hpricot gem unpacked, the build process for it would look something like this:
cd vendor/gems/hpricot-0.8.1/ext/hpricot_scan ruby extconf.rb make
That's done for you automatically when you use the "rake gems:build" task. Once the compilation is done, you'll have a pre-compiled library file for the gem. It's file extension will differ depending on the platform you're on. On a Mac OS X machine, you'll end up with a file that has a .bundle file with same name as the directory in the ext directory mentioned above. Hpricot is a little bit different in that it generates two separate compiled libraries -- fast_xs.bundle and hpricot_scan.bundle. These files are automatically moved from the ext/ directory to the lib/ directory when you run "rake gems:build".
The main idea here is to create subdirectories inside vendor/gems/hpricot-0.8.1/lib/ for each platform your application will be running on that will each contain the pre-compiled libraries. So, you might end up with a directory listing something like this:
vendor/gems/hpricot-0.8.1/lib/universal-darwin9.0/fast_xs.bundle vendor/gems/hpricot-0.8.1/lib/universal-darwin9.0/hpricot_scan.bundle vendor/gems/hpricot-0.8.1/lib/i386-linux/fast_xs.so vendor/gems/hpricot-0.8.1/lib/i386-linux/hpricot_scan.so
To get this to work will require editing some of your libraries files as they won't be looking for the compiled files inside a directory like that. To find which files you need to update, you'll want to run the following command from inside the lib directory for the gem adjusting it for the name of the compiled files name. In this case we run this:
$ grep -r -n -I -E "require.*?fast_xs" . ./hpricot/builder.rb:2:require 'fast_xs' $ grep -r -n -I -E "require.*?hpricot_scan" . ./hpricot.rb:20:require 'hpricot_scan'
As you can see from the above output, you've got two files you need to edit -- lib/hpricot/builder.rb and lib/hpricot.rb. You want to edit those require lines so that they look like this:
require "#{RUBY_PLATFORM}/fast_xs"
require "#{RUBY_PLATFORM}/hpricot_scan"
How much editing is required will vary from gem to gem. A brief explanation of what that grep command actually did is probably in order. The -r parameter tells grep to search all subdirectories and files recursively. The -n parameter tells grep to include the line number the hit was found on. The -I command tells it to ignore binary files (so it doesn't actually return the compiled library as a hit). The -E command tells grep to allow extended regular expressions so that the search pattern that's needed will work. The "require.*?hpricot_scan" is the regex being searched for -- basically any line including "require" followed by any characters until it hits the string "hpricot_scan". The "." at the end simply tells it to start searching in the current directory (which is the lib directory in this case).
You'll need to repeat this process for any other gems you have that require compiled native extensions. Generally, editing gem source files is considered bad form because it's non-standard and makes upgrading more difficult. For this reason, if you do this, you should document what you've done well for your application and make notes of the changes you've made somewhere easily accessible, both so others who may look at the code can find it and so you can remember what you did when you need to upgrade the gem at a later date. In many ways, just incorporating the "rake gems:build" task into your deploy process is easier, but this method may be more what you're looking for.
Freezing Rails Gems
Rails can be frozen to your application just like a gem! Doing this is actually a best practice and is highly recommended -- especially in a shared hosting environment. For DreamHost in particular, it's great because as of the time of this writing their servers haven't been upgraded to Rails 2.3.2. By freezing Rails to your application, you're essentially saying "use this copy of Rails rather than the one installed on the system". Rails is treated slightly different from other gems though and even has it's own command. It also unpacks to vendor/rails rather than vendor/gems. To do this you run "rake rails:freeze:gems". You should see output similar to this after running that:
$ rake rails:freeze:gems (in /path/to/your/app) Freezing to the gems for Rails 2.3.2 rm -rf vendor/rails mkdir -p vendor/rails cd vendor/rails Unpacked gem: '/path/to/your/app/vendor/rails/activesupport-2.3.2' mv activesupport-2.3.2 activesupport Unpacked gem: '/path/to/your/app/vendor/rails/activerecord-2.3.2' mv activerecord-2.3.2 activerecord Unpacked gem: '/path/to/your/app/vendor/rails/actionpack-2.3.2' mv actionpack-2.3.2 actionpack Unpacked gem: '/path/to/your/app/vendor/rails/actionmailer-2.3.2' mv actionmailer-2.3.2 actionmailer Unpacked gem: '/path/to/your/app/vendor/rails/activeresource-2.3.2' mv activeresource-2.3.2 activeresource Unpacked gem: '/path/to/your/app/vendor/rails/rails-2.3.2' cd -
If you're doing this from a Mac OS X machine, you might run into the following message: "WARNING: Installing to ~/.gem since /Library/Ruby/Gems/1.8 and /usr/bin aren't both writable." You can safely ignore that message -- the gems were still unpacked to the correct location.
External Links
- Vendor Everything - a nice article (though dated) explaining the concept of freezing gems
- Gem Dependencies Railscast - a screencast explaining Rails gem dependency support
- Localizing Gems with Native Extensions - an explanation for handling multiple hardware architectures with frozen gems that have native extensions
