"Install All The Things!": a Capistrano extension to run Puppet

If you use Capistrano to deploy applications onto Unix systems you may find our new tool Capistrano-ext-puppetize useful.

Now instead of having to prepare your deployment host beforehand with all the third party libraries and tools it might need (Ruby interpreter, databases, development libraries, ImageMagick, redis, etc) you can keep a Puppet manifest alongside your codebase in a Git repository, and apply it automatically every time you deploy.  The benefits are twofold:

• It's now a single step to deploy an entire application and all its dependencies onto a very-nearly-empty generic OS image (needs only Puppet and ssh and your-choice-of-version-control-system)
• Your code and your Puppet manifest are in the same repository, meaning that every time you deploy, your OS environment gets updated to match the requirements of that version of your app: if upgrades, downgrades and different branches need different libraries or different third party apps, that's all handled without your having to think about it

To use capistrano-ext-puppetize in your project, add it to the Gemfile or config/deploy.rb if you're using that), and then create a manifest in config/puppet/manifests/site.pp containing the things you need installed. More comprehensive instructions are available in the README on GitHub - if you only want to use it, stop reading this and read that instead. If you want to find out how it was put together, read on.

capistrano-ext-puppetize, The Making Of

CEP grew out of some work we did to migrate our platform to a new "cloud" hosting provider. We wanted to automate building our servers from scratch starting with a generic base image. Having settled on "masterless Puppet" as a general approach here see some reasons that puppetmaster is not a good fit for our requirement, this left us with a couple of questions:

• How do we get the Puppet config onto the boxes?
• How do we ensure it's the right version of the config for the version we're deploying?

The simplest way to answer the second question is to keep the config alongside the app in the same git repository where it can be branched, merged, tagged, pulled, pushed, filed, stamped, indexed, briefed, debriefed and numbered This happily and conveniently solves a good chunk of the first question too, as all the Puppet files will be installed with the app by Capistrano's deploy:update_code recipe. CEP is in essence, an after hook for deploy:update_code that shells into the deployed host and runs puppet apply. Easy, right?

In practice it's a little more complicated than it is in theory (this maxim seems to be generally applicable to Devops, and possibly also to life), though not by much. The first consideration is that we want to use files and templates and modules and stuff, so it's not really puppet apply we're running, it's something more like

puppet apply --modulepath=$PROJECT_DIR/config/puppet/modules:$PROJECT_DIR/config/puppet/vendor/modules --templatedir=$PROJECT_DIR/config/puppet/templates --fileserverconfig=$PROJECT_DIR/config/puppet/fileserver.conf $PROJECT_DIR/config/puppet/manifests/site.pp

Given that we quite likely want to apply the Puppet manifest at times other than deploy, such as when the box is booted or at scheduled intervals to make sure that unmanaged interactive changes are reported and reverted, CEP deals with this by creating a shell script /etc/puppet/apply which invokes Puppet with all the options.

The second requirement is that we really would like our manifests to be able to do different things depending on the role (web, app, db, etc) and environment (staging, production, etc.) of the server they're running on. The traditional Puppet way of doing this is to make it dependent on the hostname, but obviously that's not going to fly when we're using Puppet to provision the host from a base OS install, as it's grabbed an IP address from DHCP and doesn't have a meaningful hostname. But we do have this data in Capistrano, so what if we export our Capistrano variables as Puppet facts?

Consulting the official Puppet documentation on how to create your own facts may leave you under the impression that it requires writing Ruby files, putting them in Puppet modules and indulging in various kinds of gymnastics to get those modules from the puppetmaster onto clients. Happily there is an easier way, as described in the Puppet Cookbook: you can simply define a shell environment variable FACTER_foo to make the $foo fact available to manifests. Coupling this with a bit of Capistrano internals poking and some random "why does it keep asking for a password?" head-scratching, we arrived at the following in our Capistrano recipe

# Export capistrano variables as Puppet facts so that the
# site.pp manifest can make decisions on what to install based
# on its role and environment. We only export string variables
# -- not class instances, procs, and other outlandish values

facts = variables.find_all { |k, v| v.is_a?(String) }.map {|k, v| "FACTER_cap_#{k}=#{v.inspect}" }.join(" ")

The password-prompt puzzlement was precipitated by procs. Capistrano allows variables whose values are procs, which are lazily evaluated on demand. Forcing the evaluation of all variables here, as we originally did, is not a good idea because some of them prompt for passwords and if they're not passwords you needed to know, you've just broken the ability to walk away and get a coffee while you're deploying. Hence the is_a?(String) criterion to filter these things out.

Vagrancy

It is of course, a complete dog to test your manifests this way, as each change requires a git commit and a push to whatever upstream Capistrano is set to pull from. That's why we also have a puppet:install_vagrant recipe. This is basically a hack we use for testing with Vagrant on VirtualBox. It writes /etc/puppet/vagrant-apply which looks just like the real apply script except that all the pathnames point to /vagrant. In a Vagrant VM this is a shared folder that points to your project directory, so you can change your Puppet syntax and see the effect immediately.

Miscellany

CEP has entertaining interactions with rvm-capistrano, which are described in lurid detail in the README. It expects to find the various kinds of Puppet config in Config/puppet/{manifests/site.pp,modules,vendor/modules,templates,files}

• there will almost certainly will be some configuration options for this in a future version. It doesn't do anything with the Puppet "Module Forge" system, because nobody here has got their head around that - instead, we use git submodules to arrange that third-party Puppet bits appear under config/puppet/vendor/modules. Git submodules are of course the second worst system ever invented for doing this kind of thing, but all the others that we've seen are still tied for first place. It works for us.

As always, YMMV.

Feel free to comment on how you find using it!