Everybody wants to be special (I blame our moms). The price of special, when it comes to IT, is time. Consider how long you’ve spent on just your damn PROMPT and you’ll realize why automation gives any good sysadmin a case of the giggles. Like the cobbler’s kids, though, your laptop and development environment are always the last to be visited by the automation gnomes.

Until now.

Will Farrington gave a great talk at Puppetconf 2012 about managing an army of developer laptops using Puppet + some Github love that left more than a couple of people asking for his code. That request has unfortunately been denied.

Until now.

Boxen, and hand-tune no more

Enter Boxen (née ‘The Setup’), a full-fledged open source project from the guys at Github that melds Puppet with your Github credentials to create a framework for automating everything from applications, to dotfiles, and even printers and emacs extensions (that last bit’s a lie - no one should be using emacs).

How does it work? Think ‘Masterless Puppet’ (or, just a bunch of Puppet modules that are enforced by running puppet apply on your local machine). Boxen not only includes the framework ITSELF, but is a project on Github that hosts over 75 individual modules for managing things like rbenv, homebrew, git, mysql, postgres, riak, redis, npm, erlang, dropbox, skype, minecraft, heroku, 1password, iterm2, and much more. Odds are there’s a module for many of the common things you setup on your laptop. And what about things like dotfiles that are intrinsically personal? You can create your own repository and manage them like you would any other file/directory/symlink on the file system. The goal is to model every little piece of your laptop that makes it ‘unique’ from everyone else until you have your entire environment managed and the hardware becomes…well…disposable. How many times have you shied away from doing an upgrade because some component of your laptop required you to spend countless hours tinkering with it? If you’ve done the time, you should do something to make sure that you NEVER have to repeat that process manually ever again.

Boxen is also very self-contained. Packages and binaries that come out of Homebrew are installed into /opt/boxen/homebrew/bin, frameworks like Ruby and Node are installed into /opt/boxen/{rbenv,nvm}, and individual versions of those frameworks are kept separate from your system version of those frameworks. These details are important when you consider that you could purge the whole setup without having to rip out components scattered around your system.

You may be reading this and thinking “There’s no way in hell I can use this to manage every laptop in my organization!”, and you’re right. The POINT of Boxen is that it’s a tool written by developers for developers to automate THEIR systems. The goal of developing is to have as little friction between the process of writing code and deploying that code into production. A tool like Boxen allows you to more quickly GET to the state where your laptop is ready for you to start developing. If you want a tool to completely manage and lock down all the laptops on your system, look to using Puppet in agent/master mode or to a tool like Munki to manage all packages on your system. If you’re interested in giving your developers/users the freedom to manage their OWN ‘boxes’ because they know best what works for them, then Boxen is your tool.

There IS one catch - it’s targeted to OS X (10.8 to be exact).

Diary of an elitist

I was fortunate to have early-access to Boxen in order to kick its Ruby tyres. As someone who’s managed Macs with Puppet before (all the way down to the desktop/laptop level), I was embarrassed to admit that I had NOTHING about my laptop automated. Will unlocked the project and basically said “Have fun, break shit, fix it, and file pull requests” and away I went. To commit completely to the project, I did what any sane person would do.

I reformatted my laptop and started entirely from scratch.

(Let’s be clear here - you don’t have to do that. Initially Will reported problems getting Boxen running in VMs, but I never ran into an issue. I ran Boxen in VMware Fusion 5 a number of times to make sure the changes I made were going to do the right thing on a fresh install. I’d recommend going down THAT road if you’re hesitant of immediately throwing this on your pretty snowflake of a laptop.)

Installing Boxen was pretty easy - the only prerequisite was downloading the XCode Command-Line Tools (which included git), pulling down the Boxen repo, and running script/boxen. It was stupid simple. What you GOT, by default, was:

• Homebrew
• Git
• Hub
• DNSMasq w/ .dev resolver for localhost
• NVM
• RBenv
• Full Disk Encryption requirement
• NodeJS 0.4
• NodeJS 0.6
• NodeJS 0.8
• Ruby 1.8.7
• Ruby 1.9.2
• Ruby 1.9.3
• Ack
• Findutils
• GNU-Tar

Remember, this is all tunable and you don’t need to pull down ALL of these packages, but, since it was new, I decided to install everything and sort it out later. Yes, the initial setup took a good number of minutes, but think about everything that’s being installed. In the end, I had a full Ruby development environment with rbenv, multiple versions of Ruby, and a laptop that could be customized without much work at all.

Which end do I blow in?

The readme on the project page for Boxen describes how to clone the project into /opt/boxen/repo, so that’s the directory we’ll be working with. To see what will be enforced on your machine, check out manifests/site.pp to see something that looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

require boxen::environment
require homebrew::repo

Exec {
  group       => 'staff',
  logoutput   => on_failure,
  user        => $luser,

  path => [
    "${boxen::config::home}/rbenv/shims",
    "${boxen::config::home}/homebrew/bin",
    '/usr/bin',
    '/bin',
    '/usr/sbin',
    '/sbin'
  ],

  environment => [
    "HOMEBREW_CACHE=${homebrew::cachedir}",
    "HOME=/Users/${::luser}"
  ]
}

File {
  group => 'staff',
  owner => $luser
}

Package {
  provider => homebrew,
  require  => Class['homebrew']
}

Repository {
  provider => git,
  extra    => [
    '--recurse-submodules'
  ],
  require  => Class['git']
}

Service {
  provider => ghlaunchd
}

This is largely scaffolding setting up the Boxen environment and resource defaults. If you’re familiar with Puppet, this should be recognizable to you, but for everyone else, let’s dissect one of the resource defaults:

1
2
3
4

File {
  group => 'staff',
  owner => $luser
}

This block basically means that any file you declare with Puppet should default to having its owner set as your username and its group set to ‘staff’ (which is standard in OS X). You can override this explicitly with a file declaration by providing the owner or group attribute, but if you omit it then it’s going to default to these values.

The rest of the defaults are customized for Boxen’s preferences (i.e. homebrew will be used to install all packages unless you specify otherwise, exec resources will log all output on failure, service resources will use githubs’s customized service provider, and etc…). Now let’s look below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

node default {
  # core modules, needed for most things
  include dnsmasq
  include git
  include hub
  include nginx
  include nvm
  include ruby

  # fail if FDE is not enabled
  if $::root_encrypted == false {
    fail('Please enable full disk encryption and try again')
  }

  # node versions
  include nodejs::0-4
  include nodejs::0-6
  include nodejs::0-8

  # default ruby versions
  include ruby::1-8-7
  include ruby::1-9-2
  include ruby::1-9-3

  # common, useful packages
  package {
    [
      'ack',
      'findutils',
      'gnu-tar'
    ]:
  }

  file { "${boxen::config::srcdir}/our-boxen":
    ensure => link,
    target => $boxen::config::repodir
  }
}

These are the things that Boxen has chosen to enforce ‘out of the box’. Knowing that Boxen was designed so that developers could customize their ‘boxes’ THEMSELVES, it makes sense that there’s not much that’s being enforced on everyone. In fact, the most significant thing being ‘thrust’ upon you is the fact that the machine must have full disk encryption enabled (which is a good idea anyways).

If you want to pare down what Boxen gives you by default, you can choose to comment out lines providing, for example, nvm and nodejs versions (if you don’t use node.js in your environment). I’m a Ruby developer, so all the Ruby builds (and rbenv) are very helpful to me, but you could also remove those if you were so inclined. The point is that this file contains the ‘knobs’ to dial your base Boxen setup up or down.

Customizing (or, my dotfiles are better than yours)

The whole point of Boxen is to customize your laptop and keep its customization automated. To do this, we’re going to need to make some Puppet class files.

CAUTION: PUPPET AHEAD

If you’ve not had experience with Puppet before, I can’t recommend the learning Puppet series enough. In the vein of “Puppet now, learn later”, I’m going to give you Puppet code that works for ME and only explain the trickier bits.

Boxen has some ‘magic’ code that’s going to automatically look for a class called people::<github username>, and so I’m going to create a file in modules/people/manifests called glarizza.pp. This file will contain Puppet code specific to MY laptop(s). Here’s a snippit of that file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

class people::glarizza {

  notify { 'class people::glarizza declared': }

  # Changes the default shell to the zsh version we get from Homebrew
  # Uses the osx_chsh type out of boxen/puppet-osx
  osx_chsh { $::luser:
    shell   => '/opt/boxen/homebrew/bin/zsh',
    require => Package['zsh'],
  }

  file_line { 'add zsh to /etc/shells':
    path    => '/etc/shells',
    line    => "${boxen::config::homebrewdir}/bin/zsh",
    require => Package['zsh'],
  }

  ##################################
  ## Facter, Puppet, and Envpuppet##
  ##################################

  repository { "${::boxen_srcdir}/puppet":
    source => 'puppetlabs/puppet',
  }

  repository { "${::boxen_srcdir}/facter":
    source => 'puppetlabs/facter',
  }

  file { '/bin/envpuppet':
    ensure  => link,
    mode    => '0755',
    target  => "${::boxen_srcdir}/puppet/ext/envpuppet",
    require => Repository["${::boxen_srcdir}/puppet"],
  }
}

The notify resource is only to prove that when you run Boxen that this class is being declared - it only displays a message to the console when you run the boxen binary.

The osx_chsh resource is a custom defined type that Github has created to ensure a line shows up in /etc/shells as an acceptable shell. Because Boxen installs zsh from homebrew into /opt/boxen/homebrew, we need to ensure that /etc/shells is correct. Note the syntax of $boxen::config::homebrewdir which refers to a variable called $homebrewdir in the boxen::config class.

Next, I’ve setup a couple of resources to make sure the puppet and facter repositories are installed on my system. Github has also developed a lightweight repository resource that will simply ensure that a repo is cloned at a location on disk. $::boxen_srcdir is one of the custom Facter facts that Boxen provides in shared/boxen/lib/facter/boxen.rb in the Boxen repository.

The file resource sets up a symlink from /bin/envpuppet to /Users/glarizza/src/puppet/ext/envpuppet on my system. The attributes should be pretty self-explanatory, but the newest attribute of require says that the repository resource must come BEFORE this file resource is declared. This is a demonstration of Puppet’s ordering metaparameters that are described in the Learning Puppet series.

Since we briefly touched on $::boxen_srcdir, what are some other custom facts that come out of shared/boxen/lib/facter/boxen.rb?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

require "json"
require "boxen/config"

config   = Boxen::Config.load
facts    = {}
factsdir = File.join config.homedir, "config", "facts"

facts["github_login"]   = config.login
facts["github_email"]   = config.email
facts["github_name"]    = config.name

facts["boxen_home"]     = config.homedir
facts["boxen_srcdir"]   = config.srcdir

if config.respond_to? :reponame
  facts["boxen_reponame"] = config.reponame
end

facts["luser"]          = config.user

Dir["#{config.homedir}/config/facts/*.json"].each do |file|
  facts.merge! JSON.parse File.read file
end

facts.each { |k, v| Facter.add(k) { setcode { v } } }

This file will also give you $::luser, which will evaluate out to your system username, and $::github_name, which is equivalent to your Github username (note that this is what Boxen uses to find your class file in modules/people/manifests). If you’re looking for all the other values set by these custom facts, check out config/boxen/defaults.json after you run Boxen.

Using modules out of the Boxen namespace

Not only is Boxen its own project, but it’s a separate organization on Github that hosts a number of Puppet modules. Some of these modules are pretty simple (a single resource to install a package), but the point is that they’ve been provided FOR you - so use, fork, and improve them (but most of all, submit pull requests). The way you use them with Boxen may not be readily clear, so let’s walk through that with a simple module for installing Google Chrome.

1. Add the module to your Puppetfile
2. Classify the module in your Puppet setup
3. Run Boxen

Add the module to your Puppetfile

Boxen uses a tool called librarian-puppet to source and install Puppet modules from Github. Librarian-puppet uses the Puppetfile file in the root of the Boxen repo to install modules. Let’s look at a couple of lines in that file:

1
2
3
4
5
6
7
8
9
10
11
12

mod "boxen",    "0.1.8",  :github_tarball => "boxen/puppet-boxen"
mod "dnsmasq",  "0.0.1",  :github_tarball => "boxen/puppet-dnsmasq"
mod "git",      "0.0.3",  :github_tarball => "boxen/puppet-git"
mod "hub",      "0.0.1",  :github_tarball => "boxen/puppet-hub"
mod "homebrew", "0.0.17", :github_tarball => "boxen/puppet-homebrew"
mod "inifile",  "0.0.1",  :github_tarball => "boxen/puppet-inifile"
mod "nginx",    "0.0.2",  :github_tarball => "boxen/puppet-nginx"
mod "nodejs",   "0.0.2",  :github_tarball => "boxen/puppet-nodejs"
mod "nvm",      "0.0.5",  :github_tarball => "boxen/puppet-nvm"
mod "ruby",     "0.4.0",  :github_tarball => "boxen/puppet-ruby"
mod "stdlib",   "3.0.0",  :github_tarball => "puppetlabs/puppetlabs-stdlib"
mod "sudo",     "0.0.1",  :github_tarball => "boxen/puppet-sudo"

This evaluates out to the following syntax:

1

mod, <module name>, <version or tag>, <source>

The HARDEST thing about this file is finding the version number of modules on Github (HINT: it’s a tag). Once you’re given that information, it’s easy to pull up a module on Github, look at its tags, and then fill out the file. Let’s do that with a line for the Chrome module:

1

mod "chrome",     "0.0.2",   :github_tarball => "boxen/puppet-chrome"

Classify the module in your Puppet setup

In the previous section, we created modules/people/manifests/<github username>.pp. We COULD continue to fill this file with a ton of resources, but I tend to like to separate out resources into separate subclasses. Puppet has module naming conventions to ensure that it can FIND your subclasses, so I recommend browsing that guide before randomly naming files (HINT: Filenames ARE important and DO matter here). I want to create a people::glarizza::applications subclass, so I need to do the following:

1
2
3
4

## YES, make sure to replace YOUR USERNAME for 'glarizza'
$ cd /opt/boxen/repo
$ mkdir -p modules/people/manifests/glarizza
$ vim modules/people/manifests/glarizza/applications.pp

It’s totally fine that there’s a glarizza directory aside the glarizza.pp file - this is intentional and desired. Puppet’s not going to automatically declare anything in the people::glarizza::applications class until we TELL it to, so let’s open modules/people/manifests/glarizza.pp and add the following line at the top:

1

include people::glarizza::applications

That tells Puppet to find the people::glarizza::applications class and make sure it ‘does’ everything in that file. Now, let’s create the people::glarizza::applications class:

1
2
3

class people::glarizza::applications {
  include chrome
}

Yep, all it takes is one line to include the module we will get from Boxen. Because of the way Boxen works, it will consult the Puppetfile FIRST, pull down any modules that are in the Puppetfile but NOT on the system, drop them into place so Puppet can find them, and then run Puppet normally.

Run Boxen

Once you have Boxen setup, you can just run boxen from the command line to have it enforce your configuration. By default, if there are any errors, it will log them as Github Issues on your fork of the main Boxen repository (this can be disabled with boxen --no-issue). As you’re just getting started, don’t worry about the errors. The good news is that once you fix things and perform a successful Boxen run, it will automatically close all open issues. If everything went well, you should now have Google Chrome in your /Applications directory!

¡Más Puppet!

You’ll find as you start customizing all the things that you’re usually managing one of the following resources:

1. Packages
2. Files
3. Repositories
4. Plist files

We’ve covered managing a repository and a file, but let’s look at a couple of the other popular resources:

Packages are annoying

I would be willing to bet that most of the things you end up managing will be packages. Using Puppet with Boxen, you have the ability to install four different kinds of packages:

1. Applications inside a DMG
2. Installer packages inside a DMG
3. Homebrew Packages
4. Applications inside a .zip file

Here’s an example of every type of package installer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

  # Application in a DMG
  package { 'Gephi':
    ensure   => installed,
    source   => 'https://launchpadlibrarian.net/98903476/gephi-0.8.1-beta.dmg',
    provider => appdmg,
  }

  # Installer in a DMG
  package { 'Virtualbox':
    ensure => installed,
    source => 'http://download.virtualbox.org/virtualbox/4.1.22/VirtualBox-4.1.23-80870-OSX.dmg',
    provider => pkgdmg,
  }

  # Homebrew Package
  package { 'tmux':
    ensure => installed,
  }

  # Application in a .zip
  package { 'Github for Mac':
    ensure   => installed,
    source   => 'https://github-central.s3.amazonaws.com/mac%2FGitHub%20for%20Mac%2069.zip',
    provider => compressed_app
  }

Notice that the only thing that changes among these resources is the provider attribute. Remember from before that Boxen sets the default package provider to be ‘homebrew’, so for ‘tmux’ I omitted the provider attribute to utilize the default. Also, the ensure attribute is defaulted to ‘installed’, so technically I could remove it…but I tend to prefer to use it for people who will be reading my code later.

There’s no provider for .pkg files. Why? Well, packages on OS X are either bundles or flat-packages. Bundles LOOK like individual files, but they’re actually folders that contain everything necessary to expand and install the package. Flat packages are just that - an actual file that ends in .pkg that can be expanded to install whatever you want. Bundle packages are pretty common, but they’re also hard for curl to download them (being that it’s just a folder full of files) - this is why most installer packages you encounter on OS X are going to be enclosed in a .dmg Disk Image.

So which provider will you use? Well, if your file ends in .dmg then you’re going to be using either the pkgdmg or appdmg provider. How do you know which to use? Expand the .dmg file and look inside it. If it contains an application ending in .app that simply needs dragged into the /Applications folder on disk, then chose the appdmg provider (that’s essentially all it does - expand the .dmg file and ditto the .app file into /Applications). If the disk image contains a .pkg package installer, then you’ll chose the pkgdmg provider (which expands the .dmg file and uses installer to install the contents of the .pkg file silently in the background). If your file is a .zip file containing an Application (.app file), then you can use Github’s custom compressed_app provider that will unzip the file and ditto the app into /Applications. Finally, if you want to install a package from Homebrew, then the homebrew provider is pretty self-explanatory here.

(NOTE: There is ONE more package provider I haven’t covered here - the macports provider. It requires Macports to be installed on your system, and will use it to install a package. Macports vs. Homebrew arguments notwithstanding, if you’re into Macports then there’s a provider for you.)

Plists: because why NOT XML :\

Apple falls somewhere between “the registry” and “config files” on the timeline of tweaking system settings. Most settings are locked up in plist files that can be managed by hand or with plistbuddy or defaults. A couple of people have saved their customizations in with their dotfiles (Zach Holman has an example here), but Puppet is a great way for managing individual keys in your plist files. I’ve written a module that will manage any number of keys in a plist file. You can modify your Puppetfile to make sure Boxen picks up my module by adding the following line:

1

mod "property_list_key",  "0.1.0",   :github_tarball => "glarizza/puppet-property_list_key"

Next, you’ll need to add resources to your classes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

  # Disable Gatekeeper so you can install any package you want
  property_list_key { 'Disable Gatekeeper':
    ensure => present,
    path   => '/var/db/SystemPolicy-prefs.plist',
    key    => 'enabled',
    value  => 'no',
  }

  $my_homedir = "/Users/${::luser}"

  # NOTE: Dock prefs only take effect when you restart the dock
  property_list_key { 'Hide the dock':
    ensure     => present,
    path       => "${my_homedir}/Library/Preferences/com.apple.dock.plist",
    key        => 'autohide',
    value      => true,
    value_type => 'boolean',
    notify     => Exec['Restart the Dock'],
  }

  property_list_key { 'Align the Dock Left':
    ensure     => present,
    path       => "${my_homedir}/Library/Preferences/com.apple.dock.plist",
    key        => 'orientation',
    value      => 'left',
    notify     => Exec['Restart the Dock'],
  }

  property_list_key { 'Lower Right Hotcorner - Screen Saver':
    ensure     => present,
    path       => "${my_homedir}/Library/Preferences/com.apple.dock.plist",
    key        => 'wvous-br-corner',
    value      => 10,
    value_type => 'integer',
    notify     => Exec['Restart the Dock'],
  }

  property_list_key { 'Lower Right Hotcorner - Screen Saver - modifier':
    ensure     => present,
    path       => "${my_homedir}/Library/Preferences/com.apple.dock.plist",
    key        => 'wvous-br-modifier',
    value      => 0,
    value_type => 'integer',
    notify     => Exec['Restart the Dock'],
  }

  exec { 'Restart the Dock':
    command     => '/usr/bin/killall -HUP Dock',
    refreshonly => true,
  }

  file { 'Dock Plist':
    ensure  => file,
    require => [
                 Property_list_key['Lower Right Hotcorner - Screen Saver - modifier'],
                 Property_list_key['Hide the dock'],
                 Property_list_key['Align the Dock Left'],
                 Property_list_key['Lower Right Hotcorner - Screen Saver'],
                 Property_list_key['Lower Right Hotcorner - Screen Saver - modifier'],
               ],
    path    => "${my_homedir}/Library/Preferences/com.apple.dock.plist",
    mode    => '0600',
    notify     => Exec['Restart the Dock'],
  }

The important attributes are:

1. path: The path to the plist file on disk
2. key: The individual KEY in the plist file you want to manage
3. value: The value that the key should have in the plist file
4. value_type: The datatype the value should be (defaults to string, but could also be array, hash, boolean, or integer)

You MUST pass a path, key, and value or Puppet will throw an error.

The first resource above sets Gatekeeper in 10.8 and allows you to install packages from the web that HAVEN’T been signed (in 10.8, Apple won’t allow you to install unsigned packages or anything outside of the App Store without enabling this setting).

All of the other resources relate to making changes to the Dock. Because of the way the Dock is managed, you must HUP its process when making changes to your dock plist before they take effect. Also, the dock plist has to be owned by you or else the changes won’t take effect. Every dock plist resource has a notify metaparameter which means “any time this resource changes, run this exec resource”. That exec resource is a simple command that HUPs the dock process. It will ONLY be run if a resource notifies it - so if no changes are made in a Puppet run then the command won’t fire. Finally, the file resource to manage the dock plist ensures that permissions are set (and notifies the exec in case it needs to CHANGE permissions).

Again, this is purely dealing with Puppet - but plists are a major part of OS X and you’ll be dealing with them regularly!

But seriously, dotfiles

I know I’ve joked about it a couple of times, but getting your dotfiles into their correct location is a quick win. The secret is to lock them all up in a repository, and then symlink them where you need them. Let’s look at that:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

  # My dotfile repository
  repository { "${my_sourcedir}/dotfiles":
    source => 'glarizza/dotfiles',
  }

  file { "${my_homedir}/.tmux.conf":
    ensure  => link,
    mode    => '0644',
    target  => "${my_sourcedir}/dotfiles/tmux.conf",
    require => Repository["${my_sourcedir}/dotfiles"],
  }

  file { "/Users/${my_username}/.zshrc":
    ensure  => link,
    mode    => '0644',
    target  => "${my_sourcedir}/dotfiles/zshrc",
    require => Repository["${my_sourcedir}/dotfiles"],
  }

  file { "/Users/${my_username}/.vimrc":
    ensure => link,
    mode   => '0644',
    target => "${my_sourcedir}/dotfiles/vimrc",
    require => Repository["${my_sourcedir}/dotfiles"],
  }

  # Yes, oh-my-zsh. Judge me.
  file { "/Users/${my_username}/.oh-my-zsh":
    ensure  => link,
    target  => "${my_sourcedir}/oh-my-zsh",
    require => Repository["${my_sourcedir}/oh-my-zsh"],
  }

It’s worth mentioning that Puppet does not do things procedurally. Just because the dotfiles repository is listed before every symlink DOES NOT mean that Puppet will evaluate and declare it first. You’ll need to specify order here, and that’s what the require metaparameter does.

Based on what I’ve already shown you, this code block should be very simple to follow. Because I’m using symlinks, the dotfiles should always be current. Because the dotfiles are under revision control, updating them all is as simple as making commits and updating your repository. If you’ve ever had to migrate these files to a new VM/machine, then you know how full of win this block of code is.

Don’t sweat petty (or pet sweaty)

When I show sysadmins/developers automation like this, they usually want to apply it to the HARDEST part of their day-job IMMEDIATELY. That’s a somewhat rational reaction, but it’s not going to give you the results you want. The cool thing ABOUT Boxen and Puppet is that it’s going to remove those little annoyances in your day that slowly sap your time. START by tackling those small annoyances to remove them and build your confidence (like the dotfiles example above). Yeah, you’ll only save a couple of minutes a day, but it grows exponentially. Also, when you solve a problem during the course of your day, MANAGE it with Boxen by putting it in your Puppet class (then, test it out on a VM or another machine to make sure it does what you expect).

Don’t worry that you’re not saving the world with a massive Puppet class - sometimes the secret to happiness is opening iTerm on a new machine and seeing your finely-crafted prompt shining in its cornflower-blue glory.

Now show me some cool stuff

So that’s a quick tour of the basics of Boxen and the kinds of things you can do from the start. I’m really excited for everyone to get their hands on Boxen and do more cool stuff with Puppet. I’ve done a bunch of work with Puppet for OS X, and that’s enough to know that there’s still PLENTY that can be improved in the codebase. A giant THANK YOU to John Barnette, Will Farrington, and the entire Github Boxen team for all their work on this tool (and letting me tinker with it before it hit the general public)! Feel free to comment below, email me (gary at puppetlabs), or yell at me on Twitter for more information!

I miss testing code in production. In smaller organizations, ‘testing’ and ‘development’ can sometimes consist of making changes directly on a server, walking to an active machine, and hoping things work. Once you were done, you MIGHT document what changes you made, but more often than not you kept that information in your head and referred to it later.

I lied - that is everything that sucks about manual configuration of machines.

The best way to get out of this rut is to get addicted to automating first the menial tasks on your machines, and then work your way up from there. We STILL have the problem, though, of doing this in production - that’s what this post is meant to address.

What we want is the ability to spin up a couple of test nodes for the purpose of testing our automation workflow BEFORE it gets committed and applied to our production nodes. This post details using Vagrant and Puppet to both establish a clean test environment and also test automation changes BEFORE applying them to your production environment.

Puppet is a Configuration Management tool that automates all the annoying aspects of manual configuration out of your infrastructure. The bulk of its usage is beyond the scope of THIS post, however we’re going to be using it as the means to describe the changes we want to make on our systems.

Vagrant is a magical project that uses minimal VM templates (boxes) to spin up clean virtualized environments on your workstation for the purpose of testing changes. Currently, it only supports a Virtualbox backend, but its creator, Mitchell Hashimoto, has teased a preview of upcoming VMware integration that SHOULD be coming any day now. In this post, Vagrant will be the means by which we spin up new VMs for development purposes

Getting setup

The only moving piece you need installed on your system is Vagrant. Fortunately, Mitchell provides native package installers on his website for downloading Vagrant. If you’ve never used Vagrant before, and you AREN’T a Ruby developer who maintains multiple Ruby versions on your system, then you’ll want to opt for the native package installer since it’s the easiest method to get Vagrant installed (and, on Macs, Vagrant embeds its own Ruby AND Rubygems binaries in the Package bundle…which is kind of cool).

IF, however, you are developing in Ruby and you use RVM or Rbenv to maintain multiple copies of Ruby on your system, then you’ll want to favor installing Vagrant via Rubygems a la:

1

$ gem install vagrant --no-ri --no-rdoc

If you have no idea how to use RVM or Rbenv - stick with the native installers :)

Puppet does NOT need to be on your workstation since we’re only going to be using it on the VMs that Vagrant spins up - so don’t worry about Puppet yet.

My kingdom for a box

Vagrant uses box files as templates from which to spin up a new virtual machine for development purposes. There are sites that host boxes available for download, OR, you could use an awesome project called Veewee to build your own. Again, building your box file is outside the scope of this article, so just make sure you download a box with an OS that’s to your liking. This box DOES NOT need to have Puppet preinstalled - in fact, it’s probably better that it doesn’t (because the version will probably be old, and we’re going to work around this anyways). I’m going to choose a CentOS 6.3 box that the SE team at Puppet Labs uses for demos, but, again, it’s up to you.

Vagrantfile, assemble!

Now that we’ve got the pieces we need, let’s start stitching together a repeatable workflow. To do that, we’ll need to create a directory for this project and a Vagrantfile to direct Vagrant on how it should setup your VM. I’m going to use ~/src/vagrant_projects for the purpose of this demo:

1
2
3

$ mkdir -p ~/src/vagrant_projects
$ cd ~/src/vagrant_projects
$ vim Vagrantfile

Let’s take a look at a sample Vagrantfile that I use to get Puppet installed on a box:

1
2
3
4
5
6
7
8

Vagrant::Config.run do |config|
  config.vm.box       = "centos-6.3-x86_64"
  config.vm.box_url   = "https://saleseng.s3.amazonaws.com/boxfiles/CentOS-6.3-x86_64-minimal.box"
  config.vm.host_name = "development.puppetlabs.vm"
  config.vm.network :hostonly, "192.168.33.10"
  config.vm.forward_port 80, 8084
  config.vm.provision :shell, :path => "centos_6_x.sh"
end

Stepping through this file line-by-line, the first two config.vm lines establish the box we want to use for our development VM as well as the URL to the box file where it can be downloaded (in the event that it does not exist on our system). Because, initially, this box will NOT be known to Vagrant, it will attempt to reach out to that address and download it (note that the URL to THIS PARTICULAR BOX is subject to change - please find a box file that works for you and substitute its URL in the config.vm.box_url config setting). The next three lines define the machine’s hostname, the network type, and the IP address for this VM. In this case, I’m using a host-only network and giving it an IP address on a made-up 192.168.33.0/24 subnet (feel free to use your own private IP range as long as it doesn’t conflict with anything). The next line is forwarding port 80 on the VM to port 8084 on my local laptop - this allows you to test out web services by simply navigating to http://localhost:8084 from your web browser. I’ll save explaining the last line for the next section.

NOTE: For more documentation on these settings, visit Vagrant’s documentation site as it’s quite good

Getting Puppet on your VM

The final line in the sample Vagrantfile runs what’s called the ‘Shell Provisioner’ for Vagrant. Essentially, it runs a shell script on the VM once it’s been booted and configured. What does this shell script do?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

#!/usr/bin/env bash
# This bootstraps Puppet on CentOS 6.x
# It has been tested on CentOS 6.3 64bit

set -e

REPO_URL="http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-6.noarch.rpm"

if [ "$EUID" -ne "0" ]; then
  echo "This script must be run as root." >&2
  exit 1
fi

if which puppet > /dev/null 2>&1; then
  echo "Puppet is already installed"
  exit 0
fi

# Install puppet labs repo
echo "Configuring PuppetLabs repo..."
repo_path=$(mktemp)
wget --output-document=${repo_path} ${REPO_URL} 2>/dev/null
rpm -i ${repo_path} >/dev/null

# Install Puppet...
echo "Installing puppet"
yum install -y puppet > /dev/null

echo "Puppet installed!"

As you can see, it sets up the Puppet Labs el6 repository containing the current packages for Puppet/Facter/Hiera/PuppetDB/etc and installs the most recent version of Puppet and Facter that are in the repository. This will ensure that you have the most recent version of Puppet on your VM, and you don’t need to worry about creating a new box every time Puppet releases a new version.

This code came from Mitchell’s puppet-bootstrap repo where he maintains a list of scripts that will bootstrap Puppet onto many of the common operating systems out there. This code was current as of the initial posting date of this blog, but make sure to check that repo for any updates. If you’re maintaining your OWN provisioning script, consider filing pull requests against Mitchell’s repo so we can ALL benefit from good code and don’t have to keep creating ‘another wheel’ just to provision Puppet on VMs!

Spin up your VM

Once you’ve created a Vagrantfile in a directory, the next logical thing to do is to test out Vagrant and fire up your VM. Let’s first check the status of the vm:

1
2
3
4
5
6
7
8

$ vagrant status

Current VM states:

default                  not created

The environment has not yet been created. Run `vagrant up` to
create the environment.

As expected, this VM has yet to be created, so let’s do that by doing a vagrant up

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

$ vagrant up

[default] Box centos-6.3-x86_64 was not found. Fetching box from specified
URL...
[vagrant] Downloading with Vagrant::Downloaders::HTTP...
[vagrant] Downloading box:
https://saleseng.s3.amazonaws.com/boxfiles/CentOS-6.3-x86_64-minimal.box
[vagrant] Extracting box...
[vagrant] Verifying box...
[vagrant] Cleaning up downloaded box...
[default] Importing base box 'centos-6.3-x86_64'...
[default] The guest additions on this VM do not match the install version of
VirtualBox! This may cause things such as forwarded ports, shared
folders, and more to not work properly. If any of those things fail on
this machine, please update the guest additions and repackage the
box.

Guest Additions Version: 4.1.18
VirtualBox Version: 4.1.23
[default] Matching MAC address for NAT networking...
[default] Clearing any previously set forwarded ports...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] -- 80 => 8084 (adapter 1)
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Booting VM...
[default] Waiting for VM to boot. This can take a few minutes.
[default] VM booted and ready for use!
[default] Configuring and enabling network interfaces...
[default] Setting host name...
[default] Mounting shared folders...
[default] -- v-root: /vagrant
[default] Running provisioner: Vagrant::Provisioners::Shell...
Configuring PuppetLabs repo...
warning: 
/tmp/tmp.FvW0K7FJWU: Header V4 RSA/SHA1 Signature, key ID 4bd6ec30: NOKEY
Installing puppet
warning: 
rpmts_HdrFromFdno: Header V4 RSA/SHA1 Signature, key ID 4bd6ec30: NOKEY
Importing GPG key 0x4BD6EC30:
 Userid : Puppet Labs Release Key (Puppet Labs Release Key) <info@puppetlabs.com>
 Package: puppetlabs-release-6-6.noarch (installed)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs
Warning: RPMDB altered outside of yum.
Puppet installed!

Vagrant first noticed that we did not have the CentOS box on our machine, so it downloaded, extracted, and verified the box before importing it and creating our custom VM. Next, it configured the VM’s network settings according to our Vagrantfile, and finally it provisioned the box using the script we passed in the Vagrantfile.

We’ve now got a VM running and Puppet is installed. Let’s ssh to our VM and check the Puppet Version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ vagrant ssh

Last login: Tue Jul 10 22:56:01 2012 from 10.0.2.2
[vagrant@development ~]$ puppet --version
3.0.2
[vagrant@development ~]$ hostname
development.puppetlabs.vm
[vagrant@development ~]$ exit
logout
Connection to 127.0.0.1 closed.

$ vagrant destroy -f
[default] Forcing shutdown of VM...
[default] Destroying VM and associated drives...

Cool - so we demonstrated that we could ssh into the VM, check the Puppet version, check the hostname to ensure that Vagrant had set it correctly, exit out, and then we finally destroyed the VM with vagrant destroy -f. The next step is to actually configure Puppet to DO something with this VM…

Using Puppet to setup your node

The act of GETTING a clean VM is all well and good (and is probably magic enough for most people out there), but the purpose of this post is to demonstrate a workflow for testing out Puppet code changes. In the previous step we showed how to get Puppet installed, but we’ve yet to demonstrate how to use Vagrant’s built-in Puppet provisioner to configure your VM. Let’s use the example of a developer wanting to spin up a LAMP stack. To manually configure that would require installing a number of packages, editing a number of config files, and then making sure services were installed (among other things). We’re going to use some of the Puppet modules from the Puppet Forge to tackle these tasks and make Vagrant automatically configure our VM.

Scaffolding Puppet

We need a way to pass our Puppet code to the VM Vagrant creates. Fortunately, Vagrant has a way to define Shared Folders that can be shared from your workstation and mounted on your VM at a particular mount point. Let’s modify our Vagrantfile to account for this shared folder:

1
2
3
4
5
6
7
8
9
10
11

Vagrant::Config.run do |config|
  config.vm.box       = "centos-6.3-x86_64"
  config.vm.box_url   = "https://saleseng.s3.amazonaws.com/boxfiles/CentOS-6.3-x86_64-minimal.box"
  config.vm.host_name = "development.puppetlabs.vm"
  config.vm.network :hostonly, "192.168.33.10"
  config.vm.forward_port 80, 8084
  config.vm.provision :shell, :path => "centos_6_x.sh"

  # Puppet Shared Folder
  config.vm.share_folder "puppet_mount", "/puppet", "puppet"
end

The syntax for the config.vm.share_folder line is that the first argument is a logical name for the shared folder mapping, the second argument is the path IN THE VM where this folder will be mounted (so, a folder called ‘puppet’ in the root of the filesystem), and the last argument is the path to the folder ON YOUR WORKSTATION that will be mounted in the VM (it can be a full or relative path - which is what we’ve done here). This folder hasn’t been created yet, so let’s create it (and a couple of subfolders):

1
2

$ cd ~/src/vagrant_projects
$ mkdir -p puppet/{manifests,modules}

This command will create the puppet directory in the same directory that contains our Vagrantfile, and then two subdirectories, manifests and modules, that will be used by the Puppet provisioner later. Now that we’ve told Vagrant to create our shared folder, and we’ve created the folder structure, let’s bring up the VM with vagrant up again, ssh into the VM with vagrant ssh, and then check to see that the folder has been mounted.

1
2
3
4
5
6
7
8
9

$ vagrant up

<output suppressed - see above for example output>

$ vagrant ssh

Last login: Tue Jul 10 22:56:01 2012 from 10.0.2.2
[vagrant@development ~]$ ls /puppet
manifests  modules

Great! We’ve setup a shared folder. To further test it out, you can try dropping a file in the puppet directory or one of its subdirectories - it should immediately show up on the VM without having to recreate the VM (because it’s a shared folder). There are pros and cons with this workflow - the main pro is that changes you make on your workstation will immediately be reflected in the VM, and the main con is that you can’t symlink folders INSIDE the shared folder on your workstation because of the nature of symlinks.

Installing the necessary Puppet Modules

Since we’ve already spun up a new VM and ssh’d into it, let’s use our VM to download modules we’re going to need to setup our LAMP stack:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

[vagrant@development ~]$ puppet module install puppetlabs/apache --target-dir /puppet/modules/
Notice: Preparing to install into /puppet/modules ...
Notice: Downloading from https://forge.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/puppet/modules
└─┬ puppetlabs-apache (v0.5.0-rc1)
  ├── puppetlabs-firewall (v0.0.4)
  └── puppetlabs-stdlib (v3.2.0)

[vagrant@development ~]$ puppet module install puppetlabs/mysql --target-dir /puppet/modules/
Notice: Preparing to install into /puppet/modules ...
Notice: Downloading from https://forge.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/puppet/modules
└── puppetlabs-mysql (v0.6.1)

[vagrant@development ~]$ ls /puppet/modules/
apache  concat  firewall  mysql  stdlib

The puppet binary has a module subcommand that will connect to the Puppet Forge to download Puppet modules and their dependencies. The commands we used will install Puppet Labs’ apache and mysql modules (and their dependencies). We’re also passing the --target-dir argument that will tell the puppet module subcommand to install the module into our shared directory (instead of Puppet’s default module path).

I’m choosing to use puppet module to install these modules, but there are a multitude of other methods you can use (from downloading the modules directly out of Github to using a tool like librarian-puppet). The point is that we need to ultimately get the modules into the modules directory in our shared puppet folder - however you want to do that works for me :)

Once the modules are in puppet/modules, we’re good. You only ever need to do this step ONCE. Because this folder is a shared folder, you can now vagrant up and vagrant destroy to your heart’s content - Vagrant will not remove the content in our shared folder when a VM is destroyed. Remember, too, that any changes made to those modules from either the VM or on your Workstation will be IMMEDIATELY available to both.

Since we’re now done with the VM for now, let’s destroy it with vagrant destroy

1

$ vagrant destroy

Classifying your development VM

The modules we installed are a framework that we will use to configure the node. The act of directing the actions that Puppet should take on a particular node is called ‘Classification’. Puppet uses a file called site.pp to map Puppet code with the corresponding ‘node’ (or, in our case, our VM) that should receive it. Let’s create a site.pp file and open it for editing:

1
2

$ cd ~/src/vagrant_projects
$ vim puppet/manifests/site.pp

Let’s create a site.pp that will setup the LAMP stack on our development.puppetlabs.vm that we create with Vagrant:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

node 'development.puppetlabs.vm' {
  # Configure mysql
  class { 'mysql::server':
    config_hash => { 'root_password' => '8ZcJZFHsvo7fINZcAvi0' }
  }
  include mysql::php

  # Configure apache
  include apache
  include apache::mod::php
  apache::vhost { $::fqdn:
    port    => '80',
    docroot => '/var/www/test',
    require => File['/var/www/test'],
  }

  # Configure Docroot and index.html
  file { ['/var/www', '/var/www/test']:
    ensure => directory
  }

  file { '/var/www/test/index.php':
    ensure  => file,
    content => '<?php echo \'<p>Hello World</p>\'; ?> ',
  }

  # Realize the Firewall Rule
  Firewall <||>
}

Again, the point of this post is not about writing Puppet code but more about testing the Puppet code you write. The above node declaration will setup MySQL with a root password of ‘puppet’, setup Apache and a VHost for development.puppetlabs.vm with a docroot out of /var/www/test, setup an index.php file for Apache, and setup a Firewall rule to allow access through to port 80 on our VM.

Setting up the Puppet provisioner for Vagrant

We’re going to have to modify our Vagrantfile one more time to tell Vagrant to use the Puppet provisioner to execute our Puppet code and setup our VM:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Vagrant::Config.run do |config|
  config.vm.box       = "centos-6.3-x86_64"
  config.vm.box_url   = "https://saleseng.s3.amazonaws.com/boxfiles/CentOS-6.3-x86_64-minimal.box"
  config.vm.host_name = "development.puppetlabs.vm"
  config.vm.network :hostonly, "192.168.33.10"
  config.vm.forward_port 80, 8084
  config.vm.provision :shell, :path => "centos_6_x.sh"

  # Puppet Shared Folder
  config.vm.share_folder "puppet_mount", "/puppet", "puppet"

  # Puppet Provisioner setup
  config.vm.provision :puppet do |puppet|
    puppet.manifests_path = "puppet/manifests"
    puppet.module_path    = "puppet/modules"
    puppet.manifest_file  = "site.pp"
  end
end

Notice the block for the Puppet provisioner that sets up the manifest path (i.e. where to find site.pp), the module path (i.e. where to find our Puppet modules), and the name of our manifest file (i.e. site.pp). Again, this is all documented on the Vagrant documentation page should you need to use it for reference.

This bumps the number of provisioners in our Vagrantfile to two, but which one goes first? Vagrant will iterate through the Vagrantfile procedurally, so the Shell provisioner will always get checked first and then the Puppet provisioner will get checked second. This allows us to be certain that Puppet will always be installed before attempting to use the Puppet provisioner. You could continue to add as many provisioning blocks as you like - Vagrant will iterate through them procedurally as it encounters them.

Give the entire workflow a try

Now that we have our Vagrantfile finalized, our Puppet directory structure setup, our Puppet modules installed, and our site.pp file set to classify our new VM, let’s actually let Vagrant do what it does best and setup our VM:

1

$ vagrant up

You should see Vagrant use the Shell provisioner to install Puppet, hand off to the Puppet provisioner, and then use Puppet to setup a LAMP stack on our VM. After everything completes, try visiting http://localhost:8084 in your web browser and see if you get a shiny “Hello World” staring back at you. If you do - Awesome! If you don’t, check the error messages to determine if there are typos in the Puppet code or if something went wrong in the Vagrantfile.

Where do you take it from here?

The first thing to do is to take the Vagrantfile you’ve created and put it under revision control so you can track the changes you make. I personally have a couple of workflows up on Github that I use as templates when I’m testing out something new. You’ll probably find that your Vagrantfile won’t change much - just the modules you use for testing.

Now that you understand the pattern, you can expand it to fit your workflow. Single-vm projects are great when you’re testing a specific component, but the next logical step is to test out multi-tiered components/applications. In these instances, Vagrant has the ability to spin up multiple VMs from a single Vagrantfile. That workflow saves a TON of time and lets you create your own private network of VMs for the purpose of simulating changes. That’s a post for another time, though…

Get involved

Stay tuned to the Vagrant website for updates on the VMware provisioner. Stability with Virtualbox has notoriously been an issue, but, as of this posting, things have been relatively rock-solid for me (using Virtualbox version 4.1.23 on OS X).

If you want to keep up-to-date on all things Vagrant, follow Mitchell on Twitter, check out #vagrant on Freenode, join the Vagrant list, and check out Google for what other folks have done!

A GIANT thank you to Mitchell Hashimoto for all the work he’s done on Vagrant - I can’t count the number of hours it’s saved me personally (let ALONE everyone at Puppet Labs!

I hate Tim Sutton in the same way I hate Steven Singer. I hate Tim because he actually IMPLEMENTS some of the things on my list of ‘shit to tinker with when you get some free time’ instead of just TALKING about them. He and Pepijn Bruienne have been working on some code for Veewee that will allow you to automate the creation of OS X VMs in VMware Fusion. For those who are interested, you can check out the pull request containing this code and comment/help them out. For everyone else, read on and entertain the idea…

Prerequisites:

1. OS X
2. VMware Fusion
3. Git
4. Ruby 1.9.3 and rbenv
5. Mountain Lion (10.8) installer application

OS X

This walkthrough assumes that you’re running OS X on your development workstation. That’s what I use, and it’s the workflow I know.

VMware Fusion

If you’ve used Veewee before, odds are good that you MIGHT have used it to build baseboxes for Vagrant and/or Virtualbox. Veewee + Vagrant is a post for ANOTHER time, but in short it’s an awesome workflow for testing automation like Puppet on Linux or Windows. When I originally tried using Veewee to build OS X VMs, I had absentmindedly tried to do this using Virtualbox…which isn’t supported. As such, VMware Fusion is the only virtualization platform (as of this posting date) that is supported with this method. I’m using VMware Fusion 5 pro, but YMMV.

Git

Because the code that supports OS X only exists in Tim’s fork of Veewee (as of this posting date), we’re going to have to install/use Veewee from source. That introduces a bit more complexity, but hold my hand - we’ll get through this together. Git comes with the XCode Command Line Tools or can be installed with native packages.

Ruby 1.9.3 and rbenv

Veewee uses the gssapi gem which requires Ruby 1.9.1 or higher. The problem, though, is that the version of Ruby that comes with OS X is 1.8.7. There are typically two camps when it comes to getting a development version of Ruby on OS X: rbenv or RVM. I recommend rbenv because it doesn’t screw with your path and it’s a bit more lightweight, so that’s the path I’m going to take in this writeup. Octopress has instructions for getting rbenv on your machine - so make sure to check those out for this step. The instructions describe using rbenv to install Ruby version 1.9.3 - and that’s the version we’ll use here.

Mountain Lion installer application

This workflow supports creating a VM for Mountain Lion (10.8), but it SHOULD also work for Lion (10.7). The final piece of our whole puzzle is the installer application from the App Store. Get that application somehow and drop it in the /Applications directory (that’s where the App store puts it by default). We REALLY only need a single disk image from the installer, and we’ll get that next.

Some assembly required…

Now that you have all the pieces, let’s tie this FrankenVM up with some code and ugly bash, shall we? Splendid.

Copy out the installation disk image

In the last step, we downloaded the Install OS X Mountain Lion.app file from the App Store to /Applications, but we’ll want to extract the main installation disk image somewhere where we can work with it. I’m going to make a copy on the Desktop so we don’t screw up the main installer:

1

$ cp /Applications/Install\ OS\ X\ Mountain\ Lion.app/Contents/SharedSupport/InstallESD.dmg ~/Desktop

Beautiful. This should take a minute as it’s a sizeable file, but in the end you’ll have the installation disk image on your Desktop. For now this is fine, but we’ll be revisiting it later…

Clone Tim’s fork of Veewee

As if the writing of this blog post, the code you need is ONLY in Tim’s fork, so let’s pull that down to somewhere where we can work with it:

1
2
3
4
5
6

## I prefer to work with code out of a 'src' directory in my home directory
$ mkdir ~/src
$ cd ~/src

$ git clone http://github.com/timsutton/veewee
$ cd ~/src/veewee

Install Gems for veewee

We now have the Veewee source in ~/src/veewee, but we need to ensure all the Rubygems necessary to make Veewee work have been installed. We’re going to do this with Bundler. Let’s switch to Ruby 1.9.3 and get Bundler installed:

1
2
3

$ cd ~/src/veewee
$ rbenv local 1.9.3
$ gem install bundler

Next, let’s use bundler to install the rest of the gems we need to use Veewee:

1

$ bundle install

Once that command completes, Bundler will have installed all the necessary gems for Veewee and we can move on.

Define your new VM

Veewee has templates for most operatingsystems that can be used to spin up a ‘vanilla’ VM. Tim’s code provides a template called ‘OSX-10.8.2’ containing the necessary scaffolding for building a vanilla 10.8.2 VM. Let’s create a new VM project based on this template called ‘osx-vm’ with the following:

1
2

$ cd ~/src/veewee
$ bundle exec veewee fusion define 'osx-vm' 'OSX-10.8.2'

This will create definitions/osx-vm inside the Veewee directory with the template code from templates/OSX-10.8.2. We’re almost ready to let Veewee create our VM, but we need an installation ‘ISO’ first…

Prepare an ‘ISO’ for OS X

The prepare_veewee_iso.sh script in Veewee’s templates/OSX-10.8.2/prepare_veewee_iso directory provides awesome detail as to why we can’t use the vanilla InstallESD.dmg file to install 10.8 in our new VM. Feel free to open that file and read through the detail, or check out the details online for more information. Let’s use that script and prepare an installation ‘ISO’ for our new VM:

1
2
3

$ cd ~/src/veewee
$ mkdir iso
$ sudo templates/OSX-10.8.2/prepare_veewee_iso/prepare_veewee_iso.sh ~/Desktop/InstallESD.dmg iso/

You’ll need to be root to do this, but the script should handle everything necessary to prepare the ISO and drop it into the iso directory we created.

Define any additional post-installation tasks

Veewee supports post-installation tasks through the postinstall.sh script in the definitions/osx-vm folder. By default, this script will install VMware tools, setup Vagrant keys, download the XCode Command Line Tools, and install Puppet via Rubygems. Because this is all outlined in the postinstall.sh script, you’re free to modify this code or add your own steps. Here’s the current postinstall.sh script as of this posting:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

date > /etc/vagrant_box_build_time
OSX_VERS=$(sw_vers -productVersion | awk -F "." '{print $2}')
# Install VMware tools if we were built with VMware
if [ -e .vmfusion_version ]; then
  TMPMOUNT=`/usr/bin/mktemp -d /tmp/vmware-tools.XXXX`
  hdiutil attach darwin.iso -mountpoint "$TMPMOUNT"
  installer -pkg "$TMPMOUNT/Install VMware Tools.app/Contents/Resources/VMware Tools.pkg" -target /
  # This usually fails
  hdiutil detach "$TMPMOUNT"
  rm -rf "$TMPMOUNT"
fi

# Installing vagrant keys
mkdir /Users/vagrant/.ssh
chmod 700 /Users/vagrant/.ssh
curl -k 'https://raw.github.com/mitchellh/vagrant/master/keys/vagrant.pub' > /Users/vagrant/.ssh/authorized_keys
chmod 600 /Users/vagrant/.ssh/authorized_keys
chown -R vagrant /Users/vagrant/.ssh

# Get Xcode CLI tools for Lion (at least to build Chef)
# https://devimages.apple.com.edgekey.net/downloads/xcode/simulators/index-3905972D-B609-49CE-8D06-51ADC78E07BC.dvtdownloadableindex
TOOLS=clitools.dmg
if [ "$OSX_VERS" -eq 7 ]; then
  DMGURL=http://devimages.apple.com/downloads/xcode/command_line_tools_for_xcode_os_x_lion_november_2012.dmg
elif [ "$OSX_VERS" -eq 8 ]; then
  DMGURL=http://devimages.apple.com/downloads/xcode/command_line_tools_for_xcode_os_x_mountain_lion_november_2012.dmg
fi
curl "$DMGURL" -o "$TOOLS"
TMPMOUNT=`/usr/bin/mktemp -d /tmp/clitools.XXXX`
hdiutil attach "$TOOLS" -mountpoint "$TMPMOUNT"
installer -pkg "$(find $TMPMOUNT -name '*.mpkg')" -target /
hdiutil detach "$TMPMOUNT"
rm -rf "$TMPMOUNT"
rm "$TOOLS"

# Get gems - we should really be installing rvm instead, since we can't even compile Chef or have a Ruby dev environment..
gem update --system
gem install puppet --no-ri --no-rdoc
# gem install chef --no-ri --no-rdoc
exit

Build the VM

Everything we’ve done has all been for this step. With the ISO built, the VM defined, and all necessary Gems installed, we can finally BUILD the vm with the following command:

1
2

$ cd ~/src/veewee
$ bundle exec veewee fusion build osx-vm

This process takes the longest - you should see VMware Fusion fire up, a new VM get created, and follow the process of OS X being installed into the VM. When it completes, your VM will have been created. Just like most Vagrant workflows, the resultant vm will have a vagrant user whose password is also vagrant. Feel free to login and ensure that everything looks good.

Now what?

At this point, I would snapshot the VM before making any changes. Because Virtualbox isn’t yet supported with Veewee for building OS X VMs (and Vagrant doesn’t currently include VMware Fusion support for its workflow), this VM isn’t going to fit into a Vagrant workflow…yet. What you have, though, is a vanilla OS X VM that can be built on-demand (or reverted to a snapshot) to test whatever configuration changes you need to make (all 11.36 Gb worth of it).

As you would expect, this is all pretty experimental for the moment. If you’re a Ruby developer who needs an OS X VM for testing purposes but have never managed OS X and it’s ‘quirky’ imaging process, this workflow is for you. For everyone else, it’s probably an academic proof-of-concept that’s more interesting from the point of view of “Look what you can do” versus “Let’s make this my primary testing workflow.”

Credit goes to Patrick Dubois for creating and managing the Veewee project, and to Tim Sutton and Pepijn Bruienne - like I mentioned before - for the work they’ve done on this. You can speak with them directly in the ##osx-server chatroom on Freenode, or by checking them out on Twitter.

Masochists are everywhere. Managing a system by hand appeals to a certain subset of the population, but I joined Puppet Labs because I was a fan of automating the shit out of menial tasks. I have no burning desire to handcraft an artisanal blog made out of organic bits. I tried web development at one point in my life and learned an important lesson:

I am not a web developer

What I DO want is to have a platform to share information and code I’ve accumulated along the way that:

1. Doesn’t look like hell
2. Doesn’t take forever to update
3. Doesn’t require me being online to write a post
4. Allows me to post code that’s syntactically highlighted and easy to copy/paste
5. Accepts Markdown
6. Fits into my DVCS workflow
7. Is free - because screw paying for A BLOG

Seriously, is this too much to ask?

Originally I waded the waters of Posterous but found that not only did it have awkward Markdown syntax, but staging of blog posts was cumbersome. Also, while there WAS gist integration, the code you posted looked like crap. That sucks because most of what I post has code attached.

Others have sold their soul for Wordpress/Drupal/Bumblefuck CMS (whether hosted or unhosted platforms), but it still felt like too much process and not enough action for my taste.

Github to the rescue…again

The answer, it turns out, was right in front of my face the whole time. Github Pages has always been available to host web content or Jekyll sites - I’m just late to the damn party.

Hosting static web content was out; like I said before, I don’t really want to ‘manage’ this thing. Jekyll, however, intrigued me. Jekyll is essentially a static site generator that allows you to throw Markdown at it and get static content in the end. There are even Jekyll bootstrap projects aimed at making it (even more) stupid simple. I tried plain vanilla Jekyll and realized that I didn’t really want/need that level of control (again, I’m not into web development). I pulled down Jekyll Bootstrap, but this time it felt a little TOO ‘template-y’. Next, I pulled down an awesome Jekyll template called Left by Zach Holman (seen, conveniently, in the picture on the left). I REALLY liked the look of Left, but was stuck with Jekyll’s code formatting that was…less than ideal. Jekyll is pluggable (and people have made plugins to fix this sort of thing), but I still didn’t have enough experience at that time to be able to deal with the plugins in an intelligent manner.

Octopress = Jekyll + Plugins + <3

During my plugin party, I discovered Octopress, which basically had EVERYTHING I wanted wrapped with a templated bow. I loved the way it rendered code, it supported things like fenced code blocks , and it seemed REALLY simple to update and deploy code. The thing I DIDN’T like was that NEARLY EVERY DAMN OCTOPRESS SITE LOOKS EXACTLY THE SAME! I know I said that I’m not a web developer and didn’t want to tweak styles a great bit, but damn - couldn’t I get a BIT of differentiation? That can be done, but you’re still editing styles (so some CSS is necessary - but not much). After evaluating all three options, I opted for Octopress.

Documentation? Who knew!?

Octopress.org has GREAT documentation for getting setup with an Octopress blog. They even have documentation for setting up Ruby 1.9.3 on rbenv or RVM (I recommend rbenv for reasons beyond this blog), so make sure to check it out if you’re unfamiliar with either. To not reinvent that documented wheel, make sure to check out that site to get Octopress setup (I recommend cloning all repositories to somewhere in your home directory like ~/src or ~/repos, but other than that their docs are solid). Normally I post the dirty technical details, but the point of this post is to outline WHY I made the decision I did and WHY it works for ME.

Pretty code is pretty

I’m not gonna lie - syntactical highlighting with the Solarized theme was pretty sexy. Let’s look at some Ruby:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

def create
    if resource[:value_type] == :boolean
      unless resource[:value].first.to_s =~ /(true|false)/i
        raise Puppet::Error, "Valid boolean values are 'true' or 'false', you specified '#{resource[:value].first}'"
      end
    end

    if File.file? resource[:path]
      plist = read_plist_file(resource[:path])
    else
      plist = OSX::NSMutableDictionary.alloc.init
    end

    case resource[:value_type]
    when :integer
      plist_value = Integer(resource[:value].first)
    when :boolean
      if resource[:value].to_s =~ /false/i
        plist_value = false
      else
        plist_value = true
      end
    when :hash
      plist_value = resource[:value].first
    else
      plist_value = resource[:value]
    end

    plist[resource[:key]] = plist_value

    write_plist_file(plist, resource[:path])
  end

How about some Puppet code?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

property_list_key { 'simple':
  ensure => present,
  path => '/tmp/com.puppetlabs.puppet',
  key    => 'simple',
  value  => 'value',
}

property_list_key { 'boolean':
  ensure     => present,
  path       => '/tmp/com.puppetlabs.puppet',
  key        => 'boolean',
  value      => false,
  value_type => 'boolean',
}

property_list_key { 'hashtest':
  ensure     => present,
  path       => '/tmp/com.puppetlabs.puppet',
  key        => 'hashtest',
  value      => { 'key' => 'value' },
  value_type => 'hash'
}

Doesn’t that look awesome? What if you have a gist? You can embed those too (and they’re also linkable):

Want to know how much code it took to do that? About this much:

1

{% gist 3218523 %}

For a blog with a ton of code, something like this is pretty damn important.

Markdown formatting

Markdown is becoming more prolific as a lightweight way to format text. If you’ve used Github to comment on code, then you’ve probably used markdown syntax at some point in your life. You’ll notice the recurring goal of being quick and precise, and markdown really ‘does it for me’.

Workflow++

It’s really liberating to write a blog post in (insert favorite text editor). Travelling as often as I do, I’m usually jotting down notes in vi because I’m in a plane or somewhere without internet access. Octopress not only lets you write offline, but it will let you generate and preview your site locally while being offline. That’s totally handy. My workflow typically looks like this:

• Generate a post

To start a new post, you can create a new file by hand, or use the provided Rakefile to scaffold it out for you (NOTE: to see all commands available to the Rakefile, you can run rake -T). Here’s how to scaffold with rake:

1
2

$ rake 'new_post["Title of my post"]'
$ vi source/_posts/2013-01-17-title-of-my-post.markdown

• Edit the post file
• Generate the site content

Remember that Octopress/Jekyll generates static content to be uploaded to your host. With every change, you’ll need to re-generate the content. Yeah, that kinda sucks, but that action has been abstracted away in the Rakefile with the following command:

1

$ rake generate

• Display and view the site

1

$ rake preview

The following command will serve up your page over Webrick in the terminal, just navigate to http://localhost:4000 in your browser to see a local copy of how your site will look once deployed. Once you’re done, just do a Control+c from your terminal to cancel the process.

• Edit/generate/preview until done
• Commit your code and deploy

Because everything is a series of flat-files, you can use git to keep it all under version control. Did you make a mistake? Revert to a previous commit. Deploying to your blog is similarly easy, as you’ll see in the next steps

Be a unique snowflake

I used this article to help change the color scheme of my blog, and checked out a list of other Octopress sites to steal a couple of other tweaks. That’s all I needed for customization, but if you need more knobs they’re there for you to twist.

Hosted by your pals at Github

Github Pages is a free service for any Github user (again free to join) and is an EXCELLENT FIT for an Octopress blog. As you would expect, Octopress has great documentation for enabling deployment to Github Pages. If you have your own custom domain, you can STILL use Github Pages (hint, the instructions are on that page too).

Fuck it. Ship it.

The act of updating a blog has GOT to be frictionless. Is this simple enough for you:

1

rake deploy

Yep. That’s all it takes to deploy code to your blog once you’ve setup Github Pages. Forget about logging in, wading through a GUI, cutting/pasting content, FTPing things up to a host, and any of that crap. I’ve done that. ‘That’ sucks.

Write your own damn blog

You now have no excuse to NOT share all the cool things you’re doing. Seriously, if you can EDIT A TEXT FILE then you can ‘maintain’ a blog.

JAMF’s NetSUS Appliance - Netboot in a Box

Today, JAMF released a new appliance VM based on Ubuntu 10.04 LTS (Lucid) that, for once, provides an ‘out of the box’ implementation of Netboot and Software Update Service WITHOUT requiring OS X hardware (based on Reposado) and other open-source technologies.

For those people who are relatively new to Linux, I thought I’d provide an easy walkthrough of how to get started using this VM on your Laptop. I’m going to demo this using VMware Fusion instead of VirtualBox mainly because I prefer how Fusion handles Networking. I’ve used many VMs, and Fusion always tends to be much more solid and stable for me. If you don’t have Fusion or don’t WANT to use Fusion, feel free to use VirtualBox.

Download and Convert the OVA File

The appliance can be downloaded directly from JAMF and comes as a VirtualBox OVA file. Rich Trouton has provided a great walkthrough on converting the OVA file to a VMX file, so I’ll link you to that post (since he did it so well).

Start up the VM

Once you have your VMX file, you can open it up with Fusion and import it into your Virtual Machine Library. There is one change we’ll want to make, so click on the VM and open its Settings panel (this is different from Fusion 3.0 and 4.0 - I’ll be describing the 4.0 method). In the Settings panel, you’ll want to select the Network Adapter icon and make sure the radio button next to “Share the Mac’s network connection (NAT)” is selected. This will create a local IP address so we can contact it directly from our Laptop. Once you’ve done that, start the VM and get to the login screen (you’ll need to click the Ok button through the message about logging into the NetSUS instance). Feel free to login with the username ‘shelluser’ and the password ‘shelluser’. You should see a Message of the Day splash with one of the sections being “IP address for eth0:”. Note the IP address, we’ll need it in the next section (if you don’t see this, just run ifconfig from the command line and look for the IP address for the eth0 adapter - it should be in the format of 192.168.x.x).

Prepare to Break Free!

It really sucks to work with VMs from WITHIN the VM window for various reasons (lack of Copy/Paste, loss of mouse, etc…), so we’re going to setup a host entry on your local laptop so we can SSH into the VM as we need. I’m going to give my VM a hostname of ‘netsus.puppetlabs.vm’ but you can name it whatever you want. Let’s first edit the /etc/hosts file on your laptop (NOT within the VM, this is ON your laptop) by running the following from the command line:

1

sudo vim /etc/hosts

Feel free to use pico/nano to edit /etc/hosts in case you don’t have any experience with vim. We want to add the following line in the /etc/hosts file:

1

192.168.217.154 netsus.puppetlabs.vm

NOTE SUBSTITUTE 192.168.217.154 WITH THE IP ADDRESS YOU GOT FROM THE PREVIOUS STEP! That’s the IP address that was assigned on MY laptop, and it will definitely be different from the IP address you got on YOUR computer. Also, you don’t HAVE to use ‘netsus.puppetlabs.vm’ - you can assign it any name you want. Save and close /etc/hosts and then let’s test connectivity from the command line by doing a ping:

1

ping 192.168.217.154

As long as we have connection, we’re fine (remember, substitute your IP Address for mine). If you DON’T have connectivity, make sure your VM is running and review your /etc/hosts file to make sure the changes were saved (also, check the IP address on your VM again).

Snapshot Time

The beauty of Fusion is that you can take snapshots and revert to them should you mess things up. Let’s take a snapshot now in case we screw things up in the following steps. I’m using VMware Fusion 4.0, so I take a snapshot by clicking on the VM window, clicking on the Virtual Machine menu at the top, and then down to Snapshots. From there, you can click the ‘Take Snapshot’ button and let it do its thing. When it’s done, hit the close button and return to the VM

Enable SSH and Login

By default, the VM doesn’t have the SSHD process running, so we can’t yet SSH in from our laptop. Enable this by doing the following from within the VM:

1

sudo apt-get install openssh-server

Hit “Y” when it asks if it should download and install. When it completes, switch back to your LAPTOP (i.e. NOT inside the VM), open up Terminal, and do the following:

1

ssh shelluser@netsus.puppetlabs.vm

Type yes to add the SSH key to your known_hosts file and use the password ‘shelluser’ to login. Tada! Now you can work directly from Terminal and enable copy/pasting directly from your Terminal window. You can always type exit to close the ssh connection when you’re done.

Install Puppet

So, it wouldn’t be one of my write-ups if I DIDN’T get Puppet installed. Actually, if you’ve only managed Macs, then you might not have seen the benefit of Puppet before. Now that we’re on Linux, you’ll definitely see the benefit of Puppet (especially if you’re not familiar with the service/package manipulation commands). Puppet can help abstract that for you, so we’re going to enable it.

The version of Puppet that’s in the main package repository is quite old (0.25 as opposed to the current 2.7.10 version we support). Let’s add Puppet Labs’ Apt Repository to the list of repositories queried so we can get a recent version of Puppet. Do that by opening up the /etc/apt/sources.list file for editing with the following:

1

sudo vim /etc/apt/sources.list

If it prompts you for your password, go ahead and enter it. Also, if you’re not familiar with vi or vim, I’ll try and help you with the basics. Arrow down to the line JUST above the line that begins with “deb http://” and press the i button on your keyboard to insert the following lines:

1
2

deb http://apt.puppetlabs.com/ubuntu lucid main
deb-src http://apt.puppetlabs.com/ubuntu lucid main

When you’re done, hit the escape key and press ZZ (hold shift and hit Z twice, you want two capital-Z’s) to save and close the file. This will add Puppet Labs’ repository to the list of repos queried.

Next, we’ll need to add Puppet Labs’ GPG key to the system so we can validate the packages that are downloaded. From the terminal, execute the following:

1

 gpg --recv-key 4BD6EC30

The first time you run it, it will bother you about needing to create the keyserver file and won’t run correctly. Let’s execute it correctly to receive the key:

1

 gpg --recv-key 4BD6EC30

Great, we should have the key, now we want to load it. Do that with this command:

1

gpg -a --export 4BD6EC30 | sudo apt-key add -

Finally, let’s update apt so that it knows about the repository. Do that with this command:

1

sudo apt-get update

When it finishes running, you can finally install Puppet with the following command:

1

sudo apt-get install puppet

Type “Y” when it asks you if it should download and install all the dependencies. That’s it! Puppet should be installed! Run the following to see what version we’ve installed:

1

puppet --version

As of this writing, 2.7.10 is the current version. As long as you don’t have version 0.25 you should be good!

Snapshots away

It would be wise to take another snapshot at this point. Don’t worry, I’ll wait.

Inspect the System

Puppet lets us see what’s been installed on the VM. Take a look at all the packages on the system by doing the following:

1

sudo puppet resource package

Type your password when prompted and Puppet will return you with a list of all packages on the system and the versions that are running. Similarly, you can see what services are running with the following:

1

sudo puppet resource service

Notice that sshd is running and other services like netatalk (that enables AFP on Linux) should also be running. Sweet. We’ll come back to Puppet later.

Login to the Web GUI

If your VM is running, you should be able to open a web browser on your Laptop and navigate to the following address:

1

https://netsus.puppetlabs.vm

Note the httpS as we’ll need to use SSL to get into the Web GUI. When it asks if you want to accept the self-signed cert, make sure to choose Yes. You can login with the user ‘webadmin’ and the password ‘webadmin’

Once you’re logged in, feel free to read through JAMF’s instructions for enabling their services. They do a much better job than I about walking you through that.

Access to the Raw Files

Back in the Terminal window, you can access all the PHP/Shell magic that comprises the Web GUI in the /var/www/webadmin directory. You can poke around through all the files (and use sudo to open some of the key bits) at your leisure.

Note that /etc/dhcpd.conf is the main configuration file for the DHCPD service and it’s got some special bits to accommodate Apple Hardware.

Also, /var/appliance has some magic (and the /var/appliance/configurefornetboot script has some sed statements for fixing a stock dhcpd.conf file and enabling those Apple Hardware bits).

More Later?

That’s it for now - I just wanted to get something online to show people how to setup and poke around in the VM. Later I can show you how to manipulate and manage the VM with Puppet, which is pretty fun. Your VM should get you started and let you play around locally, but if you want to test out Netboot and the like then you’ll need an externally-accessible IP address (as apposed to the 192.168.x.x IP, which is only accessible from your Laptop). You can change the settings in VMware Fusion to enable that (like we did in step 2).

Hope this helped you out!

As a former Mac Sysadmin, I frequently felt like I had one toe in the world in which Linux Sysadmins frequently found themselves, but was surrounded by a culture of GUI-Clickery that either feared, scorned, or flat-out avoided tools that were command-line based (or, as many will point out, used the GUI (Graphic User Interface) tools because ‘they worked’ and that was all they needed). This sucked because Linux Sysadmins have some tools that, as a current colleague would say, ‘are kind-of awesome.’

One of those tools is Git, which is a Distributed Version Control System (or, DVCS). Git, in a TOTAL nutshell, is a tool that allows you to create multiple ‘what-if scenarios’ with very little cost and the ability to save yourself from…well, yourself.

Yes - You’re Going to Use the Command Line

Like I mentioned before, the GUI is ingrained in the culture of the Mac. It wasn’t until OS X that Mac Sysadmins finally had a proper command-line (which I will refer to as the CLI, which is short for Command Line Interface. If you’re not familiar with that terminology, I’m speaking about the interface you get when you run /Applications/Utilities/Terminal.app), so we’re a bit ‘new’ to the whole idea. I can help you if you fear or avoid the CLI; there’s always room for learning and I’ll try to hold your hand as much as possible. For those that downright refuse to use the CLI…well, I’m sorry?

What IS Git?

Git is relatively new in the DVCS world (being initially released in 2005). Developers have been using DVCS for MANY years with tools like CVS, Subversion, Git, Mercurial, and others. For a team of developers who ALL need to make contributions to a SINGLE project or code base, it’s vital that they have a way to modify a file without breaking something that someone else is doing. That’s what DVCS tools do - allow you to take something as large as a code base like OS X 10.7 (or something as small as a directory with a couple of files) and make changes without affecting everyone else who’s working on the SAME code base (or files in the directory).

So why am I choosing to talk about Git instead of, say, Subversion? That’s easy: lightweight branching and merging. Git allows you to easily create a ‘topic branch’ (or, using the metaphor above, a ‘what-if scenario’), make changes, and see if your changes are favorable or if they downright stink. If your changes are desirable, you can then ‘merge’ your topic branch back in with your master branch (or, the original state the files were in when you created the topic branch in the first place). If the changes don’t work well, you can just delete the topic branch and ‘no-harm, no-foul’; your original files remain unchanged and in the pristine condition that they existed when you first created the topic branch. True, Subversion could do the same thing, but Git allows MANY people to do the same thing and then EASILY merge in EVERYONE’S changes without having to fight Subversion (just trust me on this if you’ve never used Subversion).

And why am I choosing to talk about Git instead of Mercurial? Easy - Git is what I learned :) As I understand it Mercurial functions similarly, and I truly don’t have the experience with Mercurial to tell you the differences. I’m talking about Git because Git is what I use and Git is what my company uses.

So what does this have to do with me as a Mac Sysadmin?

Good question. Mac Sysadmins are more frequently finding themselves having to use the CLI to tune their clients and servers. Also, many Mac Sysadmins are finding that certain CLI tools are working better for their teams (in terms of simplicity, responsibility, and visibility). As they dip their toes into this CLI world, they’re finding that they want the same features they had in the GUI: the ability to do a ‘save-as’, to create ‘versions’, and to have the ability to have a rollback function similar to Time Machine. Git gives you this through the CLI (or, yes, even through the GUI as I will mention later in the article).

If you’re ready to take that step then follow me…

A note on Git and Github

You might have heard of Github. Github is a free (they have a free tier and plenty of subscription tiers of service) service that allows you to ‘push’ your git repositories up to them for storage in ‘the cloud’. Github is basically centralized-storage for your git repositories and is NOT NECESSARY for you to use git as a tool. In reality, you CAN use Github or setup something like Gitolite on a local server in your environment to get most of the benefits of Github without having to push your repositories to Github’s servers.

The bottom line: Github and git (the tool) are two different things. Github didn’t create git, and git doesn’t need Github to work properly. Keep that in mind as you read through.

Installing Git

Git can be installed in a variety of ways. First and foremost, you can download the latest package from Git’s Googlecode Page. You can also choose to install git through Macports or Homebrew as long as you have them installed (both of which require the Developer Tools…but so does The Luggage, the tool we’ll be using later). Simply download the package from Git’s Googlecode Page, install it, and you should be good to go!

If you’re using Macports, make sure the Developer Tools are installed, Macports is installed, and then do the following from the command line (PLEASE NOTE: The dollar sign is a prompt - you SHOULDN’T type it. Lines BEGINNING with a dollar sign, or a prompt, should be typed by you. Lines that DON’T begin with a prompt represent the output you receive from typing the command. If you don’t see a line that DOESN’T begin with a prompt, then I’m not listing the output. Got it? Good!):

1

$ sudo port install git-core

If you’re using Homebrew, make sure the Developer Tools are installed, Homebrew is installed, and then do the following:

1

$ sudo brew install git

To check to see where and what version of git is installed, execute the following commands :

1
2

$ which git
$ git --version

You should receive the path to where git is installed after you run the first command, and you should receive the version of git that’s currently installed on your system. Any version of Git greater-than or equal to 1.7 is probably ideal, though most of the commands we’ll be using will probably work just fine with previous versions.

Your first Git repo

I find it’s easier to understand Git if I describe an actual scenario instead of talking in terms of ‘git can do…’. I’m going to talk about using git with The Luggage. I’ve written about The Luggage in several articles on this site, so feel free to familiarize yourself with it if you haven’t used the tool before. As a quick review, The Luggage is a tool that lets you create plain text Makefiles that can be used to build packages to install files, deploy scripts, and basically distribute ‘things’ to the users in your organization. Auditing HOW a package is made can be incredibly difficult for a team of sysadmins, but The Luggage helps you with this by using plain text Makefiles. Git will give you the ability to track WHO made changes, WHEN they were made, and ROLLBACK the changes if you notice something undesirable happening. Git and The Luggage are a perfect combo if you’re a sysadmin.

Incidentally, The Luggage exists as a Github repository, so we have to CLONE it (or download it) from Github first before we can start playing with it. I recommend creating a directory in your home directory (/Users/-username-) and cloning your repositories there - that way you retain ownership of the files and can work on them at will. I put all my git repositories in ~/src, but if you’re using something like Mobile Home Directories or even Network Home Directories, then you’ll want to make sure that you’re not syncing these directories back to some central location and burning up your quota in some fashion. Let’s create our repositories directory and clone The Luggage source code with these steps:

1
2
3

$ mkdir -p ~/src
$ cd ~/src
$ git clone git://github.com/unixorn/luggage.git

Doing this will create the ~/src directory where we’re putting all our repositories and download The Luggage source code to ~/src/luggage. As you might be able to infer, git clone will clone an existing git repository from its location on another server (we call that the remote location, or just remote for short) down to a local directory. This is usually how many sysadmins first encounter git - by needing to download something from Github or a remote server. Let’s look and see what git knows about this repo by doing the following:

1
2
3

$ git remote -v
origin  git://github.com/unixorn/luggage.git (fetch)
origin  git://github.com/unixorn/luggage.git (push)

This command will list all the remote repositories that we’re tracking. By default, you will get a remote by the name of ‘origin’ that lists the location where you originally cloned the repository (in our case, this is Github).

Why is it listed twice? Well, with git, you can pull down, or fetch, changes from a remote repository as well as push up changes from your LOCAL repository to the remote repository. Frequently these paths will be the same, but in our case these links are to the READ-ONLY version of this repository, so trying to push changes will always fail because we don’t have access. We’ll talk later about how you can push up changes, but for now let’s create our OWN repository instead of working with someone ELSE’S.

Creating your OWN git repo

The Luggage has a directory called Examples, but it’s absent of any actual examples. We want to create our own directory where we can store our Luggage Examples. For the purposes of this demo, I’m going to create a directory called luggage_examples that will contain our…well… Luggage examples:

1
2

$ cd ~/src
$ mkdir luggage_examples

Now that we have a directory, we need to make sure that git is tracking our changes so that we can create those ‘what if’ scenarios and save our work.

1
2
3

$ cd ~/src/luggage_examples
$ git init
Initialized empty Git repository in /Users/gary/src/luggage_examples/.git/

Git created a new empty repository at ~/src/luggage_examples and created a .git directory inside luggage_examples. This .git directory is where git stores all its dynamic data that’s pertinent to your repository, so MAKE SURE not to delete it or modify any files in there (without knowing what you’re doing).

That’s it! We’ve created a repository! Now let’s actually see how git works locally on your machine.

Working with Files

As I mentioned before, you don’t NEED Github to use git. Git functions as a mechanism to ‘version’ your files so you can rollback to a previous version or create our ‘what-if’ scenarios and then accept (merge) or deny (destroy) them depending on our needs. You can easily create a git repository on your local computer ONLY, but of course you lose the ability to restore your work if your hard drive malfunctions. This is what Github gives us, and we’ll look at that functionality later.

The example I’m going to use is based on the previous blog post I wrote on Using the Google Mac Sysops Crankd Code Let’s create a directory for this example and create a starter Makefile:

1
2
3

$ mkdir -p ~/src/luggage_examples/crankd_google
$ cd ~/src/luggage_examples/crankd_google
$ vim Makefile

I use vi or vim to edit files from the Terminal. Feel free to use any text editor you like (nano, emacs, Textmate, TextWrangler, etc…), but CREATE a file called ‘Makefile’ (note the capital M) and put it in ~/src/luggage_examples/crankd_google. Here’s what I’ll put into this file:

1
2
3
4
5
6
7
8
9

    # Title:       Crankd-Google Example
    # Author:      Gary Larizza
    # Description: Something so Marnin doesn't kill me w/ postinstalls :)

    include /Users/gary/src/luggage/luggage.make

    TITLE=Crankd-Google
    REVERSE_DOMAIN=com.googlecode
    PAYLOAD=pack-crankd

Great, so we have our skeleton of a file! The file is saved in our folder, but we haven’t actually told git to track the file (and so git is AWARE that the file EXISTS, but it isn’t tracking its changes…yet). This is a good time to explain the three states in which files can exist in a git repository.

The Three Stages of Files: Working Tree, Index, and Repository

Git has three ‘locations’ by which a file can exist (even though it technically exists, in our case, in ~/src/luggage_examples/crankd_google). These three locations are called the working tree, the index, and the repository

The Working Tree

When you put a file in a directory that’s being managed by git, but you haven’t told git to track the files changes, then this file is said to be in the working tree. Our file we just created, Makefile, exists in the working tree currently. Git knows that the file EXISTS, but it hasn’t saved the file’s ‘state’ so we can rollback the file if need be.

The Index

The index is also known as the cache or the staging area, and it’s a place to put files before you’re ready to make a ‘commit’ (which is something of a ‘milestone’ or a version). A commit is a representation of a point in time in the history of your git repository. Think of it as a snapshot of your repository - ‘this is how our files looked at this specific time’. With commits, you can always rollback to a specific ‘commit’ (or even advance to a future commit if you had rolled-back to an earlier commit). In order to rollback to a specific version of a file, though, you need to make a commit, and in order to make a commit you need to tell git what FILES will comprise our commit. As I mentioned, a commit is just a point in time…but it doesn’t HAVE to be a point in time for a SINGLE file. A commit can represent a specific point in time for a NUMBER of files all at once. To make life easier for everyone, though, if you make a commit comprised of multiple files, then you want to make sure that the changes to the multiple files all represent a single functional change (for example, say we wanted to change a file that existed in our crankd_google folder that was being copied into a package, but we ALSO had to change the Makefile to change WHERE this file was to be put in the final package. These two changes are related and comprise a single functional change, so you would want to make a commit with the changes to both files at once.).

Our Makefile exists in the working tree currently, but we want to put it in the index because we’re ready to make a commit (no, our Makefile functionally doesn’t DO anything yet, but we want to make a commit so we can rollback to this point in time should we mess up anything). Understand that the locations of the ‘working tree’ and ‘index’ are relative to git ONLY. Yes, this file lives in ~/src/luggage_examples, and it will CONTINUE to exist in that directory (as far as the OS X filesystem is concerned), but to git it currently exists in the working tree. How do we know that the file is in the working tree and NOT in the index? Use the git status command to show you that info:

1
2
3
4
5
6
7
8
9
10
11

$ git status

# On branch master
#
# Initial commit
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
# Makefile
nothing added to commit but untracked files present (use "git add" to track)

Anything listed under ‘Untracked files:’ is in the working tree. You can add a file to the index by doing the following:

1
2
3
4
5
6
7
8
9
10
11
12

$ git add Makefile
$ git status

# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
# new file:   Makefile
#

Great, the file is in the index! Notice that the file is listed under ‘Changes to be committed:’ meaning that it’s in the index but NOT YET committed (or, ‘in the repository’).

The Repository

The Repository is where your files exist once they’ve been made part of a commit. As I mentioned before, files must be committed before we can roll back their states (and their states can only be rolled back to individual commits). You can never rollback a file’s state UNLESS it’s tied to a commit. If you made a commit to a file last week and haven’t made another since, but wanted to revert a file to how it existed three days ago, you would be out of luck. Git ONLY rolls back to specific commits (this is why it’s important to make frequent commits). Before we make a commit, however, we need to make sure to identify ourself to git (if you haven’t previously set this up). The user.email and user.name settings are the most important settings as they will be used to trace back what code you committed. If you’re not sure of how git is setup, use the following command to list the git config settings:

1
2
3

$ git config -l
user.name=Gary Larizza
user.email=gary@puppetlabs.com

Should the user.name and user.email settings NOT be configured, you can configure them with the following commands:

1
2

$ git config user.name 'Your name'
$ git config user.email your@email.com

Now that you’ve been identified (so we can lay the blame on you in the future), let’s actually commit some code. The git commit command allows git to commit all staged files to the repository in one fell swoop.

Every commit has three things: the list of files and CONTENT of the files that should be committed, a SHA1 hash value representing the commit itself (essentially, a unique identifier for the commit so it can be tracked), and the commit message describing the what and why of the commit.

Creating a commit message seems insignificant but should NOT be taken lightly. Great commit messages usually consist of a title line containing no more than 60 characters followed by a paragraph describing the changes. Because many people may potentially need to trace your code and commits, it’s important to list how your code functioned previously, the reasoning behind the change, and what specifically was changed. Don’t skimp with your commit messages! Always assume that the next person to maintain your code owns many dangerous weapons and knows where you sleep, so do that person a favor and be as kind to them (in the form of great commit messages) as possible!

Let’s begin the process of making a commit by issuing the following command:

1

git commit

Executing git commit will drop you into the default text editor (which is vim by default - you can change this by running git config core.editor /path/to/your/editor before doing git commit) and show the following:

1
2
3
4
5
6
7
8
9
10
11

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
# new file:   Makefile
#

Note that everything that begins with a hash ( # ) is solely for YOUR benefit and will be stripped out of the commit message, so don’t worry about it being included. Here’s an example of a commit message I would use in this instance:

1
2
3
4
5

Initial commit for crankd_google code

Previously we didn't have an example for creating a package of the 
Google Mac Sysops code.  This commit adds the shell for the Makefile and
includes the luggage.make file in ~/src/luggage.

Great! When you save and close your editor, you should see something that looks like this:

1
2
3

[master (root-commit) f423ac6] Initial commit for crankd_google code
 1 files changed, 9 insertions(+), 0 deletions(-)
 create mode 100644 Makefile

This tells us that we made a commit on the master branch with a SHA1 hash ID beginning in f423ac6 being attributed to this commit. In most cases, the first 7 digits of the hash are all you need for uniqueness when you refer to this commit, and that’s usually all that git will provide you in THIS instance. YOUR SHA1 hash won’t match the hash provided above (hence a unique ID), so don’t panic if it doesn’t match up for you. And that’s it! We’ve done it! Now what?

Examining your Work

Git tracks the ENTIRE history of a repository, and thus will track any and every commit made. To display this history, run the following command:

1
2
3
4
5
6
7
8
9
10

$ git log
commit f423ac6e9548d93f53b15d1154723c5abcd69738
Author: Gary Larizza 
Date:   Sun Jan 22 23:27:31 2012 -0500

    Initial commit for crankd_google code

    Previously we didn't have an example for creating a package of the
    Google Mac Sysops code.  This commit adds the shell for the Makefile and
    includes the luggage.make file in ~/src/luggage.

By default, git log will show ALL information about ALL of your commits (including the full commit message). There’s a LARGE number of arguments that git log will accept that allows you to format the output for ANY purpose. One of the common lists of arguments that I like to employ is the following (long) command:

1
2

$ git log --pretty=format:'%C(yellow)%h%C(reset) %s %C(cyan)%cr%C(reset) %C(blue)%an%C(reset) %C(green)%d%C(reset)' --graph --date-order
* f423ac6 Initial commit for crankd_google code 10 minutes ago Gary Larizza  (HEAD, master)

This format both colorizes the output for readability and also shows a nice graph of commits as you do things like branching and merging. Here’s an example of this command (with line length chopped for brevity) being run on a repository with many commits:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

* 72a2fe0 (#5445) Create /var/lib/puppet in OS X Package 26 hours ago Gary Larizza  (HEAD, gary/bug/2.7.x/5445_crea
*   c6667c5 Merge pull request #324 from glarizza/bug/2.7.x/2273_launchd_dashw 12 days ago Daniel Pittman  (origin/
|\  
* \   bc6c642 Merge pull request #318 from glarizza/bug/2.7.x/11202_apple_rake 12 days ago Michael Stahnke 
|\ \  
| * | 6c14a28 Build a Rake task for building Apple Packages 12 days ago Gary Larizza  (gary/bug/2.7.x/11202_apple_r
* | |   5ec5657 Merge pull request #127 from adrienthebo/ticket/2.7.x/8341-prevent_duplicate_loading_of_facts 12 da
|\ \ \  
* \ \ \   450da6a Merge pull request #92 from duritong/tickets/2.7.x/1886 12 days ago Daniel Pittman 
|\ \ \ \  
| | | | * d5bef5e (#2773) Use launchctl load -w in launchd provider 12 days ago Gary Larizza  (gary/bug/2.7.x/2273_
* | | | |   1cb2b6a Merge branch 'ticket/2.7.x/11714-envpuppet' into 2.7.x 12 days ago Josh Cooper 
|\ \ \ \ \  
| |_|_|_|/  
|/| | | |   
| * | | | 5accc69 (#11714) Use **%~dp0** to resolve bat file's install directory 12 days ago Josh Cooper 
* | | | |   fe79537 Merge pull request #323 from glarizza/bug/2.7.x/fix_launchd_spec 12 days ago Jeff McCune 
|\ \ \ \ \  
| * | | | | c865a80 Clean up launchd spec tests 12 days ago Gary Larizza  (gary/bug/2.7.x/fix_launchd_spec, bug/2.7
|/ / / / /

As you can see, git log is powerful and handy to list out what’s happened to your repository. You’ll also notice that it lists WHO made the changes, which is handy if you have MANY people contributing to the same code base.

That’s it for now

Hey , what gives?! We haven’t really DONE anything! Git has a tremendous amount of power, and it’s important to know WHAT’S happening before I can progress and show you the power of git. This article is ALREADY quite long, so I’ve broken it up for ‘easier’ reading. Rest assured, in parts 2 and beyond we will modify files, create topic branches, merge topic branches, use Github (for backup AND to allow others to contribute to your project), and much more. Stay tuned for the following article(s) and as always, put any questions in the comments or email me directly!

On December 21st, the MacOps team at Google released a Googlecode repository containing a couple of scripts that they use for managing Macs. The scripts provided interface with crankd (available in the Pymacadmin code suite) and Facter, which are a couple of tools with which you MIGHT not be familiar.

I saw a couple of questions on the ##osx-server IRC channel about how these scripts work, and I decided to do a quick writeup on how you would implement the scripts in your environment. This is meant to be a walkthrough to getting their scripts up-and-running, but please READ EVERYTHING FIRST before implementing.

Prepare Crankd:

Crankd is a vital part of the Google MacOps puzzle. Crankd essentially executes Python code in response to System/Network events on your system. I have MUCH MORE information on crankd in a guide I posted a while back, but I’m going to give you the ‘quick version’ for those looking to get started quickly:

• Clone or download the Pymacadmin Github repo. If you have git installed you can accomplish that by doing the following from the command line:

1

git clone https://github.com/acdha/pymacadmin.git

If you DON’T have git on your machine just visit https://github.com/acdha/pymacadmin, click on the Downloads tab, and download the .tar.gz or .zip version of the repo. Next, double-click on the resultant file to expand it into a folder.

• Change to the pymacadmin folder and run the install-crankd.sh script which will install crankd.py into /usr/local/sbin and create /Library/Application Support/crankd Once you’ve changed to the pymacadmin source directory, run the following from the command line:

1

sudo ./install-crankd.sh

OR…

If you already have a Puppet environment configured, I’ve written a module that will install crankd for you. The upside is that this module should work out of the box, but the downside is that I’ve copied the source to the module’s files/ directory. Since Pymacadmin hasn’t been changed since 2009, though, this shouldn’t pose a big issue. Simply copy this module to your modulepath and declare it with the following line in your node declarations:

1

include crankd

Download the Google-macops source code

You can download the source by using subversion with the following command:

1

svn checkout http://google-macops.googlecode.com/svn/trunk/ google-macops-read-only

This will create a folder called ‘google-macops-read-only’ in the current directory.

Copy the Python crankd files

Change to the directory where you downloaded the Google-macops source code and you will notice two folders: crankd and facter. Change to the crankd directory and copy all of the python files to /Library/Application Support/crankd with the following command from the command line:

1

sudo cp *.py /Library/Application\ Support/crankd/

This copies two files into /Library/Application Support/crankd: ApplicationUsage.py and NSWorkspaceHandler.py. The ApplicationUsage.py file contains most of the magic. In a nutshell, it will create a SQLite database file called /var/db/application_usage.sqlite containing information on applications that are launched and quit (we will inspect this information later), and it will update the database any time an application is (you guessed it) launched or quit. Crankd provides the mechanism by which the code is run any time an application is launched/quit and these Python files actually update the database with the pertinent information.

Copy the provided crankd plist to /Library/Preferences

Changing to the crankd directory in the google-macops-read-only directory you should find a file called ‘com.googlecode.pymacadmin.crankd.plist’ which is a sample plist for utilizing Google’s code that tracks application usage via crankd. It’s up to you to merge these Python methods into your existing crankd setup, or utilize the plist they’ve provided. The plist essentially says that any time an application is LAUNCHED, you should call the ‘OnApplicationLaunch’ method in the NSWorkspaceHandler class, and any time an application is QUIT, you should call the ‘OnApplicationQuit’ method in the NSWorkspaceHandler class. Let’s accept these conditions and copy their plist into /Library/Preferences (assuming that we don’t have an existing crankd setup) with the following command:

1

cp com.googlecode.pymacadmin.crankd.plist /Library/Preferences

Test out crankd from the command line

With crankd.py installed, the Python methods copied to /Library/Application Support/crankd, and the com.googlecode.pymacadmin.crankd.plist plist copied to /Library/Preferences, we can finally test out the code to see how it works. Start crankd with the following command:

1

sudo /usr/local/sbin/crankd.py --config=/Library/Preferences/com.googlecode.pymacadmin.crankd.plist

After entering your password, you should see the following in your terminal:

1
2

INFO: Loading configuration from /Library/Preferences/com.googlecode.pymacadmin.crankd.plist
INFO: Listening for these NSWorkspace notifications:  NSWorkspaceWillLaunchApplicationNotification, NSWorkspaceDidTerminateApplicationNotification

Now, let’s launch a couple of applications and see how we track their usage through the sqlite database. Do this by starting Safari and TextEdit, and then closing them (note that you MUST start them fresh - if they’re already opened make sure to close them, then open them, and THEN close them again). You should see something that looks like the following in the terminal:

1
2
3
4

INFO: Application Launched: bundle_id: com.apple.Safari version: 6534.52.7 path: /Applications/Safari.app
INFO: Application Launched: bundle_id: com.apple.TextEdit version: 264 path: /Applications/TextEdit.app
INFO: Application Quit: bundle_id: com.apple.TextEdit version: 264 path: /Applications/TextEdit.app
INFO: Application Quit: bundle_id: com.apple.Safari version: 6534.52.7 path: /Applications/Safari.app

Finally, you can quit crankd from the terminal by holding the control button down and pressing the ‘c’ key on your keyboard.

Inspect the application_usage.sqlite database

If you’re proficient in SQL, feel free to browse through the contents of the database. To keep things simple, let’s download an app called SQLite Database Browser from the following link. Launch SQLite Database Browser, go to File -> Open Database, hold the Shift and Command buttons down on your keyboard and then press the ‘g’ button to bring up the ‘Go to the folder:’ dialog box, type in /var/db/application_usage.sqlite in the box and press return, and then finally click the open button to open the database. If you click the ‘Browse Data’ tab near the top of the window, you should be able to see the raw data in the database. It should list the last time (in epoch time), bundle id, version, path, and number of times that the application has been launched (note that you can use an online utility to convert from epoch time to regular time).

Inspect the Facter fact

If you’re not familiar with Facter, you should click on the link and check it out. Facter is an awesome binary designed to gather information about your system and report it back in the form of a ‘fact’, which is essentially a key => value pair (for example, the ‘operatingsystem’ fact will return a value of ‘Darwin’ on Macs running OS X or ‘RedHat’ on machines running RedHat Linux). Google has provided a custom Facter fact that will report a fact NAME that corresponds with an application you’ve launched on your system (such as ‘app_lastrun_com.apple.safari’ or ‘app_lastrun_com.apple.textedit’) and a VALUE with the epoch time or the last time that application was run. The output will ultimately look like this:

1
2

app_lastrun_com.apple.safari => 1325203354
app_lastrun_com.apple.textedit => 1325203357

NOTE: This Facter fact requires the SQLite database that is created by the crankd code from above, so make sure you’ve followed the previous steps or you might have problems getting fact values.

We’ll need to install Facter. You can download a copy of Facter from Puppet Labs directly. Simply download the DMG, expand it, and run the package installer inside.

To test out the fact, we will need the full path to the folder containing the google-macops code we downloaded in step two above. I downloaded the source to a path of /Users/gary/Desktop/google-macops-read-only and so that’s the path I’m going to use in the next step.

Facter is designed to work with Puppet, but you can also run it independently. Facter is ALSO designed to be plug-able with custom facts, and that’s what Google has shipped us. In order to TEST these custom facts, we need to set the RUBYLIB environment variable (as Facter will look for a custom facts inside a ‘facter’ directory that’s located inside the path returned by the RUBYLIB environment variable). RUBYLIB should be set to the path of the downloaded google-macops source, so in my case I would run the following from the command line:

1

export RUBYLIB=/Users/gary/Desktop/google-macops-read-only

NOTE: make sure RUBYLIB is in capital letters. If you downloaded the google-macops source to a different path, substitute your path for ‘/Users/gary/Desktop/google-macops-read-only’.

Next, we need to make sure we have the ‘sqlite3’ ruby library installed on our system as the custom fact requires this library for correct operation. One way to install sqlite3 is by using Rubygems with the following command from the command line:

1

sudo gem install sqlite3

NOTE: If you have troubles using Rubygems to install sqlite3, you can either build it from source (which is beyond the scope of this article), or you can use Macports (if you have it installed) to install it with the following command:

1

sudo port install sqlite3

The last step is to actually run Facter to test out the fact. You may have a problem TESTING this fact if you installed sqlite3 via Rubygems - the reason is because the custom fact doesn’t actually load Rubygems before it tries to load sqlite3 (this isn’t a problem if you use the fact with Puppet as Puppet loads Rubygems. It’s ALSO not a problem if you’re running version 1.9 or greater of ruby, as you no longer need to load Rubygems BEFORE trying to include a ruby library with ruby 1.9 or greater). To remedy this problem, let’s add a single line to the custom fact. Notice the line that says the following:

1

require 'sqlite3'

This line is where the custom fact loads the sqlite3 library. Now, the line BEFORE this line we need to add a single line:

1

require 'rubygems'

This line will load Rubygems so that it knows where to access the sqlite3 library. Note that this line is unnecessary if you use this custom fact with Puppet as, like I mentioned before, Puppet already loads Rubygems. For the purpose of our testing, though, we will need this line.

Finally, let’s test the fact by running Facter from the command line:

1

facter

You SHOULD get a list of ALL your Facter facts, but up at the top you should have a couple of facts that begin with ‘app_lastrun_’ and then end with the bundle_id of the application that was launched. These are the custom facts that have been created.

You’ll note that the time reported is in epoch time. If you’d prefer a date/time string that’s more legible, you’ll need to modify the apps.rb file in the facter directory. Line 39 in the file (or, line 40 if you added the require ‘rubygems’ line previously) should say ‘appdate’ but change it to say the following:

1

Time.at(appdate.to_i)

Running facter again (like above) will give you a more legible date:

1
2

app_lastrun_com.apple.safari => Thu Dec 29 19:02:34 -0500 2011
app_lastrun_com.apple.textedit => Thu Dec 29 19:02:37 -0500 2011

???

Profit!

Now that you know how the code works, you’re free to modify, extend, and improve it as you see fit! The one thing that’s missing here is a LaunchDaemon that keeps crankd running in the background. Since you’re maintaining your environment, I’ll leave that up to you (though I walk you through this step in ‘Step 5. A LaunchDaemon to keep crankd running’ in my previous crankd writeup). Feel free to check out my other writeups on Facter, Puppet, and Crankd to see how I used to manage hundreds of Macs in an educational environment!

As of version 2.6, Puppet introduced a feature called “Run Stages” that will allow you to better control the order that Puppet configures your system. The problem with Run Stages (as of right now) is that there’s not that much good Documentation out there. Hopefully this document helps someone else out there who wants to setup Run Stages in their own environment.

A NOTE ON SYNTAX!!!

One of THE MOST DIFFICULT concepts to understand for puppet newbies is that these two things are identical:

1
2

include foo
class { 'foo': }

Both of these lines INVOKE the ‘foo’ class for use on a particular node. The problem is that the second method (the parameterized class method) looks nearly IDENTICAL to the method you would use to DECLARE the foo class in foo.pp. The only difference is that the class name is now INSIDE the curly braces (versus being OUTSIDE the braces when you DECLARE the class in your foo.pp file). It is VERY important that you distinguish the difference between DECLARING a class and using a parameterized class to INCLUDE the class in your site.pp file. I will point this out later too…

A Linux Example - Repositories

I would bet that the resource with the most “require” and “before” dependencies in the Linux world (Well, in the RHEL world anyways) would be the yumrepo. I’m sure most of us have many declared yumrepo resources in our manifests that each have their own tangled web of dependencies. My goal was to create a “repo” class that would have all of my repositories and use a Run Stage to ensure that my repo class was installed prior to any other package installs. Let’s look at some code:

1
2
3
4
5
6
7
8

class general::repo {
  yumrepo { 'huronrepo':
    descr    => 'Huron Repository',
    enabled  =>  '1',
    gpgcheck =>  '0',
    baseurl  =>  'http://10.13.0.6/huronrepo',
  }
}

This is my repo subclass based on my general base class. I have a single repository, but could easily have another 5 or 10 as need arises (I’m keeping things simple for demo purposes). Let’s look at another class:

1
2
3
4
5
6
7
8
9

class general::centos {
  include general::repo

  package { 'mcollective':
    name    => "mcollective",
    ensure  => 'present',
    require => Package['stomp'],
  }
}

Here’s a centos subclass off of the general base class. We’re including our repo class and declaring a single package that requires our “huronrepo” yumrepo. If we only needed to install a single package, we could use the “require” parameter; but if you assume that we will eventually need to install MULTIPLE packages, then this logic doesn’t scale. Just to keep things consistent, here’s my general base class:

1
2
3
4
5
6

class general {
  case $::operatingsystem {
    'CentOS': { include general::centos }
    'Darwin': { include general::osx }
  }
}

You can see that we include the general::centos class as long as Puppet is running on a CentOS box. There’s one final piece to this puzzle - it’s the actual declaration and assignment of the Run Stages. I’m doing this in my site.pp file. Here’s a snippit of what’s in my site.pp file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# /etc/puppet/manifests/site.pp

Exec { path => "/usr/bin:/usr/sbin:/bin:/sbin" }

# Run Stages
stage { 'pre':
  before => Stage["main"],
}

# Node Declaration
node 'server.whomever.com' {
  class { 'general::repos':
    stage => 'pre'
  }
}

Here’s where we’ve declared a “pre” stage that needs to run before the “main” stage. We don’t have to declare the “main” stage because Puppet uses that stage by default. Next, we’re including the general::repos class inside our ‘server.whomever.com’ node declaration and assigning it to the “pre” stage (which means that everything in that class will get configured prior to our “main” stage). Remember that you can declare as many stages as you need and chain them all to setup the order that you want, but that can become very unmanageable very quickly. I find it ideal to setup a ‘pre’ stage for repo setup, and then setting up dependencies within class files.

The World is Your Stage

This might not be the “best” way to use Run Stages, but it works for me. Hopefully this little writeup cements the concept for you too.

With the current movement in the Mac Community toward modular imaging strategies, there’s a spike in the need for properly formed package installers. Apple’s package format is well documented for its benefits and flaws and there are a string of applications that will help you create your perfect package (from Apple’s Packagemaker to the many third-party applications). While all the various package creation applications will ultimately create a desirable package in the end, very few of them have the triple-threat of easy code-review, replication, and portability. This is where The Luggage comes in.

The Luggage is a packaging system based on GNU’s make command that is found on most every *nix-based operating system (OS X included). Joe Block, the author, open-sourced the project based on an internal Google tool for creating packages with Makefiles. By using Makefiles, every member of your team can quickly glance at the file and see exactly what is being inserted into a package. Changes can be made and committed quickly, errors can be squashed without tearing apart .pkg files, and you can reproduce packages quickly and efficiently without wading through many GUI windows.

Why do I need to Package?

Many vendors already ship properly formatted package installers for their software - and for that we thank them. There are still a couple of major software vendors that choose to use unique third-party package “wrappers” to install their software. While this is fine if you only ever need to install that software on one machine, it makes software deployment…difficult. Because of this, systems administrators need to re-package this software into a proper package installer that will deploy and install silently.

If you’re a fan of open-source software, you will find that many projects do not offer their software in Apple-friendly packages. The Luggage will help you wrap their source files into a package format that can be distributed to your machines.

Finally, you may have a whole bevy of scripts that you use for configuration/customization of your clients. These scripts can easily be deployed via ARD or wrapped into a payload-free package with a single postinstall script. The Luggage will help you keep track of all your scripts and package them for distribution.

As I’ve said before, there are other third-party applications out there that will create a package to your needs. Many will use snapshotting (or fseventsd monitoring) to create a package based on what’s changed on your system. While this is lightning fast (in most cases), you will need to redo this whole process if something needs to be changed in the resultant package.

How do we use The Luggage?

As we said before, The Luggage is just a giant Makefile. Make has its own unique language, so you need to obey its syntax and formatting standards. I’ve linked to the GNU make manual here so you can get a quick overview of how it works (WARNING, it’s quite large), but this guide will cover all the basics you need to know to get started.

Note that while it is NOT NECESSARY TO COMPLETELY UNDERSTAND MAKE TO BEGIN USING THE LUGGAGE, it will help you out tremendously as you start to create complicated packages if you DO understand make’s nuances. This article may be long, but that’s only to make sure that the reader understands what is going on in the background.

The luggage.make File

The base of everything you will do with The Luggage is a file called luggage.make. It is by no means definitive, and you’re encouraged to add to it should you encounter a situation where a recipe doesn’t exist for a directory into which you want to install files (Don’t worry, we’ll get into this later), but it does serve as the basis for all packages you’re going to be creating.

At the top of the file are all the variable declarations. Anything that begins with SOMETHING=somethingelse is setting a variable that we will encounter later. Many of these variables (such as PACKAGEMAKER, WORK_D, CP, INSTALL, and so on) are paths to various commands that we will need in our rules (setting absolute paths to common commands saves typing later and helps us avoid errors with things like PATH environment variables). Dereferencing these variables in a Makefile is done with a syntax that looks like this -> ${CP} (this outputs the value of the CP variable, which is actually /bin/cp). Note that you DO NOT use quotes when you set a variable (i.e. if you want to set the path to CP you do it with CP=/bin/cp and NOT by doing CP=”/bin/cp”) - if you DO use quotes, they will be included in the value of that variable (which will cause you all kinds of problems).

Next, we have the target stanzas. A target (or a rule - I will use the word rule throughout the article) is setup to look like this:

1
2
3

        o-something: l_usr_bin
        @-echo "Commands are entered here"
        @-echo "Everything below our rule is executed"

In this case, the target, or rule, is called do-something and has dependencies based on ANOTHER rule that’s called l_usr_bin. Below this line is called our recipe, and right now there are two echo commands. If the above code was in a blank text file called Makefile we could execute the two echo commands by running the following command from the command line:

1

    make do-something

This would in turn echo the two lines of text below the do-something rule (This is not totally true - since we haven’t defined a rule for l_usr_bin it would probably error out, but I’ve kept that dependency in the above example to show you how a rule works.). Looking specifically at luggage.make, the target stanzas define the behavior for The Luggage’s various behaviors (make pkg, make dmg, make zip, and so on). Note that since many of these rules have dependencies on OTHER rules, a simple command of make pkg will trigger.

Next, you will encounter the Target directory rules. This is the part of the luggage.make file that you may need to edit. Joe has done a great job of defining the most popular directories into which you will install files/applications/etc. but it would be impossible for him to define EVERY location that you could possibly need. Here is the structure of creating a Directory Rule:

1
2
3
4

        l_etc_puppet: l_etc
        @sudo mkdir -p ${WORK_D}/etc/puppet
        @sudo chown -R root:wheel ${WORK_D}/etc/puppet
        @sudo chmod -R 755 ${WORK_D}/etc/puppet

This rule will create an /etc/puppet directory in Luggage’s working directory (The variable WORK_D will be used frequently - it’s the temporary directory that The Luggage creates to simulate the target volume onto which it’s installing files. Everything following ${WORK_D} will be the EXACT PATH into which files will be installed on the machine that runs your package.) These Directory Rules become very important when we create custom Makefiles because they serve as dependencies which create the directories in The Luggage’s Working Directory. In a nutshell, if you’re installing files into a directory with The Luggage, you need to have a Directory Rule that creates this directory first. Bottom Line: if you don’t FIND a Directory Rule for a directory where you will be installing files, then you’ll need to create one.

Finally are the File Packaging Rules. These are handy shortcut rules that will keep your makefiles very short and readable. In a nutshell, Joe has defined some of the most common directories to which files are installed and created one-line commands that will install specific files into those directories. For example, say you were creating a custom Makefile in a directory that also had a launchd plist called com.foo.bar.plist in it. If ALL you needed to do was create a package that installed that launchd plist into the /Library/LaunchDaemons directory you could setup your Makefile like this:

1
2
3
4
5

include /usr/local/share/luggage/luggage.make

TITLE=install_foo_launchd_plist
REVERSE_DOMAIN=com.foo
PAYLOAD=pack-Library-LaunchDaemons-com.foo.bar.plist

The PAYLOAD variable tells The Luggage which rules to execute. In this case, it’s executing the File Packaging Rule for launchd plists (pack-Library-LaunchDaemons-%) that creates the ${WORK_D}/Library/LaunchDaemons directory, installs the com.foo.bar.plist file into it, sets the correct permissions, and then executes the Packagemaker command line tool to create your package installer. Congratulations, you created a package in 5 lines of code!

Preparing for using The Luggage

The Luggage is fairly self contained, but it DOES use Apple’s Packagemaker command line tool - and the way to install THAT is to download and install the Developer Tools for your computer’s OS version (There are different Developer Tools packages for 10.6, 10.5, and so on). The Developer Tools can be downloaded from Apple’s Developer Site, but you must create a (free) developer account first. If you’re a Mac sysadmin, you should already have all of this.

Once the Developer Tools have been installed you will then need to install The Luggage. You can use The Luggage to create an installer package for The Luggage (Weird, yes) by first downloading the source code from Github. Just click on the big Downloads button in the upper right corner of the screen and download a zip version of the files. From there, double-click on the downloaded zip file to open it, open up Terminal and change to the folder that was created which contains The Luggage’s source code, and execute the following command:

1

    sudo make pkg

This will create an installer package which can be run. This package installs The Luggage into the /usr/local/share/luggage directory (Don’t believe me? Check the Makefile for your self!). You’re now ready to use The Luggage!

Before we get into the package creation examples, we need to talk about directory structure and version control systems (svn, hg, git, etc…). When you create new packages with The Luggage, it’s necessary to create a new Makefile in a new directory. I like to use the /usr/local/share/luggage/examples directory to create my packages. The easiest way to begin a new package is to copy a directory that contains a working Makefile and simply tailor it to your needs. Unless your job is SOLELY packaging (or you’re a unix graybeard), you’re NOT going to remember all the syntax and nuances of make. Don’t re-create the wheel - just copy and edit.

Next, you’re going to want to backup your files and/or have a versioning system. Version control systems (like Subversion, Mercurial, Git, and the like) are becoming increasingly popular with the current DevOps movement, so it might be a good idea to start playing with one NOW while you’re learning a new skill! If you’re TOTALLY new to this concept, I recommend using git, but it’s entirely up to you. I maintain a git repository of my Luggage fork that’s open to anyone to review and borrow. If you decide to use something like Git, well then GOOD ON YA, MAN! If not, make sure you have a backup of your Makefiles. They’re small files; make two backups :)

Package Creation Examples

Now that we’ve described how make works, how luggage.make works, given a quick example of how to create a package in 5 lines, and had the installation/backup talk let’s walk you through some basic package creation examples. We’ll create packages that install an application, create a package that installs printers and executes a postinstall script, create a package that installs source code in many directories, and finally create a complex Makefile that uses variables and bash commands.

An Application Package

The most common packages will simply install an application file into the /Applications folder on your computer. Since .app files are actually bundles (a directory that contains subdirectories of executable code), we will need to use tar to wrap these bundles into a single file. From there we can have The Luggage untar the application into the correct folder. Joe’s Firefox example does this, but he also has an added step of curling the tarred/bzipped application from a web server. I’m going to skip that step (for now) so you understand how the basic process works, but it’s a best practice to use the curl method so you can keep your applications up-to-date on your fileserver without having to copy the new files to your luggage directory every time.

Let’s first create the /usr/local/share/luggage/examples/calculator directory and then make a copy of /Applications/Calculator.app into the /usr/local/share/luggage/examples/calculator directory. Next, lets open Terminal and change to our /usr/local/share/luggage/examples/calculator directory. Finally, run the following command to tar.bz2 our Calculator.app:

1

    tar cvfj Calculator.app.tar.bz2 Calculator.app

If you did it correctly, it should create the Calculator.app.tar.bz2 file that we need for The Luggage. Make sure this file is in the SAME directory (and level - so don’t create subfolders) as the Makefile we’re going to create. You can actually delete the Calculator.app copy that we created - we won’t need it. From there, our Makefile is simple - it should look like this:

1
2
3
4
5

include /usr/local/share/luggage/luggage.make

TITLE=Calculator_Install
REVERSE_DOMAIN=com.yourdomain
PAYLOAD=unbz2-applications-Calculator.app

That’s it! The PAYLOAD variable is the magic variable here - it contains the rule(s) that are to be executed in creating our package. The unbz2-applications-% File Packaging Rule is defined in our luggage.make file (which we’ve included at the top of our Makefile), so we don’t NEED anything else in our Makefile. MAKE SURE that the capitalization and spelling of Calculator.app in “unbz2-applications-Calculator.app” and your “Calculator.app.tar.bz2” files are IDENTICAL or this package will NOT be created (make relies on this).

UPDATED: An easier Application Package

Joe has actually created a Ruby script that will perform all of the actions outlined in the previous example for you. It’s called app2luggage.rb and it can save you a few steps if you know how to use it. Let’s take a look and see what we need to use it.

The first thing you will need to do is install the trollop rubygem as that’s what app2luggage.rb uses to parse its options. You can do this with the following command:

1

    sudo gem install trollop

Next, make sure the app2luggage.rb script exists in your /usr/local/share/luggage directory. Finally, let’s run the command with the following arguments:

1

    sudo /usr/local/share/luggage/app2luggage.rb -a /Applications/Calculator.app -i /usr/local/share/luggage/examples/Calculator_Application/ -l /usr/local/share/luggage/luggage.make -r com.huronhs -p Calculator_Application

Here’s what each argument means:

1. -a is the path to the Application we want to tar up and install with The Luggage - we’re using the path to Calculator.app for now
2. -i is the path to the folder that app2luggage.rb will create that contains our Makefile and tarred up application. This folder SHOULD NOT EXIST or app2luggage.rb will exit (so as not to overwrite your data).
3. -l is the path to our luggage.make file
4. -r is the reverse domain for our organization
5. -p is the package id for the Makefile

There are other arguments available, simply run app2luggage.rb with the –help argument to see them all. Once app2luggage.rb runs successfully, it will create the directory you specified in the -i argument and populate it with a Makefile and the tarred up application. The only thing left to do is to make your pkg, zip, or dmg.

Installing a printer with a preinstall script

One of the most popular solutions I offer with using The Luggage is to create a package that will install a printer and then install a pre-configured PPD file that sets all the initial settings for the printer. I do this with Puppet currently, and I know it’s popular in Munki too. You can also optionally install a specific driver (if the drivers for your printer aren’t already on your machines). For those people who like to skip ahead, I have this example on my luggage repo.

This Makefile demonstrates the use of a preinstall/preflight script (which isn’t actually installed into the payload of the package). The Luggage has a special packaging rule for this called pack-script-% that works so long as the name of your script corresponds exactly with what you write in the PAYLOAD variable. This file also demonstrates the use of multiple rules in the PAYLOAD variable by using the \ character. While this isn’t difficult to understand, it is a necessary syntax.

My preflight script is right here for those who are interested. It’s copied into the directory we create that contains our makefile and I name it preflight. Note that it contains variables that need to be changed (all of which are at the top of the script) before you deploy this package.

Next, we need to get a .ppd file that contains all the configuration data for our printer. The easiest way to do this is to install your printer on a demo machine using the EXACT SAME SETTINGS that will be configured in your script (protip: actually RUN the script on your computer first), configure it how you want (number of trays, memory settings, type of finisher, etc…), and then open the /etc/cups/ppd directory on your model computer. Inside should be a .ppd file for your printer containing all the settings you’ve just configured. Copy this .ppd file into the folder that contains your Makefile and preflight script. Mine (in this example) will be called psm_HHS_Office_9050.ppd

Now, let’s look at our Makefile:

1
2
3
4
5
6
7
8
9
10
11
12
13

    include /usr/local/share/luggage/luggage.make

    TITLE=HHS_Main_Office_9050_Managed_Installer
    REVERSE_DOMAIN=com.huronhs
    PAYLOAD=\
        pack-hp-ppd \
        pack-script-preflight

    pack-hp-ppd: l_etc_cups_ppd
      @sudo ${CP} ./psm_HHS_Office_9050.ppd ${WORK_D}/etc/cups/ppd/psm_HHS_Office_9050.ppd
      @sudo chmod 644 ${WORK_D}/etc/cups/ppd/psm_HHS_Office_9050.ppd
      @sudo chown root:_lp ${WORK_D}/etc/cups/ppd/psm_HHS_Office_9050.ppd

The first thing you should notice is that our PAYLOAD variable starts with a \ and has three lines. The \ signifies that the contents of our variable spans multiple lines. In reality, the value of PAYLOAD is actually “pack-hp-ppd pack-script-preflight” but it’s formatted so it’s easier to read. Next, notice that the pack-script-preflight rule contains the word “preflight” after pack-script-. This means that our script must be named preflight (EXACTLY - case is sensitive here). Finally, we’ve also specified a pack-hp-ppd rule. Since luggage.make DOES NOT define this rule, we’ve defined it in our Makefile.

The pack-hp-ppd rule has a dependency on the l_etc_cups_ppd rule. This rule IS defined in luggage.make (well, it is for me - I can’t remember if I created it or not. If it isn’t there for you, then you’ll need to create it using the other folder creation rules as a guide) - and it creates the /etc/cups/ppd folder structure that we need in our package. Lets look at the three lines which are called our recipe*:

1
2
3

@sudo ${CP} ./psm_HHS_Office_9050.ppd ${WORK_D}/etc/cups/ppd/psm_HHS_Office_9050.ppd
@sudo chmod 644 ${WORK_D}/etc/cups/ppd/psm_HHS_Office_9050.ppd
@sudo chown root:_lp ${WORK_D}/etc/cups/ppd/psm_HHS_Office_9050.ppd

The first line references the CP variable (if you remember - its value is /bin/cp) to copy the psm_HHS_Office_9050.ppd file from the directory that contains our Makefile into Luggage’s working directory/etc/cups/ppd. The second line sets its mode to 644 (to match the mode for all .ppd files in that directory), and the third line sets the owner to root and the group to _lp (note that this group is ONLY available in 10.5 and 10.6 machines - so you’ll need another package for 10.4 clients or below).

That’s it! The resultant package will run the preflight script and then install the correct .ppd file after the script is run. This package will successfully install and configure your printer in one fell swoop.

Installing files into multiple directories

So far our Makefiles have been pretty easy. In fact, most of them have been a couple of lines. Lets look at a package that installs files into multiple locations. There’s a cool open source tool called Angelia that was created by R.I Pienaar. I use it to send alerts to my iPhone and to also process Nagios Alerts. The problem is that it’s only packaged for Linux machines. Since I manage Mac machines, I thought I’d lend R.I. a hand and create a package installer for it. This package is also on my luggage repo if you want to work ahead. Let’s look at the Makefile:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

    #  Angelia Package Creation:

    include /usr/local/share/luggage/luggage.make

    TITLE=Angelia_Installer
    REVERSE_DOMAIN=com.huronhs
    PAYLOAD=\
        pack-angelia-etc \
        pack-angelia-binaries \
        pack-angelia-lib \
        pack-angelia-spool \
        pack-angelia-log \
        pack-angelia-launchd


    pack-angelia-launchd: l_Library_LaunchDaemons
      @sudo ${CP} net.devco.angelia.plist ${WORK_D}/Library/LaunchDaemons
      @sudo chmod -R 644 ${WORK_D}/Library/LaunchDaemons

    pack-angelia-binaries: l_usr_sbin
      @sudo ${CP} ./angelia-send.rb ${WORK_D}/usr/sbin/angelia-send
      @sudo ${CP} ./angelia-spoold.rb ${WORK_D}/usr/sbin/angelia-spoold
      @sudo ${CP} ./angelia-nagios-send.rb ${WORK_D}/usr/sbin/angelia-nagios-send
      @sudo chmod -R 755 ${WORK_D}/usr/sbin

    pack-angelia-lib: l_Library_Ruby_Site_1_8
      @sudo ${CP} -R ./angelia ${WORK_D}/Library/Ruby/Site/1.8
      @sudo ${CP} ./angelia.rb ${WORK_D}/Library/Ruby/Site/1.8
      @sudo chmod -R 755 ${WORK_D}/Library/Ruby/Site/1.8

    pack-angelia-etc: l_etc_angelia
      @sudo ${CP} -R ./templates ${WORK_D}/etc/angelia
      @sudo ${CP} ./angelia.cfg ${WORK_D}/etc/angelia
      @sudo ${CP} ./COPYING ${WORK_D}/etc/angelia
      @sudo ${CP} ./README.markdown ${WORK_D}/etc/angelia
      @sudo chmod -R 755 ${WORK_D}/etc/angelia

    pack-angelia-log: l_var_log_angelia
      @sudo touch .create
    pack-angelia-spool: l_var_spool_angelia
      @sudo touch .create

It’s definitely longer, but I don’t think it’s any more complex than the examples we’ve seen before. The PAYLOAD variable spans multiple lines using the \ character and I’ve defined all the custom rules. All of the rules depend on folder rules in my luggage.make file (which were created if they didn’t exist before), and the recipes inside those rules are extremely simple to navigate (mainly copying files and changing permissions). Creating a script that makes this package would be QUITE A BIT longer than this Makefile, and the Makefile itself took less than 3 minutes to create. The only caveat is that we need to make sure all the files/directories we’re copying into Luggage’s WORK_D directory are in the folder that contains the Makefile. Because of this, anytime Angelia is updated I will need to copy over new files.

It would be much easier to create a Makefile that downloaded the current version of all of these files before it copied them into our package…

Creating a more complex Makefile

The basis of this example was my thought that it would be MUCH easier for me to have a Makefile that downloaded the newest versions of the files that I need before it copies them into the package. Make supports this through its support of shell commands and variables - it only requires you to code it up. Let’s look at a package I created for Marionette-Collective.

Marionette-Collective is awesome software (also created by R.I Pienaar and now owned by Puppet Labs) that allows you to communicate with multiple nodes at once. They DID have a script that created their packages, but I wanted to create a Makefile that would create a package of ANY version of their software. Let’s look at the Makefile first and I’ll break it down:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

    #  Example:  A dynamic installer for Marionette-Collective
    #
    #  Author:  Gary Larizza
    #  Created: 12/17/2010
    #  Last Modified: 12/18/2010
    #
    #  Description:  This Makefile will download the version of MCollective specified
    #    in the PACKAGE_VERSION variable from the Puppet Labs website, untar it,
    #    and then install the source files into their Mac-specific locations.  
    #    The MAJOR and MINOR versions must be specified for the Info.plist file
    #    that Packagemaker requires, but I use awk on the PACKAGE_VERSION to 
    #    get these.  See inline comments.
    #
    include /usr/local/share/luggage/luggage.make


    # Luggage Variables:
    #    If the TYPE variable isn't specified via the CLI, we will install everything
    #    into the resultant package
    TITLE=MCollective_Installer_Full
    REVERSE_DOMAIN=com.puppetlabs
    PAYLOAD=\
        unpack-mc-${MCFILE} \
        pack-mc-libexec \
        pack-mc-binaries \
        pack-mc-lib \
        pack-mc-config \
        pack-mc-config-server \
        pack-mc-config-client \
        pack-mc-launchd \
        pack-mc-mcollectived \
        pack-mc-preflight-all

    # Variable Declarations:  
    #    Any variable can be set from the command line by doing this:
    #    "make pkg PACKAGE_VERSION=1.0.0"
    PACKAGE_VERSION=1.0.0
    PACKAGE_MAJOR_VERSION=`echo ${PACKAGE_VERSION} | awk -F '.' '{print $$1}'`
    PACKAGE_MINOR_VERSION=`echo ${PACKAGE_VERSION} | awk -F '.' '{print $$2$$3}'`
    MCFILE=mcollective-${PACKAGE_VERSION}
    MCURL=http://puppetlabs.com/downloads/mcollective/${MCFILE}.tgz

    # Package Creation Limiters:
    #    These if-statements will check for one of three values for the TYPE variable:
    #    "COMMON, CLIENT, or BASE"  If either of these values are present (CASE SENSITIVE)
    #    the PAYLOAD variable will be changed to limit what is installed into the package.

    # COMMON Package:
    #    This package includes the Ruby libraries and MCollective plugins with nothing else.
    ifeq (${TYPE},COMMON)
      PAYLOAD=\
          unpack-mc-${MCFILE} \
          pack-mc-libexec \
          pack-mc-lib \
          pack-mc-preflight-common
      TITLE=MCollective_Installer_Common
    endif

    # CLIENT Package:
    #    This package includes the MCollective Binaries and the configuration file for
    #    MCollective's client binaries.  
    ifeq (${TYPE},CLIENT)
      PAYLOAD=\
          unpack-mc-${MCFILE} \
          pack-mc-config \
          pack-mc-binaries \
          pack-mc-config-client \
          pack-mc-preflight-client
      TITLE=MCollective_Installer_Client
    endif

    # BASE Package:
    #    This package includes the mcollectived daemon, Ruby Libraries, a launchd plist
    #    to call mcollectived, and the configuration files for the MCollective server.
    ifeq (${TYPE},BASE)
      PAYLOAD=\
          unpack-mc-${MCFILE} \
          pack-mc-config \
          pack-mc-config-server \
          pack-mc-launchd \
          pack-mc-mcollectived \
          pack-mc-lib \
          pack-mc-preflight-base
      TITLE=MCollective_Installer_Base
    endif

    # This rule will curl the selected version of MCollective and untar it into the directory
    #    in which the Makefile resides.
    unpack-mc-${MCFILE}:
      curl ${MCURL} -o ${MCFILE}.tgz
      @sudo ${TAR} xzf ${MCFILE}.tgz

    # This rule will install MCollective's plugin files to /usr/libexec/mcollective