1. Introduction
2. A sample deployment
3. Variations
4. Distribution
5. Conclusions

Introduction

Systems management has received little attention in the Linux world so far: we have very good tools to manage packages, make package installation and update as painless as possible, and provision systems to a specified state. But to truly deserve the label 'systems management', a lot more is needed: managing packages is only a small part of maintaining systems. Much attention has been focused on individual systems, though individual systems, especially with the advent of virtualization, become a smaller and smaller piece of the puzzle.

Linux systems are usually not deployed for a standalone purpose; instead they are part of a larger service that is provided by one or more systems. A very typical example are web applications: in the simplest case, all the components of the web application, webserver, database, and the application itself are deployed on a single system. As the application grows in popularity, the deployment changes from a single system to a number of interdependent and interoperating systems, usually made up of a load balancer that distributes the work across a number of webservers, who in turn talk to a backend database that may consist of a single database server or a server cluster, whose storage may be provided by networked storage.

All these machines need to be managed, and they need to be managed in a way that lets humans understand the way in which they interoperate. At the same time, configuring the various components requires specialized knowledge: the number of people that can maintain an Oracle cluster, administer a storage array, configure an F5 load balancer, tune the base operating system for optimal performance, and replace light bulbs in the data center is very small. So when your boss comes to you and asks you to install and maintain such an application, you will most likely tell him that you can change the light bulbs right away, find some of the other information on Google, and recommend hiring an expensive consulting firm for the rest.

What you will find on Google, and what the overpriced consultants will give you are a mixed bag of scripts, and a large number of HOWTO documents that you have to carefully read and follow; and once you have done the initial setup, you will need to try and keep up with better recommendations as Google happens to find them, carefully trying to remember why and how you set up things the way they are. With that in mind, you begin to flip-flop between simply quitting your job and telling your boss what you really think of him. And as you are pondering these options, you ask yourself why Oracle doesn't publish recommended configuration options in a way that lets you just download and install them somewhere in your network. And why can't Red Hat just give you a file that contains their recommendation for running an Oracle cluster on RHEL7 on your 4-way dual-core Opteron machine so that the machine understands what it is supposed to do, leaving you unencumbered with this knowledge?

The simple answer, of course, is that these are difficult problems. There's no wide-spread, standardized way to solve them: we don't have a way to communicate the recommended configuration,nor do we have a way to express what exactly you are trying to achieve. Without automation support, we alleviate these problems with whatever tools we happen to know or have written. This is similar to how software installation and upgrade was handled before package management tools like rpm and yum were widely used.

What you (or your boss, really) want for your web application cluster is a system recipe, a formalized description of all the things that need to be present and configured on the machines in your cluster. Since it is very unlikely that your application vendor will be able to give you the best recipe for your specific hardware and software combination, such a recipe must be composed from ingredients published by a variety of sources, and leave you free to tweak it to your liking: some parts will come from your OS vendor, others from the database vendor and yet others may be contributed by the community or the result of your internal testing and tweaking.

The recipe, to really work, has to touch every aspect of the systems under its control, from low-level driver information and file system mounts to the kinds of services that need to be running. To be used reliably, recipes need to do what they do in an automated way, and should help you recover from a catastrophe (say, one of your webservers drowning in Cherry Coke) with great ease; at the very least, you should be able to set up another webserver in exactly the same way and know that it's set up in the same way. This is starting to sound a lot like configuration management, your boss says, and so you start on a quest to figure out what he means by that.

The rest of this document walks through a simple example of what such a recipe could look like, and illustrates how a specific configuration management tool, Puppet, can be used to solve some of these problems. The example doesn't pretend to provide a complete implementation of system recipes, there are many more questions that need to be answered, but it should give you a good impression how Puppet can help you modularize the setup for a specific web application, and how these modules could be provided by a variety of sources, and made to work together.

A sample deployment

This section shows how Puppet can be used to describe the deployment and configuration of a web application. The application that was chosen for this purpose is Nag, a task list manager, part of the Horde framework. Nag is written in PHP and can work with a variety of relational databases.

Apart from putting the PHP scripts that make up Nag and Horde into the right place, a number of things need to be done before the application can function: most obviously, a webserver and a database need to be installed, the proper schema must be created in the database, the webserver and database configurations need to be modified to fit Nag/Horde's needs etc.

Puppet is controlled by a manifest, a set of instructions expressed in its own domain-specific declarative language. It is highly recommended that you read the brief Puppet language tutorial at this point. Before we run the Puppet manifests for Nag, we'll go through what they do and how they fit together; the manifest and accompanying configuration files for this example are available as a tarball. If Puppet were a standard component of every Fedora or RHEL machine the way rpm is today, the pieces of this manifest would come from a number of places, and not just one tarball.

The manifest

For this example, we will assume that Nag will be deployed to a single machine, with hostname nag.example.com. The section Variations explains how this simple setup can be extended to fit more complex requirements.

The toplevel manifest for this example is very simple:

import "yum.pp"
import "horde.pp"

node nag.example.com {
  include yum-rawhide
  include horde-db-pg
  include horde-nag
}

It simply states that the host nag is a member of the classes yum-rawhide, horde-db-pg, and horde-nag, all defined in the two files imported at the top. A class in this context is a set of configurations that are to be applied together. Puppet classes also capture the essential aspect of classes in object-oriented languages, and make it possible to override settings from a base class in a subclass. This feature, essential to successfully modularizing manifests, is not supported by the majority of configuration management tools and sets Puppet apart from them in this very important aspect.

The setup of Nag is structured in several steps, in accordance with the fact that we want to start with a basic OS install and control as much of the deployment and configuration of the application through the manifest as possible, from the yum repositories that need to be enabled on nag to the exact configuration of the application.

yum repositories

The first thing we need to make sure of is that nag uses the correct yum repositories for installing stock OS components like the webserver and the database server. Since Puppet does not yet support yum repositories natively, we do this by keeping config files for all the yum repositories on the file server on puppet and use a simple Puppet component to enable/disable them selectively. Enabling/disabling will be done by either placing the corresponding configuration file on the client, or deleting it from the client; this means that the yum configuration files on the file server all have to have the enabled parameter set to 1. The enabling/disabling is achieved by the yumrepo component defined in yum.pp:

define yumrepo (enabled = true) {
  configfile { "/etc/yum.repos.d/$name.repo":
    mode => 644,
    source => "/yum/repos/$name.repo",
    ensure => $enabled ? {
              true => file,
              default => absent
             }
  }
}

which can be used to enable the rawhide repository fedora-devel like this:

yumrepo { fedora-devel: enabled => true }

All types, elements natively built into Puppet, and components, groups of elements defined in the manifests, have a few implicit attributes, name being the most important of them. The above use of the yumrepo component executes the statements in the definition of yumrepo with the value of $name being fedora-devel and that of $enabled being true. The variable $name is used to determine both the path to the yum repo file on the client (the name of the configfile element), and on the file server (the value of the source parameter). The ensure parameter is used to either copy the file from the central file server, or to delete it on the client, depending on the value of $enabled.

You might notice that the definition of yumrepo doesn't really mention from which server the repository configuration files should be taken. That's because configfile is not a type built into Puppet, instead it's a convenience component defined in util.pp:

$server = "puppet.example.com"

# The same as a file, except that the source is always taken
# relative to the /config module on the file server
define configfile(owner = root, group = root, mode = 644, source, 
                  backup = false, recurse = false, ensure = file) {
    file { $name:
            mode => $mode,
            owner => $owner,
            group => $group,
            backup => $backup,
            recurse => $recurse,
            ensure => $ensure,
            source => "puppet://$server/config/$source"
    }
}

The file type is built into Puppet, and it spells out the source location for the file in great detail: use the Puppet file server (denoted by the puppet protocol) on the server puppet.example.com and retrieve the file /config/$source from it. With the changes we made to fileserver.conf earlier, the file server translates that to the path for the config module /var/puppet followed by whatever the $source variable passed to configfile contains.

The yumrepo component lets us define what it means to be a machine in the yum-rawhide class, again in yum.pp:

class yum-rawhide {
  yumrepo { 
    [fedora-devel, fedora-extras-devel]: 
      enabled => true;
    # Disable standard Fedora repositories that are configured
    # even on rawhide machines so they don't gunk up the works
    [fedora-extras, fedora, fedora-updates, fedora-updates-testing]: 
      enabled => false 
  }
}

The definition of yum-rawhide uses simplified syntax to declare several elements of the same type at once; the yumrepo entry declares six elements that use the yumrepo component, two with enabled set to true, and four with enabled set to false.

httpd

With the yum repositories configured properly, we can go and set up httpd, our webserver by declaring a httpd class:

class httpd {
  package { httpd:
    ensure => latest
  }

  configfile { "/etc/httpd/conf/httpd.conf":
    source => "/httpd/httpd.conf",
    mode => 644,
    require => package["httpd"]
  }

  group { apache: gid => 48 }

  user { apache: 
   comment => "Apache",
   uid => 48,
   gid => 48,
   home => "/var/www",
   shell => "/sbin/nologin"
  }

  service { httpd:
    running => true,
    subscribe => [ file["/etc/httpd/conf/httpd.conf"],
                   package["httpd"] ]
  }
}

The httpd class consists of several elements to ensure that the httpd package is installed, that a custom httpd.conf config file is deployed from our central file server, that the apache user and group are there, and that the httpd service is running. The declaration of the httpd package indicates that we always want the latest version of that package installed; Puppet will make sure that updates to the httpd packages are applied as they become available in the yum repositories.

The configfile in this declaration uses a require parameter, even though the definition of configfile does not declare one: all elements in Puppet have set of builtin "metaparameters"; require, which indicates that the configfile element should be applied after the package["httpd"] element, is such a metaparameter

The separate declaration of the apache user and group are not strictly necessary since the httpd rpm will create them as needed; they give us additional assurance though that changes that an administrator might make locally, for example, setting apache's login shell to /bin/bash will automatically be fixed by Puppet.

The declaration of the httpd service expresses that we want to have this service running, and that it needs to be restarted whenever the main Apache config file changes or when the httpd package is updated. We will see later how subclasses of the httpd class can add to this list of files to watch.

PostgreSQL

The Nag application can be run with a variety of database backends; for this example we arbitrarily chose PostgreSQL. The declarations for PostgreSQL are fairly straightforward. They are split into a class for the database server and some tools needed to configure clients; though this example sets up the database and the application on the same machine, this split is important if we ever want to deploy Nag on separate webserver and database machines, as discussed in the Variations. The server part is very similar to the declaration of the httpd class:

class postgres-server {
  package { postgresql-server:
    ensure => latest
  }
  # This mirrors what the package creates
  group { postgres: gid => 26 }
  user { postgres: 
   comment => "PostgreSQL Server",
   uid => 26,
   gid => 26,
   home => "/var/lib/pgsql",
   shell => "/bin/bash"
  }
  service { postgresql:
    running => true,
    pattern => "/usr/bin/postmaster",
    require => package["postgresql-server"]
  }
}

The pattern parameter in the postgresql service helps Puppet determine whether the service is running or not, since the init script of the postgresql-server returns a non-standard response for a status inquiry.

The client side declarations for PostgreSQL only consist of the helper scripts pgdatabase, pguser and pgscript and simple Puppet wrappers around them to handle the creation of users and databases within PostgreSQL and to execute SQL-scripts against a database. These wrappers illustrate that it is relatively easy to work around gaps in Puppet's native type library, though the hope is that most common configuration tasks will eventually be supported by Puppet natively; their definitions can be found in postgresql.pp.

Horde

We can finally turn our attention to setting up Nag, and the Horde application framework it depends on. Since there are other Horde applications besides Nag, the declarations for Horde and Nag are separated in horde.pp to make it clear how other Horde applications could be added. The file starts by defining two helper components, hordeconfig to deploy a config file for Horde, and hordesqlscript to run a SQL script against Horde's database. The more interesting of the two, hordeconfig, has to take into account that the directory structure for config files on the file server is slightly different from that on the webserver:

$hordedir = "/var/www/html/horde"

# Copy a horde config file from the central
# file server to the right place on the client machine
define hordeconfig(file = usename, component = base) {
  # Determine where the config file goes on the
  # webserver, depending on the component
  $hordbase = $component ? {
    nag => "$hordedir/nag",
    default => "$hordedir"
  }
  # Use $name as the file name, unless $file is given
  $conffile = $file ? {
    usename => $name,
    default => $file
  }
  # On the fileserver, we keep config files in directories
  # named /horde/$component, with /horde/base
  # for Horde's own files
  configfile { "$hordbase/config/$conffile.php":
    source => "/horde/$component/config/$conffile.php",
    require => package["horde"]
  }
}

Puppet identifies elements by their type and name, two elements with the same type and name refer to the same object on the client machine, and listing them repeatedly in the manifest simply changes the attributes for the same element. Both Horde and Nag use a configuration file named conf.php, and we use different names for their hordeconfig elements, and use hordeconfig's file parameter to specify the filename:

# Elements are identified by type + name; use different names to
# distinguish between horde's and nag's conf.php

# The conf.php for horde [in class horde]
# copies /horde/base/config/conf.php to $hordedir/config/conf.php
hordeconfig { conf: }  

# The conf.php for nag [in class horde-nag, a subclass of horde]
# copies /horde/nag/config/conf.php to $hordedir/nag/config/conf.php
hordeconfig { nag-conf: file => conf, component => nag }

PostgreSQL's authentication scheme requires that we enter the database user for Horde into the pg_hba.conf file on the database server. Unfortunately, PostgreSQL does not supply a tool to modify that file programmatically; as a shortcut, we will deploy the whole file, custom tailored to Horde's needs to the database server. This is done by the class horde-db-pg, together with ensuring that the postgresql service is running and is restarted whenever pg_hba.conf changes.

The webserver component of Horde needs to make sure that the Horde package and the PHP/PostgreSQL database adapter are installed, create the PostgreSQL database and database user for Horde, run the appropriate SQL scripts to create the Horde schema, and deploy Horde's config files. All this is done in a very straightforward manner in the class horde in horde.pp.

One special aspect of the horde class bears discussing in more detail: Horde needs to add a file to Apache's config to set access permissions for its files under /var/www/html. We want to make sure that Apache is restarted every time this file changes; but when we declared the httpd class, the base for our Apache configuration, we did not know that there would be an additional file to watch out for. Puppet makes it possible to override parameters set in a base class with new values in a subclass; the Apache specific part of the horde class therefore looks like this:

class horde inherits httpd {
  ...
  configfile { "/etc/httpd/conf.d/horde.conf":
    source => "/horde/base/httpd/horde.conf"
  }
  service { httpd:
    subscribe => [ file["/etc/httpd/conf/httpd.conf"],
                   file["/etc/httpd/conf.d/horde.conf"] ]
  }
  ...
}

The service element in this excerpt adds horde.conf to the list of files to watch for the httpd service; actually, Puppet currently does not allow adding entries to a list, so that we have to repeat all the files to watch here, but that feature is planned for one of the upcoming releases.

The ability to override settings from a base class in a subclass is enormously important for modularizing Puppet manifests: the writer of the httpd class should not have to anticipate how his class is going to be used. Overriding parameters in a subclass makes this possible, and gives writers of subclasses the flexibility to modify the behavior of classes provided by somebody else.

The final step in creating the Nag manifest is declaring the horde-nag class itself. It has to make sure that the horde-nag package is installed, that Nag's database schema is created and that Nag's config files are deployed to the proper place. We do not use the configuration files that come with the horde and horde-nag rpm's, instead we supply our own version of these files from the central file server; these configuration files are at the heart of adapting a Nag installation to local requirements, and will usually be changed by site administrators. By keeping just one copy on the central file server, site administrators can modify these files once, and need not worry about how they are deployed to all hosts that need them. This is less important in this example, where Nag is only being run on a single webserver; if Nag were to be run on several webservers, keeping the config files in a central location would guarantee that all the webservers use the same configuration and that no human intervention is needed to keep them in sync.

There is a second reason to keep these config files on the central fileserver: the Horde and Nag config files included in the tarball are actually not the stock files that come with the horde and horde-nag rpm's. An rpm-only installation of Nag requires that the administrator perform certain configuration tasks through Nag's web UI. The configuration files in the tarball have been created by performing these steps in the UI and saving the resulting files. They therefore represent a 'canned' configuration, i.e. a configuration performed by one party for a specific purpose that can be shared with others.

Putting it all together

We are finally at a point where we can run the manifest and deploy and control Nag through Puppet.

Installing Puppet

Puppet consists of a central server, where the manifests and supporting configuration files are stored, and clients that connect to the server, one client for each machine under Puppet's control. We'll assume that the central server has the hostname puppet.example.com, and the machine that will run the webapp has the hostname nag.example.com.

Both machines need to be installed with a basic OS install; this example has been successfully used on a machine with a RHEL4 installation for puppet, and a virtual guest under Xen from the latest Fedora Core rawhide for nag. The actual Puppet packages are available from my yum repository. Enable this repository on both machines and run yum install puppet-server on puppet and yum install puppet on nag. If your central Puppet server is not called puppet, you need to adjust the variable PUPPET_SERVER in /etc/sysconfig/puppet on nag.

The files that are needed to deploy and configure Nag and its dependencies are in this tarball. Download it and untar it in /var/puppet on puppet. Change /etc/sysconfig/puppetmaster to contain the line

 PUPPETMASTER_MANIFEST=/var/puppet/config/puppet/manifests/site.pp

The file /var/puppet/config/puppet/manifests/site.pp may also need to be changed if the machines you use are not called puppet and nag: the name of the puppet server must be assigned to the variable $server, instead of puppet.example.com, and the host you are using as nag needs to be given as the argument to node, instead of node nag.

Finally, you need to create a configuration for Puppet's file server. Edit the file /etc/puppet/fileserver.conf on puppet to contain a [config] module like

      [config]
      path /var/puppet/config
      allow 192.168.0.0/24 # Put your local network CIDR address here

Generating a certificate

Puppet secures the communication between client and server with SSL certificates. Before nag can successfully talk to puppet, we need to generate a client certificate for nag. One way to do this is to run the following command on puppet:

root@puppet # puppetca --generate nag.example.com

More details about certificate generation can be found in the Puppet documentation. If the above command produces an error, check that the directory /etc/puppet/ssl are owned by the puppet user. Because of a bug, this directory and its subdirectories may not be writeable by the Puppet server; to remedy that, run chown -R puppet:puppet /etc/puppet/ssl

Running Puppet

Now that we have Puppet installed, our manifests and configuration files in the proper places on puppet, all that remains is deploying it all. Make sure that the puppetmaster service is running on puppet and run the puppet demon on nag; for testing, it is easiest to execute the following command on nag:

root@nag # /etc/init.d/puppet once -v

This will execute the manifest for nag once and print verbose information about its working on the console. In a production environment, puppet would run as a demon and check periodically with the central server to receive updates to its configuration, ensuring that nag's configuration is in agreement with the one stored on the central server.

After the Puppet client finishes, nag should have a complete and working installation of the Nag application, and going to http://nag.example.com/horde in a browser should bring up the application.

You should experiment with the configuration on puppet, and run the client demon on nag repeatedly and observe how configuration files are deployed after changes, services get restarted as needed, packages get updated when a new package appears in the yum repositories etc.

Variations

The discussion in the introduction emphasized that recipes should make it possible to incorporate configuration information from a variety of sources, and should be adaptable to a number of configurations and deployment scenarios. The example should have made it clear that Puppet makes it possible to combine manifests from different sources; for example, the manifests for httpd, PostgreSQL, and Horde/Nag could have easily come from three different sources. To shed more light on how the manifests can be made more flexible, we consider two simple variations. The discussion of these two variations should make it clear that Puppet is indeed able to handle recipe-like configuration variations, even if it needs some more work to support that fully.

Separating the webserver and the database server

It is very common that web applications are deployed to several machines, one serving as the database server and one or more running the webserver and the actual application. The most obvious change needed to achieve this with the above manifests is that the main manifest, site.pp needs to have separate entries for the webserver and the database server, running on the hosts nag-web.example.com and nag-db.example.com:

node nag-web.example.com {
  include yum-rawhide
  include horde-nag
}

node nag-db.example.com {
  include yum-rawhide
  include horde-db-pg
}

Apart from applying the classes to different hosts, we also need to change the configurations of the Horde framework and of PostgreSQL. Horde needs to know which database server to connect to; that requires that we change Horde's conf.php accordingly. Similarly, PostgreSQL's configuration needs to be changed in two places: the file pg_hba.conf needs to be changed so that the horde database user can connect to the database from nag-web.example.com, and the file postgresql.conf needs to be changed so that PostgreSQL will accept connections over TCP/IP.

Both of these changes are minor tweaks to the existing manifests; what if we wanted to produce manifests that users can easily deploy to either a shared web/database server or separate database and web servers? There are a number of ways in which that could be expressed in the Puppet manifest, none of which Puppet currently supports. The simplest option is to parametrize the horde-nag and horde-db-pg classes with the name of the database machine, so that, in the shared db/webserver scenario, the site.pp would look like:

node nag.example.com {
  include yum-rawhide
  include horde-db { database => "localhost" }
  include horde-nag { database => "localhost" }
}

To separate the database from the webserver, the site.pp would be:

node nag-web.example.com {
  include yum-rawhide
  include horde-nag { database => "nag-db.example.com" }
}

node nag-db.example.com {
  include yum-rawhide
  include horde-db-pg { database => "nag-db.example.com" }
}

The important point is that users of the prebuilt horde.pp manifest do not need to look at it or understand it to achieve these different deployments. Unfortunately, Puppet does not yet support parameterizing classes in this way.

Deploying to a different database

The Horde framework supports a number of database backends, not just PostgreSQL. What would we need to change in our manifests to use a different database backend, say MySQL? The changes needed to achieve this are more extensive than the ones for separating the database and webserver. First, we need to create a new class horde-db-mysql that installs and configures a MySQL database in analogy to what horde-db-pg does. Secondly, we will need to change the horde class in horde.pp to install the proper PHP database driver for MySQL (the package php-mysql), deploy a different Horde config file conf.php and use MySQL-specific scripts for creating a MySQL database and database user.

Once again, the changes needed to select between a PostgreSQL and a MySQL backend could be hidden from the user by passing the desired backend as a parameter to the horde class, and creating a more generic horde-db class that selects the proper database server configuration, by including either horde-db-pg or horde-db-mysql.

It is also very likely that the differences between creating a database and database users for PostgreSQL and MySQL would be pushed into more generic dbuser and database elements, making these available to applications other than just Horde/Nag. Obviously, a lot of work is needed to create such a library of generic configuration elements.

Distribution

One aspect of system recipes that Puppet does not address, and probably should not address, is distribution of recipes. The simple approach of supplying a tarball used for this example is a far cry from what is needed to make it possible for users to incorporate recipes from various sources, modify them locally, and share their modifications.

Clearly, a mechanism for expressing dependencies between recipes is needed: when a user downloads just the horde.pp manifest and the configuration files it uses directly, the user has to be able to determine that they also need a manifest for the backend database and for httpd.

In all likelihood, users will not use canned recipes in their unmodified state for long, and will modify them locally; at the same time they should not lose the possibility of incorporating changes from the original upstream provider, e.g. of the horde.pp recipe.

These two requirements taken together, and the fact that the vast majority of files that recipes deal with are text files, suggest that recipes should be distributed akin to source code that is kept in a distributed source control system like git or mercurial, modified appropriately to support users in resolving dependencies between recipes.

Conclusions

The detailed example of Horde/Nag shows that Puppet in its current state is already a good tool to automate the configuration of a moderately complex application; the modularization of the setup that Puppet enables also makes it a good basis for sharing and combining prebuilt configurations into system recipes. Nonetheless, there are several areas in which Puppet needs to be extended to fully fit into a recipe-oriented management framework. Besides core language features such as parametrization of classes, much work needs to be done to build a comprehensive library to support most common configuration tasks. But even without that library, Puppet is flexible enough to make it possible to bridge gaps in the current library by simple means such as shell scripts, a feature that will remain important even as Puppet's library matures since it is unlikely that Puppet's library will ever contain primitives for all possible management tasks. The goal for Puppet's library has to be to support as many of the common configuration tasks as possible while enabling the addition of rarely used special-purpose tasks as easily as possible.

Besides enhancements to Puppet itself, tools are needed to support the distribution, sharing, and modification of recipes. There, Puppet should play the role that rpm plays in the world of package management, while a new tool for distribution of recipes should take on a role similar to yum.

Several desirable aspects of a system management solution have not even been touched upon in this document such as monitoring and provisioning, both important topics in their own right. And finally, a comprehensive architecture that integrates the various systems management tools into a cohesive solution should be developed alongside the various enhancements to the basic management tools.