Pavefile Reference Part II


Sections and Task Library available to pave’s configuration file are described here. For syntax, conditionals, and general information, see the previous chapter Pavefile Reference.

Sections

A section is a top-level portion of a pavefile, which is organized hierarchically by indentation.

Names of sections are given by the labels at the far left in the first column. Names and the data (typically indented) below it are considered the complete section. Therefore, a name in the first column ends the preceding section and starts a new one:

# end previous section
foo:            # The "foo" section
    - bar
    - baz
# start next

The standard sections described next are handled by pave itself. Tasks implemented by library modules are described afterwards under Tasks Library and are placed in sub-sections under tasks.



vars

A fine place to define variables,  name: value  style.

When pave encounters name constructs in a string value it will replace them with a value given in this section. This expansion syntax is described in depth here, with the exception that pave uses the percent character to avoid clashing with shell scripting.

Pavefile strings support %-based substitutions, using the following rules:

  • Expansion can be avoided by doubling the % character.
  • %identifier names a substitution placeholder matching a mapping key of “identifier”, which must spell a pave variable. The name must be a letter or underscore, followed by a letter, digit, or underscore. The first non-identifier character after the % character terminates this placeholder specification.
  • %{identifier} is equivalent to %identifier.
    It is required when valid identifier characters (including underscores) follow the placeholder but are not part of the placeholder, such as %{noun}ification or %{username}_backup.zip.

Notes:

  • Passwords are added to the pavefile vars with the prefix “pwd_” e.g., pwd_appuser to support variable interpolation and avoid conflicting with existing variable names.
  • Variables in the var section will be expanded against themselves just once, in order to create vars that contain fragments of other vars—useful for building paths and such.

The order of variable expansion is as follows:

  1. Optional: A templating engine, such as Jinja.
  2. string.Template expansion of variables from the vars section performed locally by pave.
  3. Bash or other chosen shell will expand and run command strings in the the remote execution context.

Example:

vars:    # I put this at the top of the file for easy-access, baby.
    sitename: fooBar
    repo: "http://bitbucket.org/username/%sitename"  # existing var

tasks:     # usage, quotes not needed:
    - "echo %sitename rules"

Schema:

dict of
boolean || float || integer || list || string

main

Where options to pave are specified. Required.

bak-files:
Copy files to be modified to NAME.orig beforehand.
env:
A place to directly set fabric’s environment attributes. Several items, such as hostnames, will be clobbered if you set them here, since pave manages them.
include:
Include another file into this pavefile. Multiple entries accepted. Note that included sections will overwrite identical sections in the parent. Useful for separating lengthy orthogonal sections, e.g. target-groups:
inspect:

Inspect the target to determine platform details. Values:

  • True - Use the python inspector, if it fails try the shell script next. (Default)
  • False - Disable inspection, not currently useful.
  • py - Use the python inspector script only.
  • sh - Use the sh inspector script only.
jobs:
Number of “steamrollers.” Each task-list is run in parallel (per host) with the given pool size.
log-to:
Specify logging/data folder.
targets:
Specify hosts (or groups of hosts) to target. Whitespace-delimited string or list. See the target-groups: section below for group definition.
user:
Change the login user, defaults to the current one.
version:
Version of pave this file was written against, enabling time-travel. Not yet implemented.
warn-only:
Continue thru errors with a warning only.

Example:

main:
    user: ubuntu
    sudo: True
    targets:
        - 192.168.2.1     # a list or string of hosts, groups to greet.
        - www[1-3]

Schema:

dict of
bak-files:
boolean   ➤ default: True
env:
dict   ➤ default: {}
include:

list of string

string   ➤ default: []

inspect:
boolean || string   ➤ default: True
jobs:
100 True 1 True integer   ➤ default: 1
log-to:
string   ➤ default: ''
sudo:
boolean   ➤ default: True
sudo-user:
string
sys.path:

list of string

string

targets:

list of string

string   ➤ default: ''

user:
string   ➤ default: 'CURRENT'
version:
string   ➤ default: 'CURRENT'
warn-only:
boolean   ➤ default: False

passwords

Set or query passwords for later use.

  • ask NAME PROMPT
    Query the user for this password, named NAME, given PROMPT, prior to execution. As an “escape hatch,” two empty passwords given at a prompt will exit the program.

    Note: do not add a colon character to the end of the PROMPT value. One will be added automatically.

  • set NAME PASSWD
    Hard-code something temporarily (not recommended).
    This can greatly speed up authoring of pavefiles—But don’t check these in! ;) See /main/include to offload this section to another file instead. Using another file (with a standard name) and adding it to your vcs ignore file will greatly decrease the chances of a password checkin.

Note: 99% of the time when starting out you’ll want to give “password” (no quotes) as the value for NAME. This is the name fabric uses for the ssh and/or sudo password.

How to use gathered passwords in other sections:

  • use NAME
    Will plug-in the named password into another section’s options. See users: or postgres: for examples.
  • wget --password="%pwd_websvc"
    Use variable expansion to render the password into a command-line task. The variable name should be prepended with pwd_ to avoid collisions.

Example:

passwords:
    - ask password  ssh/sudo Password             # fabric default
    - ask pgsu      Postgres SuperUser Password
    - set appuser   "hello world"                 # set not recommended

Schema:

list of
string

target-groups

A mapping of group names assigned to lists of targeted hosts.

Example:

target-groups:
    dbservers: [host1, host2, host3]    # compact yaml list syntax
    webservers:
        - www[1-3]

This is a good place to mention hostname expansion. Leading 0‘s specify padding:

host[1-3]   —>   [host1, host2, host3]
host[01-03]   —>   [host01, host02, host03]

Schema:

dict of
string:

list of string

string

fab-tasks

A list of tasks to run directly from an existing fabfile.py.

These “fab-tasks” are run before standard task execution and do not interact with standard tasks. However, the fabric env (ironment) variables that have been set by pave are passed forward to these fab-tasks, allowing you use pave’s target groups and misc settings to drive fabric as if you were using the fab command.

env.fabfile:
Set the path of a fabfile here if it is not in the current or parent folders. Relative paths and tilde expansion are supported.

Arguments and keyword values are passed in using the same command-line syntax fab uses. For example:

main:
    env:
        fabfile: ../fabric/fabfile.py  # optional

fab-tasks:
    - task1
    - task2: arg1
    - task2: arg2,kwarg1=kwval1

Schema:

list of
string dict

tasks

A listing of tasks to be processed in order.

Each task in the list should be a bare string to be executed by the run module, or a mapping of a module name to the data it requires. The second form is called a “task group” at times as it may contain numerous sub-tasks.

See the Tasks Library below and Conditional Execution for more details.

Example:

tasks:
# tasks could be wide enough that you may want to skip the optional
# yaml list indent.

- banner "Hello!"     # world - strings get passed to the run module

# use the users module explicitly, a "task group"
- users:
    - name: appuser

Schema:

list of

string dict of

modname:
moddata


Tasks Library

Each mapping listed under tasks corresponds to a module in the site-packages/pave/lib folder or a third-party module found on the sys.path. See the Developer Guide for details.

cleanup

Good hygene is important.

folders:
Deletion of files under /tmp, /var/tmp, & /home/ only.
locales-except:
“Unless you’re working at the UN and administer a central server for all member states, it is difficult to conceive why you would need a system where all of these locales are installed.”
Similar to a one-time Debian locale-purge.
packages:
Run appropriate package cleanup command(s).

Example:

- cleanup:
    packages: True
    locales-except: en en_US
    folders:  # use with caution
        - /tmp/stuff/*
        - /tmp/stuff2

Schema:

dict of
do:
string
folders:
list of
string

➤ default: []

if:

if-exec:
string
if-platform:
string
locales-except:
string
packages:
boolean   ➤ default: True

configure

Configure the remote system. Also handles templating, see below.

render:

Renders a template to a local file, which is then uploaded to host.

engine:
The templating language, such as python (printf|Template|format), or the executable name of an external package, e.g: jinja2
render-remote:
Simple remote “templating” with sed.

Example:

- configure:            # str || list accepted on all sub-tasks below:
    create:  # file or folder/
        - foo.conf
        - /tmp/project/

    append:  # text remotefname
        Answer=42 %rc

    comment: # [comment char] from-regex remote-fname
        - ' "insecure on" %rcfile '

    replace: # from-regex to remote-fname
        - ^foo$ bar %rcfile
        # note that because of yaml syntax and splitting, two layers
        # of quoting or blocks may be needed when backslashes used:
        - >
            '^foo\s*$' bar %rcfile

    render: # engine [extra-vars] local-template remote-fname
        - printf swappiness=50 examples/printf_template.txt %rc2
        - format FOO=BAR       examples/format_template.txt %rc3

    render-remote: # [vars] remote-source remote-dest
        - INSTANCE_NUM=1 %sitecfg /etc/init/%svcname.conf

    update: # copy a file into place if newer, SRC DEST
        - ~/%sitename/cfg/celeryd.conf /etc/init/celeryd.conf

    access: # mode user group remote-fname # use "" for empty
        - 0640 %user %user %rc2

Schema:

dict of
access:

list of string

string   ➤ default: []

append:

list of string

string   ➤ default: []

comment:

list of string

string   ➤ default: []

create:

list of string

string   ➤ default: []

do:
string

if:

if-exec:
string
if-platform:
string
render:

list of string

string   ➤ default: []

render-remote:

list of string

string   ➤ default: []

replace:

list of string

string   ➤ default: []

update:

list of string

string   ➤ default: []

django

Django helper module. Captain Obvious recommends Django be installed on the target beforehand, see the packages section.

loaddata:
Load fixture files from given path, glob ok.
manage:
Execute a management command every time this task group is run.
manage-once:
Once and never again.

Example:

- django:
    workdir: ~%sitename/%sitename
    loaddata: "fixtures/*.json"
    manage-once: update_index
    syncdb: True

Schema:

dict of
do:
string

if:

if-exec:
string
if-platform:
string
loaddata:
string   ➤ default: ''
manage:

list of string

string   ➤ default: []

manage-once:

list of string

string   ➤ default: []

migrate-init:
boolean   ➤ default: True
syncdb:
boolean   ➤ default: True
user:
string   ➤ default: None
workdir:
string ➤ required

groups

Create operating system user groups.

Example:

- groups:           # str || list || conditional
    - appgroup
    - "dev:1100"    # how to set gid

    - if-exec: test -e /dev/cdrom
      do: cdrom

Schema:

list of string
dict of
do:
string

if:

if-exec:
string
if-platform:
string

kernel

Manage kernel parameters. Saves to memory and disk.

Example:

- kernel:           # str || list || conditional
    - vm.swappiness = 10

Schema:

list of string
dict of
do:
string

if:

if-exec:
string
if-platform:
string

keygen

Manage local/remote ssh keys and their generation.

remote:
Generates a key at the remote host under the given username.
copy-id:

Copies local keys to the remote host.

Values may be a single or list of Boolean (True copies the public key of the local user to the authorized keys file of the remote user), or string of two arguments containing the local file (.pub appended automatically) and the remote user.

Example:

- keygen:
    remote: "%sitename"             # quotes not required but
    copy-id:                        # fixes syntax highlighting
        - True                          # default src dest
        - "%local_path.pub %user"       # specified

Schema:

dict of
copy-id:
list of string
boolean

boolean || string   ➤ default: True

do:
string

if:

if-exec:
string
if-platform:
string
passphrase:
string   ➤ default: ''
remote:
None || string   ➤ default: None
type:
string   ➤ default: 'rsa'

packages

Manage software packages. Packages names for install, remove, etc. are listed as whitespace-delimited strings.

expiration:

The age in days after which the OS package index is considered to have expired, requiring action.

To be a good neighbor, the default, 0 (= 24 hours) means packages won’t be updated unless it has been a full day since the last update. A value of -1 forces action.

inventory:
Make a list of installed packages to compare against.
update/upgrade:
Update the system’s package index, upgrade packages to latest release.
upgrade-full:
Upgrade packages, even if it causes obsolete packages to be removed, or new packages to be installed, for example new kernel versions. e.g. Debian’s “dist-upgrade”.

Example:

- packages:
    if-platform: Debian
    install:
        sysvbanner python-pip
    # expiration: -1    # force attempt

Schema:

dict of
do:
string
expiration:
integer   ➤ default: 0

if:

if-exec:
string
if-platform:
string
install:
string   ➤ default: ''
inventory:
boolean   ➤ default: True
remove:
string   ➤ default: ''
update:
boolean   ➤ default: True
upgrade:
boolean   ➤ default: True
upgrade-full:
boolean   ➤ default: True

postgres

Manage postgres. Captain Obvious here to chime-in that postgres must be installed on remote target beforehand, see packages.

link current:
Create a symbolic link from the last sorted item in /etc/postgresql/ to /etc/postgresql/curr.
replaceaccess:
See the corresponding directive under configure. These are for convenience to group postgres-related configuration together.
userdbgrantsql:
postgresql-client-common (or distro equiv.) must be installed on remote target beforehand. Run the appropriate commands to create objects or configure the database. String or list of strings may be given.

Example:

- postgres:
    link current: False
    replace:
        - '#\s*password_enc password_enc %pgcfg/postgresql.conf'

    su-password:
        use pgsu                                # see passwords:
        # md5 acbd18db4cc2f85cedef654fccc4a4d8  # how to hard-code

    db: # quotes not required below, but fixes highlighting. :/
        - name: "%sitename"
    user:
        - name: "%sitename"
    grant:
        # privs   on_obj    named         to_user
        all       database  %sitename     %sitename

    sql: # str|list lines of sql, first token should be dbname:
        template1 select datname from pg_database
                  where datname = '%sitename';

Schema:

dict of
access:

list of string

string   ➤ default: []

db:
list of dict of
description:
None || string   ➤ default: []
encoding:
string
locale:
string
name:
string ➤ required
tablespace:
string
template:
string   ➤ default: 'template0'
do:
string
grant:

list of string

string   ➤ default: []

if:

if-exec:
string
if-platform:
string
link current:
boolean   ➤ default: True
replace:

list of string

string   ➤ default: []

sql:

list of string

string   ➤ default: []

su-password:
string   ➤ default: ''
user:
list of dict of
createdb:
None || boolean   ➤ default: False
createrole:
boolean   ➤ default: False
encrypted:
boolean   ➤ default: True
login:
boolean   ➤ default: True
name:
string ➤ required
superuser:
boolean   ➤ default: False

run

Executes a list of commands using the remote shell. Each command may be:

  • A string

    A command that runs under the default user.

  • A mapping with one or more of the following members:
    do: CMD_STRING
    The command to run.
    user: USERNAME
    Run as another user.
    May be sudo (root), or a valid remote username.
    title: TITLE
    to document or hide long, ugly commands.
    workdir: PATH set working directory.
    local: <bool> runs it locally instead of remotely.

See Conditional Execution for more details on if: and if-exec:.

Note

While not recommended to set passwords via command-line tools for various reasons, tasks makes an attempt to hide them (vars starting with “pwd_”) from standard logs. Passwords will still be displayed in debug logs (and perhaps shell history) upon command-line execution.

Example:

- banner "Hello!"     # world

# signal a change event and prevent errors from halting execution:
- touch %filename && echo CHANGED || true
- rm /etc/nginx/sites-enabled/default && echo CHANGED || echo SKIPPED

# Even poor-er man's conditionals:
- " [ '%mode' == 'qa' ] && echo 'QA install' "

# more robust task definition:
- run:
    - title: Nothing to see here...
      if-exec: test -f /foo
      do: echo "This is running under $USER (appuser)"
      user: appuser

Schema:

list of

string dict of

do:
string

if:

if-exec:
string
if-platform:
string
local:
boolean
title:
string
user:
string
workdir:
string

scp

Copy files to and fro. Given a source and destination, copy files.

Notes:

  • By default a put (upload) is performed, but the direction can be specified as the first parameter.
  • Paths with whitespace must be quoted.
  • Destinations folders should end in a / character for efficiency, as doing so skips the need for several filesystem tests.
  • If sudo is enabled, relative paths/tildes will be interpreted (and files owned) by the root user. Therefore it is recommended that absolute remote paths be given.
  • Does not currently handle wildcards.
  • While this module is similar to using the scp command-line program it may not work identically.

Example:

- scp:
    - SRC DEST          # put
    - SRC DEST MODE     # put, w/ Unix octal mode or "same"
    - get SRC DEST

    # example conditional
    - if: inst_ssl == True
      do: put bundle.crt "%sitefldr/ssl/bundle.crt"

Additional details:

Schema:

list of

string dict of

do:
string

if:

if-exec:
string
if-platform:
string
string:
string

services

Manage system daemons/services. Values are whitespace separated strings.

restart:, restart-fail:
restart will start the service if it was stopped, restart-fail will fail with an error if it was not running originally.

Note

Some daemon startup scripts are not written well and quit when the ssh session ends. To prevent this from occurring, prepend the service name with “nohup:” as shown below.

Example:

- services:
    # the follwing two are temporary until reboot:
    running: hostname hwclock
    stopped: whoopsie
    # permanently; /packages/remove more reliable on ubuntu
    disabled: whoopsie
    restart: nohup:memcached

Schema:

dict of
disabled:
string   ➤ default: ''
do:
string

if:

if-exec:
string
if-platform:
string
restart:
string   ➤ default: ''
restart-fail:
string   ➤ default: ''
running:
string   ➤ default: ''
stopped:
string   ➤ default: ''

users

Create operating system users.

Example:

- users:
    - name: appuser                 # required
      groups: appgroup www-data     # whitespace separated groups
      create-home: True
      shell: /bin/bash
      password: use appuser         # see passwords: section

Schema:

list of
dict of
create-home:
boolean   ➤ default: True
do:
string
groups:
None || string   ➤ default: None

if:

if-exec:
string
if-platform:
string
name:
string ➤ required
password:
string
shell:
string

vcs

Deploy code with this version control helper. $VCS support must be installed beforehand, see the packages section.

If the destination does not exist, an initial checkout is done. If it does exist, the repository is updated to the latest version.

Formats:

- VCS_TYPE REPO_URL DEST EXTRA          # simple or
- do: VCS_TYPE REPO_URL DEST EXTRA      # robust
VCS_TYPE:
Version Control System, one of svn, hg, or git.
REPO_URL:
URL to repository.
DEST:
Destination folder.
EXTRA:
Extra command line parameters.
do:
What to do.
input:
Useful to automate interactive prompts.
user::
Execute vcs under this remote username.

Example:

- vcs:
    - do: hg %repo_base/appname ~%user/appname
      input: p\nn                           # interactive answer
      user: %user

Schema:

list of

string dict of

do:
string

if:

if-exec:
string
if-platform:
string
input:
string
user:
string