salt.pillar.stack¶

Installation¶

PillarStack is already bundled with Salt since 2016.3.0 version so there is nothing to install from version 2016.3.0.

If you use an older Salt version or you want to override PillarStack with a more recent one, follow the installation procedure below.

Installing the PillarStack ext_pillar is as simple as dropping the stack.py file in the <extension_modules>/pillar directory (no external python module required), given that extension_modules is set in your salt-master configuration, see: https://docs.saltproject.io/en/latest/ref/configuration/master.html#extension-modules

Configuration in Salt¶

Like any other external pillar, its configuration takes place through the ext_pillar key in the master config file.

However, you can configure PillarStack in 3 different ways:

ext_pillar:
  - stack: /path/to/stack.cfg

ext_pillar:
  - stack:
      - /path/to/stack1.cfg
      - /path/to/stack2.cfg

ext_pillar:
  - stack:
      pillar:something:
        bar: /path/to/bar/stack.cfg
        foo: /path/to/foo/stack.cfg
      grains:custom:grain:
        value:
          - /path/to/stack1.cfg
          - /path/to/stack2.cfg
      opts:custom:opt:
        value: /path/to/stack0.cfg
      opts:saltenv:
        dev: /path/to/dev/stack.cfg
        __env__: /path/to/__env__/stack.cfg

PillarStack configuration files¶

The config files that are referenced in the above ext_pillar configuration are jinja2 templates which must render as a simple ordered list of yaml files that will then be merged to build pillar data.

The path of these yaml files must be relative to the directory of the PillarStack config file. These paths support unix style pathname pattern expansion through the Python glob module <https://docs.python.org/2/library/glob.html>.

The following variables are available in jinja2 templating of PillarStack configuration files:

• pillar: the pillar data (as passed by Salt to our ext_pillar function)

• minion_id: the minion id ;-)

• __opts__: a dictionary of mostly Salt configuration options

• __grains__: a dictionary of the grains of the minion making this pillar call

• __salt__: a dictionary of Salt module functions, useful so you don't have to duplicate functions that already exist (note: runs on the master)

So you can use all the power of jinja2 to build your list of yaml files that will be merged in pillar data.

For example, you could have a PillarStack config file which looks like:

$ cat /path/to/stack/config.cfg
core.yml
common/*.yml
osarchs/{{ __grains__['osarch'] }}.yml
oscodenames/{{ __grains__['oscodename'] }}.yml
{%- for role in pillar.get('roles', []) %}
roles/{{ role }}.yml
{%- endfor %}
minions/{{ minion_id }}.yml

And the whole directory structure could look like:

$ tree /path/to/stack/
/path/to/stack/
├── config.cfg
├── core.yml
├── common/
│   ├── xxx.yml
│   └── yyy.yml
├── osarchs/
│   ├── amd64.yml
│   └── armhf.yml
├── oscodenames/
│   ├── wheezy.yml
│   └── jessie.yml
├── roles/
│   ├── web.yml
│   └── db.yml
└── minions/
    ├── test-1-dev.yml
    └── test-2-dev.yml

Overall process¶

In the above PillarStack configuration, given that test-1-dev minion is an amd64 platform running Debian Jessie, and which pillar roles is ["db"], the following yaml files would be merged in order:

• core.yml

• common/xxx.yml

• common/yyy.yml

• osarchs/amd64.yml

• oscodenames/jessie.yml

• roles/db.yml

• minions/test-1-dev.yml

Before merging, every files above will be preprocessed as Jinja2 templates. The following variables are available in Jinja2 templating of yaml files:

• stack: the PillarStack pillar data object that has currently been merged (data from previous yaml files in PillarStack configuration)

• pillar: the pillar data (as passed by Salt to our ext_pillar function)

• minion_id: the minion id ;-)

• __opts__: a dictionary of mostly Salt configuration options

• __grains__: a dictionary of the grains of the minion making this pillar call

• __salt__: a dictionary of Salt module functions, useful so you don't have to duplicate functions that already exist (note: runs on the master)

So you can use all the power of jinja2 to build your pillar data, and even use other pillar values that has already been merged by PillarStack (from previous yaml files in PillarStack configuration) through the stack variable.

Once a yaml file has been preprocessed by Jinja2, we obtain a Python dict - let's call it yml_data - then, PillarStack will merge this yml_data dict in the main stack dict (which contains already merged PillarStack pillar data). By default, PillarStack will deeply merge yml_data in stack (similarly to the recurse salt pillar_source_merging_strategy), but 3 merging strategies are currently available for you to choose (see next section).

Once every yaml files have been processed, the stack dict will contain your whole own pillar data, merged in order by PillarStack. So PillarStack ext_pillar returns the stack dict, the contents of which Salt takes care to merge in with all of the other pillars and finally return the whole pillar to the minion.

Merging strategies¶

The way the data from a new yaml_data dict is merged with the existing stack data can be controlled by specifying a merging strategy. Right now this strategy can either be merge-last (the default), merge-first, remove, or overwrite.

Note that scalar values like strings, integers, booleans, etc. are always evaluated using the overwrite strategy (other strategies don't make sense in that case).

The merging strategy can be set by including a dict in the form of:

__: <merging strategy>

as the first item of the dict or list. This allows fine grained control over the merging process.

Merging examples¶

Let's go through small examples that should clarify what's going on when a yaml_data dict is merged in the stack dict.

When you don't specify any strategy, the default merge-last strategy is selected:

users:
  tom:
    uid: 500
    roles:
      - sysadmin
  root:
    uid: 0

users:
  tom:
    uid: 1000
    roles:
      - developer
  mat:
    uid: 1001

users:
  tom:
    uid: 1000
    roles:
      - sysadmin
      - developer
  mat:
    uid: 1001
  root:
    uid: 0

Then you can select a custom merging strategy using the __ key in a dict:

users:
  tom:
    uid: 500
    roles:
      - sysadmin
  root:
    uid: 0

users:
  __: merge-last
  tom:
    uid: 1000
    roles:
      - developer
  mat:
    uid: 1001

users:
  tom:
    uid: 1000
    roles:
      - sysadmin
      - developer
  mat:
    uid: 1001
  root:
    uid: 0

users:
  tom:
    uid: 500
    roles:
      - sysadmin
  root:
    uid: 0

users:
  __: merge-first
  tom:
    uid: 1000
    roles:
      - developer
  mat:
    uid: 1001

users:
  tom:
    uid: 500
    roles:
      - developer
      - sysadmin
  mat:
    uid: 1001
  root:
    uid: 0

users:
  tom:
    uid: 500
    roles:
      - sysadmin
  root:
    uid: 0

users:
  __: remove
  tom:
  mat:

users:
  root:
    uid: 0

users:
  tom:
    uid: 500
    roles:
      - sysadmin
  root:
    uid: 0

users:
  __: overwrite
  tom:
    uid: 1000
    roles:
      - developer
  mat:
    uid: 1001

users:
  tom:
    uid: 1000
    roles:
      - developer
  mat:
    uid: 1001

You can also select a custom merging strategy using a __ object in a list:

users:
  - tom
  - root

users:
  - __: merge-last
  - mat

users:
  - tom
  - root
  - mat

users:
  - tom
  - root

users:
  - __: merge-first
  - mat

users:
  - mat
  - tom
  - root

users:
  - tom
  - root

users:
  - __: remove
  - mat
  - tom

users:
  - root

users:
  - tom
  - root

users:
  - __: overwrite
  - mat

users:
  - mat