salt.modules.saltcheck

A module for testing the logic of states and highstates on salt minions

codeauthor:

William Cannon <william.cannon@gmail.com>

maturity:

new

Saltcheck provides unittest like functionality requiring only the knowledge of salt module execution and yaml. Saltcheck uses salt modules to return data, then runs an assertion against that return. This allows for testing with all the features included in salt modules.

In order to run state and highstate saltcheck tests, a sub-folder in the state directory must be created and named saltcheck-tests. Tests for a state should be created in files ending in *.tst and placed in the saltcheck-tests folder. tst files are run through the salt rendering system, enabling tests to be written in yaml (or renderer of choice), and include jinja, as well as the usual grain and pillar information. Like states, multiple tests can be specified in a tst file. Multiple tst files can be created in the saltcheck-tests folder, and should be named the same as the associated state. The id of a test works in the same manner as in salt state files and should be unique and descriptive.

New in version 3000: The saltcheck-tests folder can be customized using the saltcheck_test_location minion configuration setting. This setting is a relative path from the formula's salt:// path to the test files.

Usage

Example Default file system layout:

/srv/salt/apache/
    init.sls
    config.sls
    saltcheck-tests/
        init.tst
        config.tst
        deployment_validation.tst

Alternative example file system layout with custom saltcheck_test_location:

Minion configuration:

saltcheck_test_location: tests/integration/saltcheck

Filesystem layout:

/srv/salt/apache/
    init.sls
    config.sls
    tests/integration/saltcheck/
        init.tst
        config.tst
        deployment_validation.tst

Tests can be run for each state by name, for all apache/saltcheck/*.tst files, or for all states assigned to the minion in top.sls. Tests may also be created with no associated state. These tests will be run through the use of saltcheck.run_state_tests, but will not be automatically run by saltcheck.run_highstate_tests.

salt '*' saltcheck.run_state_tests apache,apache.config
salt '*' saltcheck.run_state_tests apache check_all=True
salt '*' saltcheck.run_highstate_tests
salt '*' saltcheck.run_state_tests apache.deployment_validation

Saltcheck Keywords

module_and_function:

(str) This is the salt module which will be run locally, the same as salt-call --local <module>. The saltcheck.state_apply module name is special as it bypasses the local option in order to resolve state names when run in a master/minion environment.

args:

(list) Optional arguments passed to the salt module

kwargs:

(dict) Optional keyword arguments to be passed to the salt module

assertion:

(str) One of the supported assertions and required except for saltcheck.state_apply Tests which fail the assertion and expected_return, cause saltcheck to exit which a non-zero exit code.

expected_return:

(str) Required except by assertEmpty, assertNotEmpty, assertTrue, assertFalse. The return of module_and_function is compared to this value in the assertion.

assertion_section:

(str) Optional keyword used to parse the module_and_function return. If a salt module returns a dictionary as a result, the assertion_section value is used to lookup a specific value in that return for the assertion comparison.

assertion_section_delimiter:

(str) Optional delimiter to use when splitting a nested structure. Defaults to ':'

print_result:

(bool) Optional keyword to show results in the assertEqual, assertNotEqual, assertIn, and assertNotIn output. Defaults to True.

output_details:

(bool) Optional keyword to display module_and_function, args, assertion_section, and assertion results text in the output. If print_result is False, assertion results will be hidden. This is a per test setting, but can be set globally for all tests by adding saltcheck_output_details: True in the minion configuration file. Defaults to False

pillar_data:

(dict) Optional keyword for passing in pillar data. Intended for use in potential test setup or teardown with the saltcheck.state_apply function.

skip:

(bool) Optional keyword to skip running the individual test

New in version 3000: Multiple assertions can be run against the output of a single module_and_function call. The assertion, expected_return, assertion_section, and assertion_section_delimiter keys can be placed in a list under an assertions key. See the multiple assertions example below.

Sample Cases/Examples

Basic Example

echo_test_hello:
  module_and_function: test.echo
  args:
    - "hello"
  kwargs:
  assertion: assertEqual
  expected_return:  'hello'

Example with jinja

{% for package in ["apache2", "openssh"] %}
{# or another example #}
{# for package in salt['pillar.get']("packages") #}
test_{{ package }}_latest:
  module_and_function: pkg.upgrade_available
  args:
    - {{ package }}
  assertion: assertFalse
{% endfor %}

Example with setup state including pillar

setup_test_environment:
  module_and_function: saltcheck.state_apply
  args:
    - common
  pillar_data:
    data: value

verify_vim:
  module_and_function: pkg.version
  args:
    - vim
  assertion: assertNotEmpty

Example with jinja

{% for package in ["apache2", "openssh"] %}
{# or another example #}
{# for package in salt['pillar.get']("packages") #}
test_{{ package }}_latest:
  module_and_function: pkg.upgrade_available
  args:
    - {{ package }}
  assertion: assertFalse
{% endfor %}

Example with setup state including pillar

setup_test_environment:
  module_and_function: saltcheck.state_apply
  args:
    - common
  pillar-data:
    data: value

verify_vim:
  module_and_function: pkg.version
  args:
    - vim
  assertion: assertNotEmpty

Example with skip

package_latest:
  module_and_function: pkg.upgrade_available
  args:
    - apache2
  assertion: assertFalse
  skip: True

Example with assertion_section

validate_shell:
  module_and_function: user.info
  args:
    - root
  assertion: assertEqual
  expected_return: /bin/bash
  assertion_section: shell

Example with a nested assertion_section

validate_smb_signing:
  module_and_function: lgpo.get
  args:
    - 'Machine'
  kwargs:
    return_full_policy_names: True
  assertion: assertEqual
  expected_return: Enabled
  assertion_section: 'Computer Configuration|Microsoft network client: Digitally sign communications (always)'
  assertion_section_delimiter: '|'

Example suppressing print results

validate_env_nameNode:
  module_and_function: hadoop.dfs
  args:
    - text
    - /oozie/common/env.properties
  expected_return: nameNode = hdfs://nameservice2
  assertion: assertNotIn
  print_result: False

Example with multiple assertions and output_details

multiple_validations:
  module_and_function: network.netstat
  assertions:
    - assertion: assertEqual
      assertion_section: "0:program"
      expected_return: "systemd-resolve"
    - assertion: assertEqual
      assertion_section: "0:proto"
      expected_return: "udp"
  output_details: True

Supported assertions

  • assertEqual

  • assertNotEqual

  • assertTrue

  • assertFalse

  • assertIn

  • assertNotIn

  • assertGreater

  • assertGreaterEqual

  • assertLess

  • assertLessEqual

  • assertEmpty

  • assertNotEmpty

Warning

The saltcheck.state_apply function is an alias for state.apply. If using the ACL system saltcheck.* might provide more capability than intended if only saltcheck.run_state_tests and saltcheck.run_highstate_tests are needed.

class salt.modules.saltcheck.SaltCheck(saltenv='base')

This class validates and runs the saltchecks

run_test(test_dict)

Run a single saltcheck test

class salt.modules.saltcheck.StateTestLoader(saltenv='base')

Class loads in test files for a state e.g. state_dir/saltcheck-tests/[1.tst, 2.tst, 3.tst]

add_test_files_for_sls(sls_name, check_all=False)

Detects states used, caches needed files, and adds to test list

load_test_suite()

Load tests either from one file, or a set of files

salt.modules.saltcheck.parallel_scheck(data)

triggers salt-call in parallel

salt.modules.saltcheck.report_highstate_tests(saltenv=None)

Report on tests for states assigned to the minion through highstate. Quits with the exit code for the number of missing tests.

CLI Example:

salt '*' saltcheck.report_highstate_tests

New in version 3000.

salt.modules.saltcheck.run_highstate_tests(saltenv=None, only_fails=False, junit=False)

Execute all tests for states assigned to the minion through highstate and return results

Parameters:
  • saltenv (str) -- optional saltenv. Defaults to base

  • only_fails (bool) -- boolean to only print failure results

  • junit (bool) -- boolean to print results in junit format .. versionadded:: 3007.0

CLI Example:

salt '*' saltcheck.run_highstate_tests
salt.modules.saltcheck.run_state_tests(state, saltenv=None, check_all=False, only_fails=False, junit=False)

Execute tests for a salt state and return results Nested states will also be tested

Parameters:
  • state (str) -- state name for which to run associated .tst test files

  • saltenv (str) -- optional saltenv. Defaults to base

  • check_all (bool) -- boolean to run all tests in state/saltcheck-tests directory

  • only_fails (bool) -- boolean to only print failure results

  • junit (bool) -- boolean to print results in junit format .. versionadded:: 3007.0

CLI Example:

salt '*' saltcheck.run_state_tests postfix,common

Tests will be run in parallel by adding "saltcheck_parallel: True" in minion config. When enabled, saltcheck will use up to the number of cores detected. This can be limited by setting the "saltcheck_processes" value to an integer to set the maximum number of parallel processes.

salt.modules.saltcheck.run_state_tests_ssh(state, saltenv=None, check_all=False, only_fails=False, junit=False)

This function is an alias of run_state_tests.

Execute tests for a salt state and return results Nested states will also be tested

param str state:

state name for which to run associated .tst test files

param str saltenv:

optional saltenv. Defaults to base

param bool check_all:

boolean to run all tests in state/saltcheck-tests directory

param bool only_fails:

boolean to only print failure results

param bool junit:

boolean to print results in junit format .. versionadded:: 3007.0

CLI Example:

salt '*' saltcheck.run_state_tests postfix,common

Tests will be run in parallel by adding "saltcheck_parallel: True" in minion config. When enabled, saltcheck will use up to the number of cores detected. This can be limited by setting the "saltcheck_processes" value to an integer to set the maximum number of parallel processes.

salt.modules.saltcheck.run_test(**kwargs)

Execute one saltcheck test and return result

Parameters:

test (keyword arg) --

CLI Example:

salt '*' saltcheck.run_test
    test='{"module_and_function": "test.echo",
           "assertion": "assertEqual",
           "expected_return": "This works!",
           "args":["This works!"] }'
salt.modules.saltcheck.state_apply(state_name, **kwargs)

Runs state.apply with given options to set up test data. Intended to be used for optional test setup or teardown

Reference the state.apply module documentation for arguments and usage options

CLI Example:

salt '*' saltcheck.state_apply postfix