Salt's Test Suite¶

Integration Tests¶

The Integration section of Salt's test suite start up a number of Salt daemons to test functionality in a live environment. These daemons include two Salt Masters, one Syndic, and two Minions. This allows the Syndic interface to be tested and Master/Minion communication to be verified. All of the integration tests are executed as live Salt commands sent through the started daemons.

Integration tests are particularly good at testing modules, states, and shell commands, among other segments of Salt's ecosystem. By utilizing the integration test daemons, integration tests are easy to write. They are also SaltStack's generally preferred method of adding new tests.

The discussion in the Integration vs. Unit section of the testing tutorial is beneficial in learning why you might want to write integration tests vs. unit tests. Both testing arenas add value to Salt's test suite and you should consider adding both types of tests if possible and appropriate when contributing to Salt.

• Integration Test Documentation

Unit Tests¶

Unit tests do not spin up any Salt daemons, but instead find their value in testing singular implementations of individual functions. Instead of testing against specific interactions, unit tests should be used to test a function's logic as well as any return or raises statements. Unit tests also rely heavily on mocking external resources.

The discussion in the Integration vs. Unit section of the testing tutorial is useful in determining when you should consider writing unit tests instead of, or in addition to, integration tests when contributing to Salt.

• Unit Test Documentation

Running Test Subsections¶

Instead of running the entire test suite all at once, which can take a long time, there are several ways to run only specific groups of tests or individual tests:

• Run unit tests only: nox -e 'test-3(coverage=False)' -- tests/unit/.

• Run unit and integration tests for states: nox -e 'test-3(coverage=False)' -- tests/unit/states/ tests/integration/states/.

• Run integration tests for an individual module: nox -e 'test-3(coverage=False)' -- tests/pytests/integration/modules/test_virt.py.

• Run unit tests for an individual module: nox -e 'test-3(coverage=False)' -- tests/unit/modules/test_virt.py.

• Run an individual test by using the class and test name (this example is for the test_default_kvm_profile test in the tests/pytests/integration/module/test_virt.py): nox -e 'test-3(coverage=False)' -- tests/pytests/integration/modules/test_virt.py::VirtTest::test_default_kvm_profile.

For more specific examples of how to run various test subsections or individual tests, please see the pytest documentation on how to run specific tests or the Running Specific Tests section of the Salt's Test Suite: An Introduction tutorial.

Running Unit Tests Without Integration Test Daemons¶

Since the unit tests do not require a master or minion to execute, it is often useful to be able to run unit tests individually, or as a whole group, without having to start up the integration testing daemons. Starting up the master, minion, and syndic daemons takes a lot of time before the tests can even start running and is unnecessary to run unit tests. To run unit tests without invoking the integration test daemons, simply add the unit directory as an argument:

nox -e 'test-3(coverage=False)' -- tests/unit/

All of the other options to run individual tests, entire classes of tests, or entire test modules still apply.

Running Destructive Integration Tests¶

Salt is used to change the settings and behavior of systems. In order to effectively test Salt's functionality, some integration tests are written to make actual changes to the underlying system. These tests are referred to as "destructive tests". Some examples of destructive tests are changes may be testing the addition of a user or installing packages. By default, destructive tests are disabled and will be skipped.

Generally, destructive tests should clean up after themselves by attempting to restore the system to its original state. For instance, if a new user is created during a test, the user should be deleted after the related test(s) have completed. However, no guarantees are made that test clean-up will complete successfully. Therefore, running destructive tests should be done with caution.

To run tests marked as destructive, set the --run-destructive flag:

nox -e 'test-3(coverage=False)' -- --run-destructive

Running Cloud Provider Tests¶

Salt's testing suite also includes integration tests to assess the successful creation and deletion of cloud instances using Salt-Cloud for providers supported by Salt-Cloud.

The cloud provider tests are off by default and run on sample configuration files provided in tests/integration/files/conf/cloud.providers.d/. In order to run the cloud provider tests, valid credentials, which differ per provider, must be supplied. Each credential item that must be supplied is indicated by an empty string value and should be edited by the user before running the tests. For example, DigitalOcean requires a client key and an api key to operate. Therefore, the default cloud provider configuration file for DigitalOcean looks like this:

digitalocean-config:
  driver: digitalocean
  client_key: ''
  api_key: ''
  location: New York 1

As indicated by the empty string values, the client_key and the api_key must be provided:

digitalocean-config:
  driver: digitalocean
  client_key: wFGEwgregeqw3435gDger
  api_key: GDE43t43REGTrkilg43934t34qT43t4dgegerGEgg
  location: New York 1

Once all of the valid credentials for the cloud provider have been supplied, the cloud provider tests can be run like:

nox -e 'test-cloud-3(coverage=False)'

Using Salt-Cloud on Jenkins¶

For testing Salt on Jenkins, SaltStack uses Salt-Cloud to spin up virtual machines. The script using Salt-Cloud to accomplish this is open source and can be found here: tests/jenkins.py

Naming Conventions¶

Any function in either integration test files or unit test files that is doing the actual testing, such as functions containing assertions, must start with test_:

def test_user_present(self): ...

When functions in test files are not prepended with test_, the function acts as a normal, helper function and is not run as a test by the test suite.

Submitting New Tests¶

Which branch of the Salt codebase should new tests be written against? The location of where new tests should be submitted depends largely on the reason you're writing the tests.

Module/Global Level Variables¶

If you need access to module or global level variables, please use a pytest fixture. The use of module and global variables can introduce mutable global objects and increases processing time because all globals are evaluated when collecting tests. If there is a use case where you cannot use a fixture and you are using a type of string, integer, or tuple you can use global/module level variables. Any mutable types such as lists and dictionaries must use pytest fixtures. For an example, if all of your tests need access to a string variable:

FOO = "bar"


def test_foo_bar():
    assert FOO == "bar"


def test_foo_not():
    assert not FOO == "foo"

We recommend using a pytest fixture:

import pytest


@pytest.fixture()
def foo():
    return "bar"


def test_foo_bar(foo):
    assert foo == "bar"


def test_foo_not(foo):
    assert not foo == "foo"

If you need a class to mock something, it can be defined at the global scope, but it should only be initialized on the fixture:

class Foo:
    def __init__(self):
        self.bar = True


@pytest.fixture
def foo():
    return Foo()

Test Helpers¶

Several Salt-specific helpers are available. A full list is available by inspecting functions exported under tests/support/*.py.

Test Markers¶

@pytest.mark.expensive_test -- Designates a test which typically requires a relatively costly external resource, like a cloud virtual machine. This decorator is not normally used by developers outside of the Salt core team.

@pytest.mark.destructive_test -- Marks a test as potentially destructive. It will not be run unless the --run-destructive flag is expressly passed.

@pytest.mark.requires_network -- Requires a network connection for the test to operate successfully. If a network connection is not detected, the test will not run.

These are just a small preview of the supported marker. For a full listing, please run:

nox -e 'test-3(coverage=False)' -- --markers