Troubleshooting

Generally speaking, troubleshooting anything requires finesse, but here are some common patterns for you to keep in mind. For more tips, see this Salt Troubleshooting guide.

Checking process status

The Salt master, minion, and proxies are run as system processes in the systemd service, and managed by systemctl.

Check the status to see if the process is running or if it has encountered an error:

systemctl status salt-minion

Alternately, bypass systemd and run the process in the foreground. For example, with log level debugging:

salt-master -l debug

salt-minion -l debug

Viewing log files

The log file locations for the master, minion, and proxy processes are configured via the configuration files. Their default location is /var/log/salt. It may be helpful to view the log files with tail:

tail -f /var/log/salt/master

tail -f /var/log/salt/minion

tail -f /var/log/salt/proxy

Log levels

The following log levels are available in Salt:

Level	Numeric Value	Description
quiet	1000	Nothing logged
critical	50	Critical errors
error	40	Errors
warning	30	Warnings
info	20	Normal log information
profile	15	Profiling information on salt performance
debug	10	Information useful for debugging
trace	5	Even more debugging information
all	0	Everything

You can change the log level via the configuration file:

/etc/salt/minion.d/logs.conf

 log_file: /var/log/salt/minion  # default
 log_level: debug

You must restart the process for the configuration changes to take effect:

systemctl restart salt-minion

Viewing Salt events

Salt achieves remote execution by having the master publish “events”, while the minions listening to this event bus execute the commands in which they are targeted. The master’s event bus can be viewed in real time with salt-run:

salt-run state.event pretty=True

Opening ports

Salt master and minions require two open ports for communication. By default, port 4505 is for publishing or subscribing to the master’s event bus, and port 4506 is for returning data. These can be configured via the master / minion configuration files:

In the master configuration file:

publish_port: 4505
ret_port: 4506

In the minion configuration file:

publish_port: 4505
master_port: 4506

Tip

Make sure your firewall is not blocking these ports, if applicable.

Salt keys

salt-key is a valuable utility for managing key acceptance. However, it may be beneficial to understand how Salt stores this information in the file system.

The accepted and rejected minion keys are stored on the master’s file system /etc/salt/pki/master/

In this directory you will find the master’s public and private key and several other directories for storing the incoming minion key according to its state. Once the keys are accepted, they will be stored in /etc/salt/pki/master/minions/<minion-id> <minion-id> is the id of the minion in question.

On the minion, the keys are stored in /etc/salt/pki/minion/. Here you will find the minion’s private and public keys as well as the public key for it’s master (in a file named minion_master.pub)

Tip

If the master or minion’s key has changed, Salt will have to accept the new key. If a previous key had been accepted (on the master / minion) this file may need to be deleted (manually or with salt-key), and the master / minion processes restarted.

Troubleshooting states

The following steps typically occur when a state is executed:

1. Render template (such as Jinja)

2. Compile YAML to “high-data”

3. Compile “high data” to “low data”

4. Execute each low chunk until complete

To diagnose problems, try performing single steps along this execution. For example, we can render the Jinja with the following:

salt <tgt> slsutil.renderer <my_custom_state> default_renderer=jinja

Jinja and YAML (in order) are rendered by default.

salt <tgt> slsutil.renderer <my_custom_state>

The above commands are for rendering the file from a Jinja / YAML perspective. The rendered SLS is similar since the state system will render Jinja / YAML. However, this command includes keys added during the state system render phase:

salt <tgt> state.show_sls <my_custom_state>

This can be a useful command especially when passing pillars from the command line:

salt <tgt> state.show_sls <my_custom_state> pillar='{"foo": "bar"}'

Salt states can be executed in test mode, by setting the flag test=True:

salt <tgt> state.apply <my_custom_state> test=True

Here the state is rendered and evaluated as much as possible, yet changes are not applied to the remote device.

Adding logging to custom modules

After creating a custom module, add logging in the normal python way:

/srv/salt/_modules/my_custom_module.py

import logging

log = logging.getLogger(__name__)


def hello():
    log.debug("hello is running...")
    return "hello"

In the above example, set the log level to debug or lower.

Debugging reactor states

The best way to debug Reactor States is to use multiple views into the Salt master and then fire some specific events:

• Monitor the Salt master log file: /var/log/salt/master. This will help to debug Jinja / YAML errors in the Reactor SLS file.

• Monitor the Event Bus: salt-run state.event pretty=True. Watch for the triggering event as well as the Reactor events.

• Fire events manually to trigger the Reactor.

Problems with sync/cache

Salt stores all modules and other files in the minions cache, typically located under /var/cache/salt. If custom modules aren’t behaving the way you think they should, it’s possible that the cache is not up to date. All custom modules must be synced to the minions:

salt \* saltutil.sync_all

But if a file is deleted, you must also clear the cache and re-sync:

salt \* saltutil.clear_cache
salt \* saltutil.sync_all

When in doubt, it may be helpful to restart the minion process:

systemctl restart salt-minion

Troubleshoot rendering

Render json file

Loads json data from an absolute path:

salt \* jinja.import_JSON /srv/salt/foo.json

Render YAML file

Loads YAML data from an absolute path:

salt \* jinja.import_yaml /srv/salt/foo.yaml

Render a map file

Assuming the map is loaded in your formula SLS as follows:

{% from "foo/map.jinja" import bar with context %}

Then the following syntax can be used to render the map variable bar:

$ salt \* jinja.load_map /srv/salt/foo/map.jinja bar