.. _troubleshooting: =============== Troubleshooting =============== Generally speaking, troubleshooting anything requires finesse, but here are some common patterns for you to keep in mind. For more tips, see this Salt `Troubleshooting `__ guide. Checking process status ======================= The Salt master, minion, and proxies are run as system processes in the ``systemd`` service, and managed by ``systemctl``. Check the status to see if the process is running or if it has encountered an error: .. code-block:: shell systemctl status salt-minion Alternately, bypass ``systemd`` and run the process in the foreground. For example, with log level debugging: .. code-block:: shell salt-master -l debug .. code-block:: shell salt-minion -l debug Viewing log files ================= The log file locations for the master, minion, and proxy processes are configured via the configuration files. Their default location is ``/var/log/salt``. It may be helpful to view the log files with ``tail``: .. code-block:: shell tail -f /var/log/salt/master .. code-block:: shell tail -f /var/log/salt/minion .. code-block:: shell tail -f /var/log/salt/proxy Log levels ---------- The following log levels are available in Salt: .. list-table:: :widths: 20 20 60 :header-rows: 1 * - Level - Numeric Value - Description * - quiet - 1000 - Nothing logged * - critical - 50 - Critical errors * - error - 40 - Errors * - warning - 30 - Warnings * - info - 20 - Normal log information * - profile - 15 - Profiling information on salt performance * - debug - 10 - Information useful for debugging * - trace - 5 - Even more debugging information * - all - 0 - Everything You can change the log level via the configuration file: .. code-block:: yaml :caption: /etc/salt/minion.d/logs.conf log_file: /var/log/salt/minion # default log_level: debug You must restart the process for the configuration changes to take effect: .. code-block:: shell systemctl restart salt-minion Viewing Salt events =================== Salt achieves remote execution by having the master publish “events”, while the minions listening to this event bus execute the commands in which they are targeted. The master's event bus can be viewed in real time with ``salt-run``: .. code-block:: shell salt-run state.event pretty=True Opening ports ============= Salt master and minions require two open ports for communication. By default, port ``4505`` is for publishing or subscribing to the master’s event bus, and port ``4506`` is for returning data. These can be configured via the master / minion configuration files: In the master configuration file: .. code-block:: yaml publish_port: 4505 ret_port: 4506 In the minion configuration file: .. code-block:: yaml publish_port: 4505 master_port: 4506 .. Tip:: Make sure your firewall is not blocking these ports, if applicable. Salt keys ============= ``salt-key`` is a valuable utility for managing key acceptance. However, it may be beneficial to understand how Salt stores this information in the file system. The accepted and rejected minion keys are stored on the master’s file system ``/etc/salt/pki/master/`` In this directory you will find the master’s public and private key and several other directories for storing the incoming minion key according to its state. Once the keys are accepted, they will be stored in ``/etc/salt/pki/master/minions/`` ```` is the id of the minion in question. On the minion, the keys are stored in ``/etc/salt/pki/minion/``. Here you will find the minion’s private and public keys as well as the public key for it’s master (in a file named ``minion_master.pub``) .. Tip:: If the master or minion’s key has changed, Salt will have to accept the new key. If a previous key had been accepted (on the master / minion) this file may need to be deleted (manually or with ``salt-key``), and the master / minion processes restarted. Troubleshooting states ====================== The following steps typically occur when a state is executed: #. Render template (such as Jinja) #. Compile YAML to “high-data” #. Compile “high data” to “low data” #. Execute each low chunk until complete To diagnose problems, try performing single steps along this execution. For example, we can render the Jinja with the following: .. code-block:: shell salt slsutil.renderer default_renderer=jinja Jinja and YAML (in order) are rendered by default. .. code-block:: shell salt slsutil.renderer The above commands are for rendering the file from a Jinja / YAML perspective. The rendered SLS is similar since the state system will render Jinja / YAML. However, this command includes keys added during the state system render phase: .. code-block:: shell salt state.show_sls This can be a useful command especially when passing pillars from the command line: .. code-block:: shell salt state.show_sls pillar='{"foo": "bar"}' Salt states can be executed in test mode, by setting the flag :code:`test=True`: .. code-block:: shell salt state.apply test=True Here the state is rendered and evaluated as much as possible, yet changes are not applied to the remote device. Adding logging to custom modules ================================ After creating a custom module, add logging in the normal python way: .. code-block:: python :caption: /srv/salt/_modules/my_custom_module.py import logging log = logging.getLogger(__name__) def hello(): log.debug("hello is running...") return "hello" In the above example, set the log level to ``debug`` or lower. Debugging reactor states ======================== The best way to debug Reactor States is to use multiple views into the Salt master and then fire some specific events: * Monitor the Salt master log file: ``/var/log/salt/master``. This will help to debug Jinja / YAML errors in the Reactor SLS file. * Monitor the Event Bus: ``salt-run state.event pretty=True``. Watch for the triggering event as well as the Reactor events. * Fire events manually to trigger the Reactor. Problems with sync/cache ======================== Salt stores all modules and other files in the minions cache, typically located under ``/var/cache/salt``. If custom modules aren’t behaving the way you think they should, it's possible that the cache is not up to date. All custom modules must be synced to the minions: .. code-block:: shell salt \* saltutil.sync_all But if a file is deleted, you must also clear the cache and re-sync: .. code-block:: shell salt \* saltutil.clear_cache salt \* saltutil.sync_all When in doubt, it may be helpful to restart the minion process: .. code-block:: shell systemctl restart salt-minion Troubleshoot rendering ====================== Render json file ---------------- Loads json data from an absolute path: .. code-block:: shell salt \* jinja.import_JSON /srv/salt/foo.json Render YAML file ---------------- Loads YAML data from an absolute path: .. code-block:: shell salt \* jinja.import_yaml /srv/salt/foo.yaml Render a map file ----------------- Assuming the map is loaded in your formula SLS as follows: .. code-block:: sls {% from "foo/map.jinja" import bar with context %} Then the following syntax can be used to render the map variable ``bar``: .. code-block:: shell $ salt \* jinja.load_map /srv/salt/foo/map.jinja bar