Salt SSH#

Execute salt commands and states over ssh without installing a salt-minion.


Getting Started#

Salt SSH is very easy to use, simply set up a basic roster file of the systems to connect to and run salt-ssh commands in a similar way as standard salt commands.

  • Salt ssh is considered production ready in version 2014.7.0

  • Python is required on the remote system (unless using the -r option to send raw ssh commands). The python version requirement is the same as that for a standard Salt installation.

  • On many systems, the salt-ssh executable will be in its own package, usually named salt-ssh

  • The Salt SSH system does not supersede the standard Salt communication systems, it simply offers an SSH-based alternative that does not require ZeroMQ and a remote agent. Be aware that since all communication with Salt SSH is executed via SSH it is substantially slower than standard Salt with ZeroMQ.

  • At the moment fileserver operations must be wrapped to ensure that the relevant files are delivered with the salt-ssh commands. The state module is an exception, which compiles the state run on the master, and in the process finds all the references to salt:// paths and copies those files down in the same tarball as the state run. However, needed fileserver wrappers are still under development.

Salt SSH Roster#

The roster system in Salt allows for remote minions to be easily defined.

Note

See the SSH roster docs for more details.

Simply create the roster file, the default location is /etc/salt/roster:

web1: 192.168.42.1

This is a very basic roster file where a Salt ID is being assigned to an IP address. A more elaborate roster can be created:

web1:
  host: 192.168.42.1 # The IP addr or DNS hostname
  user: fred         # Remote executions will be executed as user fred
  passwd: foobarbaz  # The password to use for login, if omitted, keys are used
  sudo: True         # Whether to sudo to root, not enabled by default
web2:
  host: 192.168.42.2

Note

sudo works only if NOPASSWD is set for user in /etc/sudoers: fred ALL=(ALL) NOPASSWD: ALL

Deploy ssh key for salt-ssh#

By default, salt-ssh will generate key pairs for ssh, the default path will be /etc/salt/pki/master/ssh/salt-ssh.rsa. The key generation happens when you run salt-ssh for the first time.

You can use ssh-copy-id, (the OpenSSH key deployment tool) to deploy keys to your servers.

ssh-copy-id -i /etc/salt/pki/master/ssh/salt-ssh.rsa.pub user@server.demo.com

One could also create a simple shell script, named salt-ssh-copy-id.sh as follows:

#!/bin/bash
if [ -z $1 ]; then
   echo $0 user@host.com
   exit 0
fi
ssh-copy-id -i /etc/salt/pki/master/ssh/salt-ssh.rsa.pub $1

Note

Be certain to chmod +x salt-ssh-copy-id.sh.

./salt-ssh-copy-id.sh user@server1.host.com
./salt-ssh-copy-id.sh user@server2.host.com

Once keys are successfully deployed, salt-ssh can be used to control them.

Alternatively ssh agent forwarding can be used by setting the priv to agent-forwarding.

Calling Salt SSH#

Note

salt-ssh on target hosts without Python 3

The salt-ssh command requires at least python 3, which is not installed by default on some target hosts. An easy workaround in this situation is to use the -r option to run a raw shell command that installs python26:

salt-ssh centos-5-minion -r 'yum -y install epel-release ; yum -y install python26'

Note

salt-ssh on systems with Python 3.x

Salt, before the 2017.7.0 release, does not support Python 3.x which is the default on for example the popular 16.04 LTS release of Ubuntu. An easy workaround for this scenario is to use the -r option similar to the example above:

salt-ssh ubuntu-1604-minion -r 'apt update ; apt install -y python-minimal'

The salt-ssh command can be easily executed in the same way as a salt command:

salt-ssh '*' test.version

Commands with salt-ssh follow the same syntax as the salt command.

The standard salt functions are available! The output is the same as salt and many of the same flags are available. Please see Salt SSH reference for all of the available options.

Raw Shell Calls#

By default salt-ssh runs Salt execution modules on the remote system, but salt-ssh can also execute raw shell commands:

salt-ssh '*' -r 'ifconfig'

States Via Salt SSH#

The Salt State system can also be used with salt-ssh. The state system abstracts the same interface to the user in salt-ssh as it does when using standard salt. The intent is that Salt Formulas defined for standard salt will work seamlessly with salt-ssh and vice-versa.

The standard Salt States walkthroughs function by simply replacing salt commands with salt-ssh.

Targeting with Salt SSH#

Due to the fact that the targeting approach differs in salt-ssh, only glob and regex targets are supported as of this writing, the remaining target systems still need to be implemented.

Note

By default, Grains are settable through salt-ssh. By default, these grains will not be persisted across reboots.

See the "thin_dir" setting in Roster documentation for more details.

Configuring Salt SSH#

Salt SSH takes its configuration from a master configuration file. Normally, this file is in /etc/salt/master. If one wishes to use a customized configuration file, the -c option to Salt SSH facilitates passing in a directory to look inside for a configuration file named master.

Minion Config#

New in version 2015.5.1.

Minion config options can be defined globally using the master configuration option ssh_minion_opts. It can also be defined on a per-minion basis with the minion_opts entry in the roster.

Running Salt SSH as non-root user#

By default, Salt read all the configuration from /etc/salt/. If you are running Salt SSH with a regular user you have to modify some paths or you will get "Permission denied" messages. You have to modify two parameters: pki_dir and cachedir. Those should point to a full path writable for the user.

It's recommended not to modify /etc/salt for this purpose. Create a private copy of /etc/salt for the user and run the command with -c /new/config/path.

Define CLI Options with Saltfile#

A Saltfile is a YAML file that supplies command-line options to Salt's command-line tools, so that frequently-used options do not have to be typed on every invocation. A Saltfile is not a configuration file: it can only set options that the tool already accepts on the command line, not arbitrary minion or master configuration settings.

A Saltfile works with any of the Salt command-line tools -- salt, salt-ssh, salt-call, salt-cloud, salt-key, and so on -- and is handy when managing several different Salt projects on the same machine.

The file is keyed by the name of the command-line tool, with that tool's options nested underneath. For example, a Saltfile with the following YAML contents supplies options to salt-ssh:

salt-ssh:
  config_dir: path/to/config/dir
  ssh_log_file: salt-ssh.log
  ssh_max_procs: 30
  ssh_wipe: True

With that file in the current directory, instead of running salt-ssh --config-dir=path/to/config/dir --max-procs=30 --wipe '*' test.version you can simply run salt-ssh '*' test.version. Boolean options are given their YAML value, for example ssh_wipe: True.

Note

The keys under each tool are the destination names of the options, not the flag spelling. In most cases this is the long option name with dashes replaced by underscores -- for example --config-dir becomes config_dir. Some options have a dest that differs from the flag name; for salt-ssh the --wipe option has a dest of ssh_wipe, so the Saltfile key is ssh_wipe. Using the flag spelling (wipe: True or w: True) will not work. The option destinations are defined in the parser for each tool in salt.utils.parsers.

Because a Saltfile only supplies command-line options, settings that are valid only in a configuration file have no effect. To supply external authentication credentials to salt, for example, use the --eauth, --username and --password command-line options (whose destinations are eauth, username and password) rather than master configuration keys such as external_auth:

salt:
  eauth: pam
  username: fred
  password: saltrules

When a Salt command starts, it uses the first Saltfile it finds from the following, in order:

  1. The path given with the --saltfile option.

  2. The path in the SALT_SALTFILE environment variable.

  3. A file named Saltfile in the current working directory.

  4. ~/.salt/Saltfile.

Note

For a Saltfile in one of the automatically-searched locations to be detected, it must be named Saltfile (with a capital S) and be readable by the user running the command.

Note

Options set in a Saltfile currently take precedence over the same options passed on the command line. For example, with ssh_max_procs: 30 in the Saltfile, running salt-ssh --max-procs=1 '*' test.version still uses 30 processes.

Advanced options with salt-ssh#

Salt's ability to allow users to have custom grains and custom modules is also applicable to using salt-ssh. This is done through first packing the custom grains into the thin tarball before it is deployed on the system.

For this to happen, the config file must be explicit enough to indicate where the custom grains are located on the machine like so:

file_client: local
file_roots:
  base:
    - /home/user/.salt
    - /home/user/.salt/_states
    - /home/user/.salt/_grains
module_dirs:
  - /home/user/.salt
pillar_roots:
  base:
    - /home/user/.salt/_pillar
root_dir: /tmp/.salt-root

It's better to be explicit rather than implicit in this situation. This will allow urls all under salt:// to be resolved such as salt://_grains/custom_grain.py.

One can confirm this action by executing a properly setup salt-ssh minion with salt-ssh minion grains.items. During this process, a saltutil.sync_all is ran to discover the thin tarball and then consumed. Output similar to this indicates a successful sync with custom grains.

local:
    ----------
    ...
    executors:
    grains:
        - grains.custom_grain
    log_handlers:
    ...

This is especially important when using a custom file_roots that differ from /etc/salt/.

Note

Please see https://docs.saltproject.io/en/latest/topics/grains/ for more information on grains and custom grains.

Debugging salt-ssh#

One common approach for debugging salt-ssh is to simply use the tarball that salt ships to the remote machine and call salt-call directly.

To determine the location of salt-call, simply run salt-ssh with the -ltrace flag and look for a line containing the string, SALT_ARGV. This contains the salt-call command that salt-ssh attempted to execute.

It is recommended that one modify this command a bit by removing the -l quiet, --metadata and --output json to get a better idea of what's going on the target system.

Different Python Versions#

The 3001 release removed python 2 support in Salt. Even though this python 2 support is being dropped we have provided multiple ways to work around this with Salt-SSH. You can use the following options:

  • ssh_pre_flight

  • Using the Salt-SSH raw shell calls to install Python3.

  • Use an older version of Salt on the target host that still supports Python 2 using the feature SSH ext alternatives