salt.fileserver.s3fs#

Amazon S3 Fileserver Backend

New in version 0.16.0.

This backend exposes directories in S3 buckets as Salt environments. To enable this backend, add s3fs to the fileserver_backend option in the Master config file.

fileserver_backend:
  - s3fs

S3 credentials must also be set in the master config file:

s3.keyid: GKTADJGHEIQSXMKKRBJ08H
s3.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs

Alternatively, if on EC2 these credentials can be automatically loaded from instance metadata.

Regional endpoints#

By default s3fs talks to the global endpoint (s3.amazonaws.com) and relies on AWS to redirect to the bucket's home region. Redirected requests sometimes fail to carry the SigV4 signature, producing errors such as:

No AWSAccessKey was presented

or:

The authorization header is malformed; the region 'us-east-1' is wrong;
expecting 'us-west-2'

To avoid the redirect, set s3.location (preferred — passed straight through to the SigV4 signer) or s3.service_url (point to a regional or non-AWS S3 endpoint):

# Tell s3fs the bucket lives in us-west-2
s3.location: us-west-2

# Alternative: hit the regional endpoint directly
s3.service_url: s3.us-west-2.amazonaws.com

# Force HTTPS and standard hostname-style URLs (defaults are True/False)
s3.https_enable: true
s3.path_style: false

# Verify the TLS certificate (default true). Disable only for
# private S3-compatible deployments with self-signed certificates.
s3.verify_ssl: true

The same options are recognized by the s3 external pillar.

Use s3.location for AWS regions. Use s3.service_url when targeting an S3-compatible service such as MinIO, Ceph RGW, or Wasabi; set s3.path_style: true if the service requires path-style addressing.

This fileserver supports two modes of operation for the buckets:

  1. A single bucket per environment

    s3.buckets:
  production:
    - bucket1
    - bucket2
  staging:
    - bucket3
    - bucket4

  2. Multiple environments per bucket

    s3.buckets:
  - bucket1
  - bucket2
  - bucket3
  - bucket4

Note that bucket names must be all lowercase both in the AWS console and in Salt, otherwise you may encounter SignatureDoesNotMatch errors.

A multiple-environment bucket must adhere to the following root directory structure:

s3://<bucket name>/<environment>/<files>

Note

This fileserver back-end requires the use of the MD5 hashing algorithm. MD5 may not be compliant with all security policies.

Note

This fileserver back-end is only compatible with MD5 ETag hashes in the S3 metadata. This means that you must use SSE-S3 or plaintext for bucket encryption, and that you must not use multipart upload when uploading to your bucket. More information here: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html

Objects without an MD5 ETag will be fetched on every fileserver update.

If you deal with objects greater than 8MB, then you should use the following AWS CLI config to avoid mutipart upload:

s3 =
  multipart_threshold = 1024MB

More info here: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html

Note

This fileserver back-end will by default sync all buckets on every fileserver update.

If you want files to be only populated in the cache when requested, you can disable this in the master config:

s3.s3_sync_on_update: False
On this page