SpiroFS

SpiroFS is the Continuous Deployment system that SaltStack has been missing.

SpiroFS is a fileserver backend for Salt focusing on deployment and saltenv management. It allows automated systems to push deployment bundles to Salt without giving them undue permissions. It is dynamic and does not require on-going reconfiguration as your technology stack grows and changes.

SpiroFS makes use of managed saltenvs and generated topfiles. This explicit management simplifies states and configurations, while allowing systems to operate more securely. Juggling project and version names, hoping not to collide with something, is no longer necessary.

Each deployment environment (simply called deployments) managed by SpiroFS is isolated into its own saltenv. A typo, unaware moment, or just forgetfulness won't lead to misconfigured servers or cross-project conflicts.

There are several components:

  • The fileserver backend
  • The deployment server
  • A deploy script
  • Management runners

SpiroFS project

spiro-deploy project

How saltenvs are organized

SpiroFS is organized around two major concepts:

  • Projects: Roughly a repo. A group of deployments for the same software.
  • Deployments: Specific deployment/operational environment for a project.

For example, your organization may have projects foo and spam, each with prod and dev deployments. SpiroFS creates the saltenvs foo/prod, foo/dev, spam/prod, and spam/dev for these project/deployment pairings.

This allows for multiple versions of states and artifacts to co-exist in your system, eg for production and development.

Additions to saltenvs

SpiroFS largely does not care about the contents of deployments, except for two additions:

  • The spiro-deploy script adds an _artifacts directory
  • You may use spirotop.sls and SpiroFS will generate a top.sls for you

Generated topfiles

In order to facilitate multi-environment installations, SpiroFS will compile spirotop.sls files into top.sls files in deployments it manages. Its format is mostly identical to top.sls, except that the environments listed are relative to the project.

SpiroFS intelligently handles if different deployments have different versions of spirotop.sls: it will only include the data applicable to a given saltenv in that saltenv's top.sls. So if you've modified your spirotop.sls file in development, it will appear in your development environment and will not conflict with the data in the production environment, while still allowing for easy VCS merges.

An example, in project foo:

prod:
  foo-prod-www:
    - server
    - logging.prod
  foo-prod-database:
    - database
    - logging.prod

dev:
  foo-dev:
    - server
    - database
    - logging.dev

This defines the saltenvs foo/prod and foo/dev.

The Process

  1. Your normal CI runs, running tests, producing artifacts, etc
  2. You run your deploy script (eg spiro-deploy) which:
  3. Gathers up salt states, artifacts, metadata, and anything else that goes into the deployment into a bundle
  4. Uploads the bundle to the SaltFS server
  5. The SaltFS server accepts the deployment and makes it live
  6. (Optional) The SaltFS server examines the tops data and highstates potentially affected minions

Installation

Server

  1. Have pip for the Python that salt is installed into
  2. pip install spirofs (Adapt as necessary)
  3. Configure the salt master (see below)
  4. Restart the salt master

Note: libsodium is used, but is pre-packaged in the binary wheels. If these are not used on your system for whatever reason, install either libsodium-dev (apt) or libsodium-devel (yum).

Deploy Script

You have a few options for installing spiro-deploy:

  • pip3 install spiro-deploy
  • Make use of the registry.gitlab.com/spirostack/spiro-deploy:master container

Either way, they will expose a spiro-deploy script with a useful --help.

Configuration

Initial Installation

On your Salt Master:

  • Make sure that spiro is listed in the fileserver_backend setting
  • Add spiro to the engines setting to enable the deploy server

The spiro engine accepts these settings:

  • port: The TCP port to listen to
  • ssl_crt: The certificate for SSL
  • ssl_key: The private key for SSL

An example configuration:

fileserver_backend:
  - roots
  - spiro

engines:
  - spiro:
      port: 4510
      ssl_key: /etc/ssl/private/ssl-cert-snakeoil.key
      ssl_crt: /etc/ssl/certs/ssl-cert-snakeoil.pem

Danger

While the deploy server can run without SSL, it is highly recommended that you use a CA-issued certificate to prevent tampering. If you do not have one, Let's Encrypt offers free ones with automatic renewal, and Salt has the acme state module to mange them.

spiro-deploy Configuration

  1. Use the spiro.issue_token runner to produce an authentication token, eg salt-run spiro.issue_token myproject
  2. Add SPIRO_URL and SPIRO_TOKEN environment variables to your build, eg SPIRO_URL=https://master.example:4510/, SPIRO_TOKEN=deadbeef
  3. Arrange for spiro-deploy to be called inside your project, with the project and deployment names, eg spiro-deploy myproject devel

spiro-deploy expects to be run next to a _salt directory, used as the source of Salt states. There are additional flags like --artifact to include additional files. See the documentation for details.

Examples