About spin

Spin is a task runner that aims so solve the problems of provisioning development environments and standardizing workflows. Spin also automates the provisioning of tools and other development requirements (including language stacks like Python, Javascript/Node). As an example, for a project that uses Python and Javascript, spin would:

  • provision the requested version of Python

  • provision the requested version of Node

  • create a virtual development environment, in which the required versions of Python and Node can be used

  • install tools and dependencies for development, testing, etc.

All with a single command: spin provision!

Second, spin standardizes workflows, best practices and how development tools are used, especially in a development group with many similar projects that share practices and tools. It’s plugin-based architecture allows to define workflows executing multiple task in sequence using a single command.

By default, spin will automatically generate the right options and arguments for the tools it runs, and show the user the precise commands. As a result, anyone will be able to check out any project, run spin provision and will be all set - Running a project’s test suite becomes as simple as doing spin test etc.

Spin’s plugin system

The knowledge of how to do all this comes from two places: reusable plugins and project-specific settings. spin has a plugin system, where reusable bits are encapsulated in plugins like csspin_python.python, csspin_python.pytest, csspin_frontend.node etc.

Plugins automatically provision the tools they need, come with meaningful default settings, provide new subcommands to spin (e.g. spin pytest will launch pytest in the development environment), and hook into generic workflows. For example, the csspin_python.pytest plugin automatically hooks into the generic spin test command in case csspin_workflows.stdworkflows is loaded. If your project one day decides to replace pytest with something else, spin test will still do the right thing.

spin has a small set of built-in plugins for example to run a shell command in the project context. It’s also easy to add local plugins to a project, or create shared plugins that live as Python plugin-packages on some Python package server or in a Git repository. A few of those addressing individual topics are listed below.

Selection of available plugin-packages

Package name

Description

csspin_ce

required for CE16 development

csspin_workflows

collection of standard workflows

csspin_frontend

the frontend development kit

csspin_java

Java ist auch eine Insel

csspin_python

a must for Python development

Spin’s configuration tree system

Spins configuration tree system manages project-specific and user-global settings in a neat, hierarchical structure in form of a supercharged OrderedDict which enables dot notation access from within plugins, parent linking as well as location tracking useful for debugging project configurations.

Spin ships its own part of the configuration tree of which mosts it’s properties are directly assigned below spin:

Excerpt of spins builtin configuration tree
spin:
  data: Path('/home/developer/.local/share/spin')
  extra_index: None
  spinfile: Path('/home/developer/src/qs/spin/csspin/spinfile.yaml')
  ...

Plugins configured in project and user settings ship their own configuration and thus extend the configuration tree.

A plugin extending the configuration tree
spin:
  ...
myplugin:
  setting1: ...
  setting2: ...

Spin is building the configuration tree during its execution by using the following sources:

  1. Default configuration that spin ships

  2. User specific configuration in form of a global.yaml (optional, see Writing global.yaml)

  3. Project configuration provided by spinfile.yaml (see Writing spinfile.yaml)

  4. Environment variables that updates or extends the configuration

  5. Command-line arguments and options to update or extend the configuration

Provisioning a project using spin

The choice of plugins to use, and other project-specific settings go into a file called spinfile.yaml in your project’s root directory. Spin is just a task-runner, so lets take a most simple Python project as an example to perform the provisioning.

Minimal spinfile.yaml for a Python project “foo”
spin:
  project_name: foo
plugin_packages:
  - csspin_python
plugins:
  - csspin_python.python
python:
  version: 3.10.19

The spin.project_name property tells spin the name of the project we’re working on. Setting it may not be required, but is always recommended to avoid errors where a project’s directory name differs from the project name, for example if a project foo has been cloned into the directory foo_new.

The plugin_packages key lists plugin-packages that are installed using pip into a project-specific plugin directory (which notably is different from the project’s virtual environment, in case it is a Python project).

plugins is a list of Python modules of plugin-packages or local modules, that are imported by spin and implement spin plugins. In this case, csspin_python.python is a plugin from the csspin_python plugin-package, that provides Python to a project. The python section is read by the Python plugin, and version specifies the release of the Python interpreter that this project wants to use.

Provisioning this project would download the csspin_python plugin-package and its dependencies, install Python 3.10.19 and create a virtual environment from it to then add the current project as editable install:

Provision a Python project using spin
$ spin provision
spin: mkdir /home/developer/src/qs/spin/csspin/.spin/plugins
spin: /home/developer/src/qs/spin/csspin/venv/bin/python3.12 -mpip install -q -t /home/developer/src/qs/spin/csspin/.spin/plugins csspin_python
spin: set PYTHON_BUILD_CACHE_PATH=/home/developer/.local/share/spin/pyenv_cache
spin: set PYTHON_CFLAGS=-DOPENSSL_NO_COMP
spin: /home/developer/.local/share/spin/pyenv/plugins/python-build/bin/python-build 3.10.19 /home/developer/.local/share/spin/python/3.10.19
Downloading Python-3.10.19.tar.xz...
-> https://www.python.org/ftp/python/3.10.19/Python-3.10.19.tar.xz
Installing Python-3.10.19...
Installed Python-3.10.19 to /home/developer/.local/share/spin/python/3.10.19
spin: /home/developer/src/qs/spin/csspin/venv/bin/python3.12 -mvirtualenv -q -p /home/developer/.local/share/spin/python/3.10.19/bin/python /home/developer/src/qs/spin/csspin/.spin/venv
spin: activate /home/developer/src/qs/spin/csspin/.spin/venv
spin: python -mpip -q install -U pip
spin: pip install -q -e .

In this case, Python was provisioned using pyenv by downloading, caching and compiling the distribution to create a Python virtual environment in which the current package under development is installed. spin can handle other stacks like Java and Node within the same venv, depending on their implementation.

Now you want to test your project using pytest. All that is necessary (besides writing the tests), is to add the csspin_python.pytest plugin to spinfile.yaml:

Minimal spinfile.yaml to run the pytest plugin
spin:
  project_name: foo
plugin_packages:
  - csspin_python
plugins:
  - csspin_python.pytest
python:
  version: 3.10.19

Spin will resolve the dependency from csspin_python.pytest to csspin_python.python without the need to define both plugins within spinfile.yaml.

Provisioning again will automatically install pytest and other packages that csspin_python.pytest depends on from PyPI:

Provision the csspin_python.pytest plugin as well as its dependencies
$ spin provision
spin: /home/developer/src/qs/spin/csspin/venv/bin/python3.12 -mpip install -q \
    -t /home/developer/src/qs/spin/csspin/.spin/plugins \
    csspin_python
spin: activate /home/developer/src/qs/spin/csspin/.spin/venv
spin: pip install -q pytest-cov pytest
spin: pip install -q -e .

After provisioning, spin gained a new subcommand pytest:

Execute the pytest subcommand
$ spin pytest
spin -p pytest.tests=tests pytest
spin: activate /home/developer/src/qs/spin/csspin/.spin/venv
spin: pytest tests
======================= test session starts =================================
platform linux -- Python 3.10.19, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/developer/src/qs/spin/csspin
configfile: pyproject.toml
plugins: cov-5.0.0
collected 113 items
tests/integration/test_provisioning.py ....
...

After a while your project has been promoted to become a company-wide standard, and thus it is required to follow your group’s best practices. Luckily, your team already has created a custom spin plugin-package that comes with all the tools and settings required. You can simply add that plugin to your spinfile.yaml:

spinfile.yaml defining a plugin-package from a git-repository
 1spin:
 2  project_name: foo
 3plugin_packages:
 4  - git+https://git.example.com/projstds#egg=projstds
 5  - csspin_python
 6plugins:
 7  - csspin_python.pytest
 8  - mycompany.projstds
 9python:
10  version: 3.10.19
11projstds:
12  # Plugin settings goes here

The plugin_packages key lists plugin-packages that are installed using pip into a project specific plugin directory (which notably is different from the project’s virtual environment, in case it is a Python project). Line 6 makes spin import and use the plugin module mycompany.projstds that has been installed from the Git URL defined in line 2.

Your team’s projstds plugin comes with lots of tools and predefined settings, among them pre-commit: note how spin automatically installs all the tools and sets up the pre-commit hooks.

Provisioning a plugin-package from a git-repository
$ spin provision
spin: /home/developer/src/qs/spin/csspin/venv/bin/python3.12 -mpip install -q \
    -t /home/developer/src/qs/spin/csspin/.spin/plugins \
    csspin_python \
    git+https://git.example.com/projstds#egg=projstds
spin: activate /home/developer/src/qs/spin/csspin/.spin/venv
spin: pip -q install pytest pre-commit flake8 black flake8-isort ...
spin: pre-commit install
pre-commit installed at .git/hooks/pre-commit

This is a basic pattern when working with spin: you modify your environment by editing spinfile.yaml and let spin re-provision the environment.

Most Frequently Asked Questions

Why not …?

There are many tools that do things similar to spin, e.g. it is customary to have standardized targets like clean, all, dist etc. for Unix Makefiles. Alas, we were not aware of tools that at the same time:

  • Are platform and technology stack independent: spin works with Python, Java, Node and C/C++ projects. Other stacks can be added by creating plugins.

  • Can provision other software.

  • Allow for re-usable definitions, that can be shared between many projects.

  • Don’t suck ;-)

Spin explicitly does not aim to be a build tool like GNU Make, CMake or SCons, nor does it try to replace or improve other tools or tech stacks: it is just a unpretentious way to store and re-use the knowledge and conventions for installing and running development tools.

Is it necessary to run everything via spin?

Absolutely not! spin intentionally echoes the verbatim commands it runs, to make users understand what is going on. It also provides activation commands for development environments, to enable users to “switch” to an environment provisioned by spin, and run arbitrary commands themselves. Spin plugins try to be well-behaved in this regard, and do not silently modify the process environment, to make everything that is going on transparent to the user.

Why YAML?

Good question. The original author Frank Patz-Brockmann wasn’t inclined to write a parser for this project, and YAML seemed like the choice that sucked least: it has comments, it is well supported by text editors, and its data model blends naturally with the configuration tree paradigm of spin. YAML has the same information model as JSON: supported data types include dictionaries, lists and literals (mostly strings).

However, YAML is a complex beast. You can do all kinds of mischievous tricks with YAML, and if you mess up the tree, the spin command will most likely fail to run.

We also concluded that the standard python config files setup.cfg or pyproject.toml aren’t quite fitting, as spin’s configuration tree paradigm is by far better visually recognizable in the spinfile.yaml.