Overview

In the last article I talked about how we can deploy a configuration onto a Salt Target that allows the Target to send events onto the event bus when defined criteria are met. This type of configuration is known as a Beacon and might be defined for a config deviation, a service crash, package version change etc. The partner component to a Beacon is known as a Reactor and it’s job is to recognize when a Beacon that matches its criteria enters onto the event bus and execute a defined response.

In this article I am going to be covering Reactors, what they can do, how and where to configure them, together with an example.

The Reactor

The Reactor is a component of the Salt Master that has an interface onto the Salt Master event bus. This interface allows it to see the events as they enter the bus and from examining the event payload contents work out whether any actions need to be carried out.

To add a Reactor configuration to a Salt Master you essentially need to add it to the default Master config file or place a config file in the “/etc/salt/master.d/” directory that has the relevant content. Here’s an example Reactor configuration.

reactor:                                       # Master config section "reactor"
  - 'salt/minion/saltclient.corp.local/start': # An event tag to look for
    - /srv/reactor/start.sls                   # A state to apply when the tag is matched

Every event that enters the event bus has a tag associated with it. Lets look at the Beacon event from the previous article as a specific example.

The event tag is the first line of the event so in the example above the tag would be “salt/beacon/saltclient.corp.local/service/crond”. Mixed into the tag you can see the Minion ID. If I was to define a Reactor using the tag from the above event as it is then the Reactor would only execute an action when the crond service status changes for the Minion “saltclient.corp.local”. That’s great if you are only interested in a single Minion but what if you want to use a Reactor for multiple Targets that all have the same role? Well in that case you need to look at generalizing the tag.

Here I have used an asterisks instead of the Minion ID which in this case will allow the Reactor to match all events issued from a Beacon defined for the “crond” service status change. You might equally use other glob pattern matching symbols such as ?, [a-c],[abc] etc.

reactor: 
  - 'salt/beacon/*/service/crond':
    - /srv/reactor/restart.sls

The State to apply when the tag is matched is just a fileserver path to a State file. The example above assumes that the Salt Master is using a local file server and not one provided by SaltStack Config or an external provider such as Github etc. In my lab environment I am using the SaltStack Config fileserver so lets see how the above Reactor config would need to change.

The highlighted line shows that I am now specifying the fileserver path by using the “salt://” prefix. This means that the Salt Master will search all configured fileservers on the Master for the State file specified that is located in the Environment specified (“Example” in this case). If the “saltenv” option was not provided then the search would be performed within the default “base” environment only.

reactor: 
  - 'salt/beacon/*/service/crond':
    - salt://reactors/example.sls?saltenv=Example

There Must Be A Better Way?

Well there is of course. If only we could define the contents of that configuration up front and use some configuration management system to place it onto our Master server….Yes, that’s right. I’m going to treat the Salt Master like any other Target since it also has a Salt Minion service running on it.

First I’m going to create a directory on my fileserver to hold the Master configuration (both State file and configuration file). The configuration file contains the contents I want Salt to place into “/etc/salt/master.d/” when a State is applied to it. I have created it in exactly the same format /layout.

The next thing is to create a State file to take the above contents it and apply it to the Master via its own Minion. As my source file is located on my SaltStack Config fileserver I can use the salt protocol prefix to access the configuration file within a State as shown below. Note that the environment specification is not required in this instance as by not specifying one it implies to search the current environment.

I’m not quite finished with this though as the Master only reads its configuration files when it starts up. Any changes after the service starts are not read until a subsequent restart happens. So now I need to add a bit more to my State File to accommodate this. I’ve added a State (“masterServiceRestart”) to put the “salt-master” service into a running state. This by itself will not restart the “salt-master” service, only ensure that it is running. However, by adding a “watch” requisite to that State (line 9) I can ensure that the “masterServiceRestart” State will only apply if “createReactorConfig” State generates a change AND it will restart the service. This is a unique feature of the “watch” requisite when applied to a service.

createReactorConfig:
  file.managed:
    - name: /etc/salt/master.d/reactorTest.conf
    - source: salt://master/reactorconfig.conf
    
masterServiceRestart:
  service.running:
    - name: salt-master
    - watch:
      - file: createReactorConfig

The last part of the configuration for the Master is to amend the current Top File to ensure that when the Master runs a Highstate it picks up the Reactor State and applies it. In this instance I have specified the Minion ID of the Salt Master in its entirety.

The Reactor State

Up until now I have just been talking about the Master configuration, making sure it has the correct configuration to identify an event. In the Reactor configuration I referenced a State file (salt://reactors/example.sls) which would be used to apply one or more States upon a matching event tag being seen by the Reactor on the event bus. In this section I’m going to now build out that State file which is slightly different to the other State file examples I have shown so far.

One of the key differences for a Reactor State File vs a traditional State File are the types of Reactions. A Reactor State File classes its States as one of four types:

• local – runs a remote function on targeted Minions
• runner – executes runner functions from runner modules on the Master itself (for things like orchestration, cloud config etc.) – see https://docs.saltproject.io/en/latest/ref/runners/all/index.html#all-salt-runners for different types of runner modules and their functions
• wheel – Master configuration management (keys, file_roots, pillars, minions etc.)
• caller – remote execution on a Minions Reactor system

When you create a State in for a Reactor the configuration must include the state classification so that the Reactor knows how to handle the State. In my example I am going to be using Reactor Local States as I want something to happen on a Minion not on a Master. Note that just because I am using “local” does not automatically mean my Reactor State will apply to the same Minion that first generated my Beacon event, but we’ll get to that in a minute.

First lets look at my example Reactor State File.

crondReaction:
  local.state.single:
    - tgt: saltclient.corp.local
    - args:
      - fun: file.managed
      - name: /tmp/reactor.tst

The layout of my State starts with an ID just like any other State but from then on things are not quite the same. The next line (line 2) defines the type of Reactor combined with a module and a function. So “local.state.single” means using the local reactor with the “state” module and it’s “single” function (the “single” function is used to apply a single state to a target).

The next part (line 3) tells the Reactor which Target to execute on. We need to specify this as nobody (i.e an admin etc) is telling Salt the Target to use. Normally when you apply an ordinary State the admin selects the Target (either via GUI or by specifying a Target definition on the CLI) but this is not the case with a Reactor. The Target does not have to be the same Target that issued the Beacon event, it could be a completely different Target. In this instance I have hard-coded the Target Minion ID but we can be a bit cleverer here if desired (more on that later).

The rest of the State defines the State Module and Function that should be executed. In this case I want to created a managed file in “/tmp” so my function name (“fun”) is “file.managed” and “name” is the name of the file I want to create.

Now what if I have many servers that have my “crond” Beacon configured. In that case the Reactor State needs to be adjusted so that the Target is taken from the event data. The event data is available to the Reactor State and with some basic Jinja templating you can generalize the Reactor State file so that it will work with any Target ID contained within a matching event.

Lets take another look at the event from the Beacon. The Target ID is contained within a dictionary (referenced as “data”) and is stored against the the “id” key. So to access that key I need to specify the path to that ID key in JSON format (the event structure is in JSON). This would be “data[‘id]”.

Line 3 shows that I am now using Jinja (denoted by the curly braces) and I have inserted my JSON path to the ID key. Depending on the structure of the event and the data it contains the Jinja template would vary accordingly.

crondReaction:
  local.state.single:
    - tgt: {{ data['id'] }}
    - args:
      - fun: file.managed
      - name: /tmp/reactor.tst

This completes the Beacon and Reactor configuration for my simple use case.