Overview

In the last article I covered some of the options available to build configuration maps for different types of servers. This allows a new Minion to be effectively assigned a role and then receive the States that are matched to that role from the execution of a Highstate. But how do you persist some of those States to make sure they are not deviated from or ensure that if a change is made then the appropriate corrective action is taken? That is a job for Beacons and Reactors which are core components of open source Salt.

In this article I’m going to concentrate on Beacons but before we get into that lets take a look at how Beacons and Reactors work together to persist States on Targets.

High Level Beacon & Reactor Process

The following diagram gives you an idea of how Salt and it’s components work together to persist configuration on a Target.

1. Beacon configuration is performed by a state.apply AND/OR local config file creation
2. Salt Beacon places an event on the Event Bus of the Master when an event happens that matches the Beacon config
3. Salt Reactor monitors the Event Bus and issues a Job to apply a State when a matching event is detected
4. Target sees a Job on the Event Bus for it and applies the defined Reactor State

Now that you understand the concept of how this is supposed to work lets start by defining a Beacon.

Beacons

There are multiple methods to create a Beacon including manually creating config files on the required Minions, adding Beacon config to a Pillar (more on those later) and using State modules. As we’ve been looking at States in the previous articles and I want the process of adding Beacon configuration to a Minion to be centrally defined and controlled I’m going to cover this method.

To apply a Beacon configuration using States you can either create a managed file on the minion that includes the Beacon configuration or you can use the Beacon state module. There are reasons why you would choose one over the other but lets start with the Beacon state module.

Here I have created a State File on my fileserver which will create a Beacon for monitoring a service, in this case crond.

#State file for applying a beacon state for the crond service
 crond_beacon: #The state ID
   beacon.present: #Beacon State module and function
     - name: service #Relevant Beacon module name
     - save: False   #Write to file
     - enabled: True
     - services: 
         crond: #Name of the service
           onchangeonly: True #Stops event on Minion start

Line 4 is something you should play close attention to. There are many Beacon modules, each one providing Beacon functionality for a different things such as disk space, packages, processes etc. For a full list take a look here https://docs.saltproject.io/en/latest/ref/beacons/all/index.html.

So in the above example I am using the Beacon State module to apply a Beacon that uses the “service” Beacon module. By electing not to “save” the Beacon I am saying to a Target to only use this configuration during the current runtime. If the Minion service of a Target where to stop and re-start my Beacon would not persist. Currently there is a bug when using the Beacon State module and the “save: True” option which results in the Beacon config not being able to be processed by the Minion so please verify whether that bug effects you if you are going to use this method.

There are other options that can be used with the “service” Beacon module including delaying events and adding unclean shutdown data to a generated event if the service being monitored crashes and leaves a pid file behind. It’s worth checking the documentation of the Beacon module you are going to use to make sure you are aware of any module specific options that are available.

Testing a Beacon

Now that I have a Beacon configuration created I can apply it a Target. To do this I am just going to add the State path and name to my Top File for my Example environment (last line). Then I can use a Highstate to apply the State to my Target.

#Top file for Example environment

Example:
  '*client*':
    - directory.file
    - beacons.services.crond

CAUTION: If you apply multiple State files to a Target and they all use the same Beacon module (i.e. service, ps etc.) then they will NOT be merged together. So if you have 2 States, one for monitoring the apache service and the other for monitoring crond. When both States are applied only one of the 2 Beacons will exist. This is a known issue. If you want to Beacon multiple items that require the same Beacon module then they need to be defined in the same State.

Once the State has been applied you can verify the Minion configuration for Beacons by jumping onto the console for the Target and issuing a command to list all the defined Beacons.

So the configuration is good but I need to know whether the Beacon event will be received by the Master and how the event payload will look so I can start planning what I want a Reactor to do and what information it will have available. To do this I am going to start the salt event runner on the command line of my Master and then stop my crond service on my Minion to see what happens.

Here’s the event runner starting. This will show every event that enters the event bus.

After I stop the crond service the following event appears. The payload contains a timestamp, the status of the service, the Minion ID where the event came from and the service name.

So that is the basics of a Beacon. Remember you can establish the same thing by using a “file.managed” State and writing out a conf file to /etc/salt/minion.d/ which will prevent any bugs from the Beacon State module.

In the next article I’m going to build on my Beacon by adding a corresponding Reactor to my Master.