Creating external pillars
As you know, pillars are like grains, with a key difference: grains are defined on the Minion, whereas pillars are defined for inpidual Minions, from the Master.
As far as users are concerned, there's not a whole lot of difference here, except that pillars must be mapped to targets on the Master, using the top.sls file in pillar_roots. One such mapping might look like this:
# cat /srv/pillar/top.sls base: '*': - test
In this example, we would have a pillar called test defined, which might look like this:
# cat /srv/pillar/test.sls test_pillar: True
Dynamic pillars are still mapped in the top.sls file, but that's where the similarities end, so far as configuration is concerned.
Configuring external pillars
Unlike dynamic grains, which will run so long as their __virtual__() function allows them to do so, pillars must be explicitly enabled in the master configuration file. Or, if running in local mode as we will be, in the minion configuration file. Let's go ahead and add the following lines to the end of /etc/salt/minion:
ext_pillar: - test_pillar: True
If we were testing this on the Master, we would need to restart the salt-master service. However, since we're testing in local mode on the Minion, this will not be required.
Adding an external pillar
We'll also need to create a simple external pillar to get started with. Go ahead and create salt/pillar/test_pillar.py with the following content:
'''
This is a test external pillar
'''
def ext_pillar(minion_id, pillar, config):
'''
Return the pillar data
'''
return {'test_pillar': minion_id}
Go ahead and save your work, and then test it to make sure it works:
# salt-call --local pillar.item test_pillar local: ---------- test_pillar: dufresne
Let's go over what's happened here. First off, we have a function called ext_pillar(). This function is required in all external pillars. It is also the only function that is required. Any others, whether or not named with a preceding underscore, will be private to this module.
This function will always be passed three pieces of data. The first is the ID of the Minion that is requesting this pillar. You can see this in our example already: the minion_id where the earlier example was run was dufresne. The second is a copy of the static pillars defined for this Minion. The third is an extra piece of data that was passed to this external pillar in the master (or in this case, minion) configuration file.
Let's go ahead and update our pillar to show us what each component looks like. Change your ext_pillar() function to look like:
def ext_pillar(minion_id, pillar, command):
'''
Return the pillar data
'''
return {'test_pillar': {
'minion_id': minion_id,
'pillar': pillar,
'config': config,
}}
Save it, and then modify the ext_pillar configuration in your minion (or master) file:
ext_pillar: - test_pillar: Alas, poor Yorik. I knew him, Horatio.
Take a look at your pillar data again:
# salt-call --local pillar.item test_pillar local: ---------- test_pillar: ---------- config: Alas, poor Yorik. I knew him, Horatio. minion_id: dufresne pillar: ---------- test_pillar: True
You can see the test_pillar that we referenced a couple of pages ago. And of course, you can see minion_id, just like before. The important part here is config.
This example was chosen to make it clear where the config argument came from. When an external pillar is added to the ext_pillar list, it is entered as a dictionary, with a single item as its value. The item that is specified can be a string, boolean, integer, or float. It cannot be a dictionary or a list.
This argument is normally used to pass arguments into the pillar from the configuration file. For instance, the cmd_yaml pillar that ships with Salt uses it to define a command that is expected to return data in YAML format:
ext_pillar: - cmd_yaml: cat /etc/salt/testyaml.yaml
If the only thing that your pillar requires is to be enabled, then you can just set this to True, and then ignore it. However, you must still set it! Salt will expect that data to be there, and you will receive an error like this if it is not:
[CRITICAL] The "ext_pillar" option is malformed
Tip
Although minion_id, pillar, and config are all passed into the ext_pillar() function (in that order), Salt doesn't actually care what you call the variables in your function definition. You could call them Emeril, Mario, and Alton if you wanted (not that you would). But whatever you call them, they must still all be there.
Another external pillar
Let's put together another external pillar, so that it doesn't get confused with our first one. This one's job is to check the status of a web service. First, let's write our pillar code:
'''
Get status from HTTP service in JSON format.
This file should be saved as salt/pillar/http_status.py
'''
import salt.utils.http
def ext_pillar(minion_id, pillar, config):
'''
Call a web service which returns status in JSON format
'''
comps = config.split()
key = comps[0]
url = comps[1]
status = salt.utils.http.query(url, decode=True)
return {key: status['dict']}
You've probably noticed that our docstring states that This file should be saved as salt/pillar/http_status.py. When you check out the Salt codebase, there is a directory called salt/ that contains the actual code. This is the directory that is referred to in the docstring. You will continue to see these comments in the code examples throughout this book.
Save this file as salt/pillar/http_status.py. Then go ahead and update your ext_pillar configuration to point to it. For now, we'll use GitHub's status URL:
ext_pillar - http_status: github https://status.github.com/api/status.json
Go ahead and save the configuration, and then test the pillar:
# salt-call --local pillar.item github local: ---------- github: ---------- last_updated: 2015-12-02T05:22:16Z status: good
If you need to be able to check the status on multiple services, you can use the same external pillar multiple times, but with different configurations. Try updating your ext_pillar definition to contain two entries:
ext_pillar - http_status: github https://status.github.com/api/status.json - http_status: github2 https://status.github.com/api/status.json
Now, this can quickly become a problem. GitHub won't be happy with you if you're constantly hitting their status API. So, as nice as it is to get real-time status updates, you may want to do something to throttle your queries. Let's save the status in a file, and return it from there. We will check the file's timestamp to make sure it doesn't get updated more than once a minute.
Let's go ahead and update the entire external pillar:
'''
Get status from HTTP service in JSON format.
This file should be saved as salt/pillar/http_status.py
'''
import json
import time
import datetime
import os.path
import salt.utils.http
def ext_pillar(minion_id, # pylint: disable=W0613
pillar, # pylint: disable=W0613
config):
'''
Return the pillar data
'''
comps = config.split()
key = comps[0]
url = comps[1]
refresh = False
status_file = '/tmp/status-{0}.json'.format(key)
if not os.path.exists(status_file):
refresh = True
else:
stamp = os.path.getmtime(status_file)
now = int(time.mktime(datetime.datetime.now().timetuple()))
if now - 60 >= stamp:
refresh = True
if refresh:
salt.utils.http.query(url, decode=True, decode_out=status_file)
with salt.utils.fopen(status_file, 'r') as fp_:
return {key: json.load(fp_)}
Now we've set a flag called refresh, and the URL will only be hit when that flag is True. We've also defined a file that will cache the content obtained from that URL. The file will contain the name given to the pillar, so it will end up having a name like /tmp/status-github.json. The following two lines will retrieve the last modified time of the file, and the current time in seconds:
stamp = os.path.getmtime(status_file)
now = int(time.mktime(datetime.datetime.now().timetuple()))
And comparing the two, we can determine whether the file is more than 60 seconds old. If we wanted to make the pillar even more configurable, we could even move that 60 to the config parameter, and pull it from comps[2].