Writing SDB modules_Extending SaltStack-QQ阅读女生现言网

上QQ阅读APP看书，第一时间看更新

Writing SDB modules

SDB is a relatively new type of module, and ripe for development. It stands for Simple Database, and it is designed to allow data to be simple to query, using a very short URI. Underlying configuration can be as complex as necessary, so long as the URI that is used to query it is as simple as possible.

Another design goal of SDB is that URIs can mask sensitive pieces of information from being stored directly inside a configuration file. For instance, passwords are often required for other types of modules, such as the mysql modules. But it is a poor practice to store passwords in files that are then stored inside a revision control system such as Git.

Using SDB to look up passwords on the fly allows references to the passwords to be stored, but not the passwords themselves. This makes it much safer to store files that reference sensitive data, inside revision control systems.

There is one supposed function that may be tempting to use SDB for: storing encrypted data on the Minion, which cannot be read by the Master. It is possible to run agents on a Minion that require local authentication, such as typing in a password from the Minion's keyboard, or using a hardware encryption device. SDB modules can be made that make use of these agents, and due to their very nature, the authentication credentials themselves cannot be retrieved by the Master.

The problem is that the Master can access anything that a Minion that subscribes to it can. Although the data may be stored in an encrypted database on the Minion, and although its transfer to the Master is certainly encrypted, once it gets to the Master it can still be read in plaintext.

Getting SDB data

There are only two public functions that are used for SDB: get and set. And in truth, the only important one of these is get, since set can usually be done outside of Salt entirely. Let's go ahead and take a look at get.

For our example, we'll create a module that reads in a JSON file and then returns the requested key from it. First, let's set up our JSON file:

{
    "user": "larry",
    "password": "123pass"
}

Go ahead and save that file as /root/mydata.json. Then edit the minion configuration file and add a configuration profile:

myjson:
    driver: json
    json_file: /root/mydata.json

With those two things in place, we're ready to start writing our module. JSON has a very simple interface, so there won't be much here:

'''
SDB module for JSON

This file should be saved as salt/sdb/json.py
'''
from __future__ import absolute_import
import salt.utils
import json


def get(key, profile=None):
    '''
    Get a value from a JSON file
    '''
    with salt.utils.fopen(profile['json_file'], 'r') as fp_:
        json_data = json.load(fp_)
    return json_data.get(key, None)

You've probably noticed that we've added a couple of extra things outside of the necessary JSON code. First, we imported something called absolute_import. This is because this file is called json.py, and it's importing another library called json. Without absolute_import, the file would try to import itself, and be unable to find the necessary functions from the actual json library.

The get() function takes two arguments: key and profile. key refers to the key that will be used to access the data that we need. profile is a copy of the profile data that we save in the minion configuration file as myjson.

The SDB URI makes use of these two items. When we build that URI, it will be formatted as:

sdb://<profile_name>/<key>

For instance, if we were to use the sdb execution module to retrieve the value of key1, our command would look like:

# salt-call --local sdb.get sdb://myjson/user
local:
 larry

With this module and profile in place, we can now add lines to the minion configuration (or to grains or pillars, or even the master configuration) that look like:

username: sdb://myjson/user
password: sdb://myjson/password

When a module that uses config.get comes across an SDB URI, it will automatically translate it on the fly to the appropriate data.

Before we move on, let's update this function a little bit to do some error handling. If the user makes a typo in the profile (such as json_fle instead of json_file), or the file being referenced doesn't exist, or the JSON isn't formatted correctly, then this module will start spitting out trace back messages. Let's go ahead and handle all of those, using Salt's own CommandExecutionError:

from __future__ import absolute_import
from salt.exceptions import CommandExecutionError
import salt.utils
import json


def get(key, profile=None):
    '''
    Get a value from a JSON file
    '''
    try:
        with salt.utils.fopen(profile['json_file'], 'r') as fp_:
            json_data = json.load(fp_)
        return json_data.get(key, None)
    except IOError as exc:
        raise CommandExecutionError (exc)
    except KeyError as exc:
        raise CommandExecutionError ('{0} needs to be configured'.format(exc))
    except ValueError as exc:
        raise CommandExecutionError (
            'There was an error with the JSON data: {0}'.format(exc)
        )

IOError will catch problems with a path that doesn't point to a real file. KeyError will catch errors with missing profile configuration (which would happen if one of the items was misspelled). ValueError will catch problems with an improperly formatted JSON file. This will turn errors like:

Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 333, in salt_call
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 58, in run
    caller.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 133, in run
    ret = self.call()
  File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 196, in call
    ret['return'] = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/modules/sdb.py", line 28, in get
    return salt.utils.sdb.sdb_get(uri, __opts__)
  File "/usr/lib/python2.7/site-packages/salt/utils/sdb.py", line 37, in sdb_get
    return loaded_db[fun](query, profile=profile)
  File "/usr/lib/python2.7/site-packages/salt/sdb/json_sdb.py", line 49, in get
    with salt.utils.fopen(profile['json_fil']) as fp_:
KeyError: 'json_fil'

...into errors like this:

Error running 'sdb.get': 'json_fil' needs to be configured

Setting SDB data

The function that is used for set may look strange, because set is a Python built-in. That means that the function may not be called set(); it must be called something else, and then given an alias using the __func_alias__ dictionary. Let's go ahead and create a function that does nothing except return the value to be set:

__func_alias__ = {
    'set_': 'set'
}


def set_(key, value, profile=None):
    '''
    Set a key/value pair in a JSON file
    '''
    return value

This will be enough for your purposes with read-only data, but in our case, we're going to modify the JSON file. First, let's look at the arguments that are passed into our function.

You already know that the key points to the data are to be referenced, and that profile contains a copy of the profile data from the Minion configuration file. And you can probably guess that value contains a copy of the data to be applied.

The value doesn't change the actual URI; that will always be the same, no matter whether you're getting or setting data. The execution module itself is what accepts the data to be set, and then sets it. You can see that with:

# salt-call --local sdb.set sdb://myjson/password 321pass
local:
 321pass

With that in mind, let's go ahead and make our module read in the JSON file, apply the new value, and then write it back out again. For now, we'll skip error handling, to make it easier to read:

def set_(key, value, profile=None):
    '''
    Set a key/value pair in a JSON file
    '''
    with salt.utils.fopen(profile['json_file'], 'r') as fp_:
        json_data = json.load(fp_)

    json_data[key] = value

    with salt.utils.fopen(profile['json_file'], 'w') as fp_:
        json.dump(json_data, fp_)

    return get(key, profile)

This function reads in the JSON file as before, then updates the specific value (creating it if necessary), then writes the file back out. When it's finished, it returns the data using the get() function, so that the user knows whether it was set properly. If it returns the wrong data, then the user will know that something went wrong. It won't necessarily tell them what went wrong, but it will raise a red flag.

Let's go ahead and add some error handling to help the user know what went wrong. We'll go ahead and add in the error handling from the get() function too:

def set_(key, value, profile=None):
    '''
    Set a key/value pair in a JSON file
    '''
    try:
        with salt.utils.fopen(profile['json_file'], 'r') as fp_:
            json_data = json.load(fp_)
    except IOError as exc:
        raise CommandExecutionError (exc)
    except KeyError as exc:
        raise CommandExecutionError ('{0} needs to be configured'.format(exc))
    except ValueError as exc:
        raise CommandExecutionError (
            'There was an error with the JSON data: {0}'.format(exc)
        )

    json_data[key] = value

    try:
        with salt.utils.fopen(profile['json_file'], 'w') as fp_:
            json.dump(json_data, fp_)
    except IOError as exc:
        raise CommandExecutionError (exc)

    return get(key, profile)

Because we did all of that error handling when reading the file, by the time we get to writing it back again, we already know that the path is value, the JSON is valid, and there are no profile errors. However, there could still be errors in saving the file. Try the following:

# chattr +i /root/mydata.json
# salt-call --local sdb.set sdb://myjson/password 456pass
Error running 'sdb.set': [Errno 13] Permission denied: '/root/mydata.json'

We've changed the attribute of the file to make it immutable (read-only), and we can no longer write to the file. Without IOError, we would get an ugly trace back message just like before. Removing the immutable attribute will allow our function to run properly:

# chattr -i /root/mydata.json
# salt-call --local sdb.set sdb://myjson/password 456pass
local:
 456pass

Using a descriptive docstring

With SDB modules, it is more important than ever to add a docstring that demonstrates how to configure and use the module. Without it, using the module is all but impossible for the user to figure out, and trying to modify a module is even worse.

The docstring doesn't need to be a novel. It should contain enough information to use the module, but not so much that figuring things out becomes confusing and frustrating. You should include not only an example of the profile data but also of an SDB URI to be used with this module:

'''
SDB module for JSON

Like all sdb modules, the JSON module requires a configuration profile to
be configured in either the minion or master configuration file. This profile
requires very little. In the example:

.. code-block:: yaml

    myjson:
      driver: json
      json_file: /root/mydata.json

The ``driver`` refers to the json module and json_file is the path to the JSON
file that contains the data.

.. code-block:: yaml

    password: sdb://myjson/somekey
'''

Using more complex configuration

It may be tempting to create SDB modules that make use of more complicated URIs. For instance, it is entirely possible to create a module that supports a URI such as:

sdb://mydb/user=curly&group=ops&day=monday

With the preceding URI, the key that is passed in would be:

user=curly&group=ops&day=monday

At that point, it would be up to you to parse out the key and translate it into something usable by your code. However, I strongly discourage it!

The more complex you make an SDB URI, the less it becomes a simple database lookup. You also risk exposing data in an unintended way. Look at the preceding key again. It reveals the following information about the database that holds the information that is supposed to be sensitive:

There is a field (abstracted or real) that is referred to as user. Since users tend to be lazier than they think, this is likely to point to a real database field called user. If that's true, then this exposes a portion of the database schema.
There is a group called ops. This means that there are other groups as well. Since ops typically refers to a team that performs server operations tasks, does that mean that there's a group called dev too? And if the dev group is compromised, what juicy pieces of data do they have for an attacker to steal?
A day was specified. Does this company rotate passwords on a daily basis? The fact that monday was specified implies that there are no more than seven passwords: one for each day of the week.

Rather than putting all of this information into the URL, it is generally better to hide it inside the profile. It's probably safe to assume that mydb refers to a database connection (if we called the profile mysql, we would be exposing what kind of database connection). Skipping over any database credentials that would be present, we could use a profile that looks like:

mydb:
  driver: <some SDB module>
  fields:
    user: sdbkey
    group: ops
    day: monday

Assuming that the module in question is able to translate those fields into a query, and internally change sdbkey to whatever actual key was passed in, we could use a URI that looks like:

sdb://mydb/curly

You can still guess that curly refers to a username, which is probably even more obvious when the URI is used with a configuration argument like:

username: sdb://mydb/curly

However, it doesn't expose the name of the field in the database.

The final SDB module

With all of the code we've put together, the resulting module should look like the following:

'''
SDB module for JSON

Like all sdb modules, the JSON module requires a configuration profile to
be configured in either the minion or master configuration file. This profile
requires very little. In the example:

.. code-block:: yaml

    myjson:
      driver: json
      json_file: /root/mydata.json

The ``driver`` refers to the json module and json_file is the path to the JSON
file that contains the data.

.. code-block:: yaml

    password: sdb://myjson/somekey
'''
from __future__ import absolute_import
from salt.exceptions import CommandExecutionError
import salt.utils
import json

__func_alias__ = {
    'set_': 'set'
}


def get(key, profile=None):
    '''
    Get a value from a JSON file
    '''
    try:
        with salt.utils.fopen(profile['json_file'], 'r') as fp_:
            json_data = json.load(fp_)
        return json_data.get(key, None)
    except IOError as exc:
        raise CommandExecutionError (exc)
    except KeyError as exc:
        raise CommandExecutionError ('{0} needs to be configured'.format(exc))
    except ValueError as exc:
        raise CommandExecutionError (
            'There was an error with the JSON data: {0}'.format(exc)
        )


def set_(key, value, profile=None):  # pylint: disable=W0613
    '''
    Set a key/value pair in a JSON file
    '''
    try:
        with salt.utils.fopen(profile['json_file'], 'r') as fp_:
            json_data = json.load(fp_)
    except IOError as exc:
        raise CommandExecutionError (exc)
    except KeyError as exc:
        raise CommandExecutionError ('{0} needs to be configured'.format(exc))
    except ValueError as exc:
        raise CommandExecutionError (
            'There was an error with the JSON data: {0}'.format(exc)
        )

    json_data[key] = value

    try:
        with salt.utils.fopen(profile['json_file'], 'w') as fp_:
            json.dump(json_data, fp_)
    except IOError as exc:
        raise CommandExecutionError (exc)

    return get(key, profile)