===================================
ksgen - Khaleesi Settings Generator
===================================
Setup
=====
It's advised to use ksgen in a virtual Python environment. ::
$ virtualenv ansible # you skip this and use an existing one
$ source ansible/bin/activate
$ python setup.py install # do this in the ksgen directory
Running ksgen
=============
**Assumes** that ksgen is installed, else follow Setup_.
You can get general usage information with the ``--help`` option. After you
built a proper settings directory ("configuration tree") structure, you need to
let ksgen know where it is. Invoke ksgen like this to show you all your
possible options::
ksgen --config-dir
help
If ``--config-dir`` is not provided, ksgen will look for the ``KHALEESI_DIR``
environment variable, so it is a good practice to define this in your
``.bashrc`` file::
export KHALEESI_DIR=
ksgen help
This displays options you can pass to ksgen to generate_ the all-in-one
settings file.
Using ksgen
===========
How ksgen works
---------------
ksgen is a simple utility to **merge** dictionaries (hashes, mappings), and
lists (sequences, arrays). Any scalar value (string, int, floats) are
overwritten while merging.
For e.g.: merging ``first_file.yml`` and ``second_file.yml``
first_file.yml::
foo:
bar: baz
merge_scalar: a string from first dict
merge_list: [1, 3, 5]
nested:
bar: baz
merge_scalar: a string from first dict
merge_list: [1, 3, 5]
and second_file::
foo:
too: moo
merge_scalar: a string from second dict
merge_list: [6, 2, 4, 3]
nested:
bar: baz
merge_scalar: a string from second dict
merge_list: [6, 2, 4, 3]
produces the output below::
foo:
bar: baz
too: moo
merge_scalar: a string from second dict
merge_list: [1, 3, 5, 6, 2, 4, 3]
nested:
bar: baz
merge_scalar: a string from second dict
merge_list: [1, 3, 5, 6, 2, 4, 3]
Organizing settings files
-------------------------
ksgen requires a ``--config-dir`` option which points to the directory where
the settings files are stored. ksgen traverses the *config-dir* to generate a
list of options that sub-commands (``help``, ``generate``) can accept.
First level directories inside the config-dir are used as options, and yml
files inside them are used as option values. You can add suboptions if you add
a directory with the same name as the value (without the extension). Inside
that directory, the pattern repeats: you can specify options by creating
directories, and inside them yml files.
If the directory name and a yml file don't match, it doesn't add a suboption
(it gets ignored). You can use this to store YAMLs for includes.
Look at the following schematic example::
settings/
├── option1/
│ ├── ignored/
│ │ └── useful.yml
│ ├── value1.yml
│ └── value2.yml
└── option2/
├── value3/
│ └── suboption1/
│ ├── value5.yml
│ └── value6.yml
├── value3.yml
└── value4.yml
The valid settings will be::
$ ksgen --config-dir settings/ help
[snip]
Valid configs are:
--option1= [value2, value1]
--option2= [value4, value3]
--option2-suboption1= [value6, value5]
A more organic _`settings example`::
settings/
├── installer/
│ ├── foreman/
│ │ └── network/
│ │ ├── neutron.yml
│ │ └── nova.yml
│ ├── foreman.yml
│ ├── packstack/
│ │ └── network/
│ │ ├── neutron.yml
│ │ └── nova.yml
│ └── packstack.yml
└── provisioner/
├── trystack/
│ ├── tenant/
│ │ ├── common/
│ │ │ └── images.yml
│ │ ├── john-doe.yml
│ │ ├── john.yml
│ │ └── smith.yml
│ └── user/
│ ├── john.yml
│ └── smith.yml
└── trystack.yml
ksgen maps all directories to options and files in those directories to
values that the option can accept. Given the above directory structure,
the options that ``generate`` can accept are as follows
+---------------------+-----------------------+
| Options | Values |
+=====================+=======================+
| provisioner | trystack |
+---------------------+-----------------------+
| provisioner-tenant | smith, john, john-doe |
+---------------------+-----------------------+
| provisioner-user | john, smith |
+---------------------+-----------------------+
| installer | packstack, foreman |
+---------------------+-----------------------+
| installer-network | nova, neutron |
+---------------------+-----------------------+
.. NOTE:: ksgen skips provisioner/trystack/tenant/common directory since
there is no ``common.yml`` file under the ``tenant`` directory.
Default settings
----------------
Default settings allow the user to supply only the minimal required flags
in order to generate a valid output file. Defaults settings will be loaded
from the given 'top-level' parameters settings files if they are defined in
them. Defaults settings for any 'non top level' parameters that have been
given will not been loaded.
Example of defaults section in settings files:
.. code-block:: yaml
:caption: provisioner/openstack.yml
defaults:
site: openstack-site
topology: all-in-one
.. code-block:: yaml
:caption: provisioner/openstack/site/openstack-site.yml
defaults:
user: openstack-user
Usage example: ::
ksgen --config-dir=/settings/dir/path generate --provisioner=openstack settings.yml
_`generate`: merges settings into a single file
-----------------------------------------------
The ``generate`` command merges multiple settings file into a single
file. This file can then be passed to an ansible playbook. ksgen also
allows merging, extending, overwriting (!overwrite_) and looking up
(!lookup_) settings that ansible (at present) doesn't allow.
Merge order
~~~~~~~~~~~
Refering back to the `settings example`_ above, if you execute the command::
ksgen --config-dir sample generate \
--provisioner trystack \
--installer packstack \
--provisioner-user john \
--extra-vars foo.bar=baz \
--provisioner-tenant smith \
output-file.yml
`generate`_ command will create an ``output-file.yml`` that include all
contents of
+----+---------------------------------------------+--------------------------------------------------+
| SL | File | Reason |
+====+=============================================+==================================================+
| 1 | provisioner/trystack.yml | The first command line option |
+----+---------------------------------------------+--------------------------------------------------+
| 2 | merge provisioner/trystack/user/john.yml | The first child of the first command line option |
+----+---------------------------------------------+--------------------------------------------------+
| 3 | merge provisioner/trystack/tenant/smith.yml | The next child of the first command line option |
+----+---------------------------------------------+--------------------------------------------------+
| 4 | merge installer/packstack.yml | the next top-level option |
+----+---------------------------------------------+--------------------------------------------------+
| 5 | add/merge foo.bar: baz. to output | extra-vars get processed at the end |
+----+---------------------------------------------+--------------------------------------------------+
Rules file
~~~~~~~~~~
ksgen arguments can get quite long and tedious to maintain, the options passed
to ksgen can be stored in a rules yaml file to simplify invocation. The command
above can be simplified by storing the options in a yaml file.
rules_file.yml::
args:
provisioner: trystack
provisioner-user: john
provisioner-tenant: smith
installer: packstack
extra-vars:
- foo.bar=baz
ksgen generate using rules_file.yml::
ksgen --config-dir sample generate \
--rules-file rules_file.yml \
output-file.yml
Apart from the **args** key in the rules-files to supply default args to
generate, validations can also be added by adding a 'validation.must_have' like
below::
args:
...
default args
...
validation:
must_have:
- topology
The generate commmand would validate that all options in must_have are supplied
else it will fail with an appropriate message.
YAML tags
=========
ksgen uses Configure_ python package to keep the yaml files DRY_. It also adds a
few yaml tags like !overwrite, !lookup, !join, !env to the collection.
.. _Configure: http://configure.readthedocs.org/en/latest/
.. _DRY: https://en.wikipedia.org/wiki/Don't_repeat_yourself
overwrite
---------
Use overwrite_ tag to overwrite value of a key. This is especially useful when
to clear the contents of an array and add new one
For e.g.: merging ::
foo: bar
and ::
foo: [1, 2, 3]
will fail since there is no reasonable way to merge a string and an array.
Use overwrite to set the contents of foo to [1, 2, 3] as below ::
foo: !overwrite [1, 2, 3]
lookup
------
Lookup helps keep the yaml files DRY_ by replacing looking up values for keys.
::
foo: bar
key_foo: !lookup foo
After ksgen process the yaml above the value of `key_foo` will be replaced by
`bar` resulting in the output below. ::
foo: bar
key_foo: bar
This works for several consecutive !lookup as well such as ::
foo:
barfoo: foobar
bar:
foo: barfoo
key_foo: !lookup foo[ !lookup bar.foo ]
After ksgen process the yaml above the value of `key_foo` will be replaced by
`foobar`
.. Warning:: (Limitation) Lookup is done only after all yaml files are loaded
and the values are merged so that the entire yaml tree can be searched. This
prevents combining other yaml tags with lookup_ as most tags are processed
when yaml is loaded and not when it is written. For example::
home: /home/john
bashrc: !join [ !lookup home, /bashrc ]
This **will fail** to set bashrc to `/home/john/bashrc` where as the snippet
below will work as expected::
bashrc: !join [ !env HOME, /bashrc ]
join
----
Use join tag to join all items in an array into a string. This is quite useful
when using yaml anchors or env_ tag. ::
unused:
baseurl: &baseurl http://foobar.com/repo/
repo:
epel7: !join[ *baseurl, epel7 ]
bashrc: !join [ !env HOME, /bashrc ]
env
---
Use env tag to lookup value of an environment variable. An optional default
value can be passed to the tag. if no default values are passed and the lookup
fails, then a runtime KeyError is generated. Second optional argument will
reduce length of value by given value ::
user_home: !env HOME
user_shell !env [SHELL, zsh] # default shell is zsh
job_name_parts:
- !env [JOB_NAME, 'dev-job']
- !env [BUILD_NUMBER, None ]
- !env [USER, None, 5]
job_name: "{{ job_name_parts | reject(none) | join('-') }}"
The snippet above effectively uses env_ tag and default option to set the
`job_name` variable to `$JOB_NAME-$BUILD_NUMBER-${USER:0:5}` if they are
defined else to 'dev-job'.
limit_chars
-----------
This function will trim value of variable or string to given length.
debug:
message: !limit_chars [ 'some really looong text' 10 ]
Debugging errors in settings
============================
ksgen is heavily logged and by default the log-level is set to **warning**.
Changing the debug level using the ``--log-level`` option to **info** or
**debug** reveals more information about the inner workings of the tool and how
values are loaded from files and merged.
Developing ksgen
=================
Running ksgen unit-tests
------------------------
::
pip install pytest
py.test tests/test_.py
# or
python tests/test_.py