Creating a Jenkins server with Khaleesi jobs

Getting a Jenkins

Deploying the jobs require a properly configured Jenkins server. We have a couple of them already, but if you want to experiment without any fear of messing with other jobs, the best is to get yourself a server. It’s recommended to use the Long Term Support (LTS) version.

You can create a VM on any of our OpenStack instances (don’t forget to use your public key for it), attach a floating IP and then install Jenkins. This should work both on Fedora and RHEL:

sudo wget -O /etc/yum.repos.d/jenkins.repo \
http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
sudo rpm --import \
http://pkg.jenkins-ci.org/redhat-stable/jenkins-ci.org.key
yum install jenkins
service jenkins start
chkconfig jenkins on

Installing plugins

Our jobs require quite a few plugins. So when your Jenkins is up and running, navigate to http://$JENKINS_IP:8080/cli and download jenkins-cli.jar.

Afterwards. just execute these commands:

java -jar jenkins-cli.jar -s http://$JENKINS_IP:8080/ install-plugin git \
xunit ansicolor multiple-scms rebuild ws-cleanup gerrit-trigger \
parameterized-trigger envinject email-ext sonar copyartifact timestamper \
build-timeout jobConfigHistory test-stability jenkins-multijob-plugin \
dynamicparameter swarm shiningpanda scm-api ownership mask-passwords \
jobConfigHistory buildresult-trigger test-stability dynamicparameter \
scm-api token-macro swarm scripttrigger groovy-postbuild shiningpanda \
jenkins-multijob-plugin ownership

Deploying the jobs

You can do this from any machine. Install JJB:

pip install jenkins-job-builder

Create a config file for it:

cat > my_jenkins << EOF
[jenkins]
user=my_username
password=my_password
url=http://$JENKINS_IP:8080/
EOF

Optional: I recommend turning off the timed runs (deleting - timed lines from the job template), otherwise they would run periodically on your test server:

sed '/- timed:/d' khaleesi-settings/jobs/main.yaml

Then just run the job creation (the last argument is the job directory of the khaleesi-settings repo, which I assume you cloned previously):

jenkins-jobs --conf my_jenkins update khaleesi-settings/jobs/

Bonus: Test your job changes

If you want to experiment with your own job changes:

  • open khaleesi-settings/jobs/defaults.yaml
  • change the khaleesi and/or khalessi-settings repo URL to your own
    and your own branch
  • execute the job building step above

Now your test server will use your own version of the repos.

Tip

you can git stash save testing these changes, and later recall them with git stash pop to make this testing step easy along the code review submission.

Creating a Jenkins slave

Now you need to either set up the machine itself as a slave, or attach/create a slave to run the jobs. The slave needs to have the ‘khaleesi’ label on it to run the JJB jobs.

You can set up a slave with the help of the khaleesi-slave repo.

git clone git@github.com:redhat-openstack/khaleesi-settings.git
cd khaleesi-settings/jenkins/slaves
cat << EOF > hosts
$SLAVE_IP

[slave]
$SLAVE_IP
EOF

Check the settings in ansible.cfg.sample. If you run into weird ansible errors about modules you probably don’t have them set up correctly. This should be enough:

[defaults]
host_key_checking = False
roles_path = ./roles

Execute the playbook, assuming that your instance uses the “fedora” user and you can access it by the “rhos-jenkins.pem” private key file. If you used a proper cloud image, it will fail.

ansible-playbook -i hosts -u fedora playbooks/basic_internal_slave.yml --private-key=rhos-jenkins.pem -v

Login to the machine, become root and delete the characters from /root/.ssh/authorized_keys before the “ssh-rsa” word. Log out and rerun the ansible command. It should now finish successfully.

Add the slave to Jenkins. If you used the same machine, specify localhost and add the relevant public key for the rhos-ci user. use the /home/rhos-ci/jenkins directory, add the khaleesi label, only run tied jobs. You’re done.

Jenkins RDO-Manager:

For using khaleesi with Jenkins, first of all see the steps Getting a Jenkins part for setting up a Jenkins slave and for use jjb.

If you want to setup a manual job on Jenkins you have to follow those steps:

Setup a slave (General):

Check the option:

Restrict where this project can be run

And put the name of your slave.

Clone the repositories (Source Code Management):

Select the choice:

Multiple SCMs

And put the urls of the khaleesi / khaleesi-settings repositories. You need to specify to jenkins to checkout the repositories in a sub-directory:

Check out to a sub-directory

And specify for each:

khaleesi
khaleesi-settings

Build Environment:

Check the option:

Delete workspace before build starts

Build:

Add a step:

Virtualenv Builder

And select:

Python version: System-CPython-2.7
Nature: Shell

And put the above informations into the shell command:

pip install -U ansible==1.9.2 > ansible_build; ansible --version
source khaleesi-settings/jenkins/ansible_rdo_mang_settings.sh

# install ksgen
pushd khaleesi/tools/ksgen
python setup.py install
popd

pushd khaleesi
# generate config
ksgen --config-dir=../khaleesi-settings/settings generate \
    --provisioner=your_provisioner (see cookbook)

# get nodes and run test
set +e
anscmd="stdbuf -oL -eL ansible-playbook -vv --extra-vars @ksgen_settings.yml"

$anscmd -i local_hosts playbooks/full-job-no-test.yml
result=$?

infra_result=0
$anscmd -i hosts playbooks/collect_logs.yml &> collect_logs.txt || infra_result=1
$anscmd -i local_hosts playbooks/cleanup.yml &> cleanup.txt || infra_result=2

if [[ "$infra_result" != "0" && "$result" = "0" ]]; then
    # if the job/test was ok, but collect_logs/cleanup failed,
    # print out why the job is going to be marked as failed
    result=$infra_result
    cat collect_logs.txt
    cat cleanup.txt
fi

exit $result

Post-build actions:

Add a post build action for collecting logs and required files for debuging and archived them:

Archive the artifacts: **/collected_files/*.tar.gz, **/nosetests.xml, **/ksgen_settings.yml

If you run tempest during the deployment add the following step for collecting the tests result:

Publish JUnit test result report
Test Report XMLs : **/nosetests.xml
Check : Test stability history