Software Infrastructure

OPNFV Software Infrastructure consists of set of components and tools that realize OPNFV Continuous Integration (CI) and provide means for community to contribute to OPNFV in most efficient way. OPNFV Software Infrastructure enables and orchestrates development, integration and testing activities for the components OPNFV consumes from upstream communities and for the development work done in scope of OPNFV. Apart from orchestration aspects, providing timely feedback that is fit for purpose to the OPNFV community is one of its missions.

CI is the top priority for OPNFV Software Infrastructure. Due to the importance the OPNFV community puts into it, the resulting CI machinery is highly powerful, capable and runs against distributed hardware infrastructure managed by OPNFV Pharos Project. The hardware infrastructure OPNFV CI relies on is located in 3 different continents, 5+ different countries and 10+ different member companies.

OPNFV CI is continuously evolved in order to fulfill the needs and match the expectations of the OPNFV community.

OPNFV Software Infrastructure is developed, maintained and operated by OPNFV Releng Project with the support from Linux Foundation.

Continuous Integration Server

Jenkins

Connecting OPNFV Community Labs to OPNFV Jenkins

Abstract

This document describes how to connect resources (servers) located in Linux Foundation (LF) lab and labs provided by the OPNFV Community to OPNFV Jenkins.

License

Connecting OPNFV Community Labs to OPNFV Jenkins (c) by Fatih Degirmenci (Ericsson AB) and others.

Connecting OPNFV Labs to OPNFV Jenkins document is licensed under a Creative Commons Attribution 4.0 International License.

You should have received a copy of the license along with this. If not, see <http://creativecommons.org/licenses/by/4.0/>.

Version History

Date Version Author Comment
2015-05-05 0.1.0 Fatih Degirmenci First draft
2015-09-25 1.0.0 Fatih Degirmenci Instructions for the Arno SR1 release
2016-01-25 1.1.0 Jun Li Change the format for new doc toolchain
2016-01-27 1.2.0 Fatih Degirmenci Instructions for the Brahmaputra release
2016-05-25 1.3.0 Julien Add an additional step after step9 to output the correct monit config file

Jenkins

Jenkins is an extensible open source Continuous Integration (CI) server. [1]

Linux Foundation (LF) hosts and operates OPNFV Jenkins.

Jenkins Slaves

Slaves are computers that are set up to build projects for a Jenkins Master. [2]

Jenkins runs a separate program called “slave agent” on slaves. When slaves are registered to a master, the master starts distributing load to slaves by scheduling jobs to run on slaves if the jobs are set to run on them. [2]

Term Node is used to refer to all machines that are part of Jenkins grid, slaves and master. [2]

Two types of slaves are currently connected to OPNFV Jenkins and handling different tasks depending on the purpose of connecting the slave.

The slaves connected to OPNFV Jenkins can be seen using this link: https://build.opnfv.org/ci/computer/

Slaves without red cross next to computer icon are fully functional.

Connecting Slaves to OPNFV Jenkins

The method that is normally used for connecting slaves to Jenkins requires direct SSH access to servers. [3] This is the method that is used for connecting slaves hosted in LF Lab.

Connecting slaves using direct SSH access can become a challenge given that OPNFV Project has number of different labs provided by community as mentioned in previous section. All these labs have different security requirements which can increase the effort and the time needed for connecting slaves to Jenkins. In order to reduce the effort and the time needed for connecting slaves and streamline the process, it has been decided to connect slaves using Java Network Launch Protocol (JNLP).

Connecting Slaves from LF Lab to OPNFV Jenkins

Slaves hosted in LF Lab are handled by LF. All the requests and questions regarding these slaves should be submitted to OPNFV LF Helpdesk.

Connecting Slaves from Community Labs to OPNFV Jenkins

As noted in corresponding section, slaves from Community Labs are connected using JNLP. Via JNLP, slaves open connection towards Jenkins Master instead of Jenkins Master accessing to them directly.

Servers connecting to OPNFV Jenkins using this method must have access to internet.

Please follow below steps to connect a slave to OPNFV Jenkins.

  1. Create a user named jenkins on the machine you want to connect to OPNFV Jenkins and give the user sudo rights.
  2. Install needed software on the machine you want to connect to OPNFV Jenkins as slave.
  • openjdk 7
  • monit
  1. If the slave will be used for running virtual deployments, Functest, and Yardstick, install below software and make jenkins user the member of the groups.
  • docker
  • libvirt
  1. Create slave root in Jenkins user home directory.
mkdir -p /home/jenkins/opnfv/slave_root
  1. Clone OPNFV Releng Git repository.

mkdir -p /home/jenkins/opnfv/repos

cd /home/jenkins/opnfv/repos

git clone https://gerrit.opnfv.org/gerrit/p/releng.git

  1. Contact LF by sending mail to OPNFV LF Helpdesk and request creation of a slave on OPNFV Jenkins. Include below information in your mail.
  • Slave root (/home/jenkins/opnfv/slave_root)
  • Public IP of the slave (You can get the IP by executing curl http://icanhazip.com/)
  • PGP Key (attached to the mail or exported to a key server)
  1. Once you get confirmation from LF stating that your slave is created on OPNFV Jenkins, check if the firewall on LF is open for the server you are trying to connect to Jenkins.

cp /home/jenkins/opnfv/repos/releng/utils/jenkins-jnlp-connect.sh /home/jenkins/ cd /home/jenkins/ sudo ./jenkins-jnlp-connect.sh -j /home/jenkins -u jenkins -n  <slave name on OPNFV Jenkins> -s <the token you received from LF> -f

  • If you receive an error, follow the steps listed on the command output.
  1. Run the same script with test(-t) on foreground in order to make sure no problem on connection. You should see INFO: Connected in the console log.

sudo ./jenkins-jnlp-connect.sh -j /home/jenkins -u jenkins -n <slave name on OPNFV Jenkins> -s <the token you received from LF> -t

  • If you receive an error similar to the one shown on this link, you need to check your firewall and allow outgoing connections for the port.
  1. Kill the Java slave.jar process.
  2. Run the same script normally without test(-t) in order to get monit script created.
sudo ./jenkins-jnlp-connect.sh -j /home/jenkins -u jenkins -n <slave name on OPNFV Jenkins> -s <the token you received from LF>
  1. Edit monit configuration and enable http interface. The file to edit is /etc/monit/monitrc on Ubuntu systems. Uncomment below lines.
set httpd port 2812 and
use address localhost # only accept connection from localhost allow localhost # allow localhost to connect to the server and
  1. Restart monit service.

  • Without systemd:

    sudo service monit restart

  • With systemd: you have to enable monit service first and then restart it.

    sudo systemctl enable monit

    sudo systemctl restart monit

  1. Check to see if jenkins comes up as managed service in monit.
sudo monit status
  1. Connect slave to OPNFV Jenkins using monit.
sudo monit start jenkins
  1. Check slave on OPNFV Jenkins to verify the slave is reported as connected.
  • The slave on OPNFV Jenkins should have some executors in “Idle” state if the connection is successful.

Notes

PGP Key Instructions

Public PGP Key can be uploaded to public key server so it can be taken from there using your mail address. Example command to upload the key to key server is

gpg --keyserver hkp://keys.gnupg.net:80  --send-keys XXXXXXX

The Public PGP Key can also be attached to the email by storing the key in a file and then attaching it to the email.

gpg --export -a '<your email address>' > pgp.pubkey

Jenkins User-Guide

TBD

Creating/Configuring/Verifying Jenkins Jobs

Clone and setup the repo:

git clone ssh://YOU@gerrit.opnfv.org:29418/releng
cd releng
git review -s

Make changes:

git commit -sv
git review
remote: Resolving deltas: 100% (3/3)
remote: Processing changes: new: 1, refs: 1, done
remote:
remote: New Changes:
remote:   https://gerrit.opnfv.org/gerrit/51
remote:
To ssh://agardner@gerrit.opnfv.org:29418/releng.git
 * [new branch]      HEAD -> refs/publish/master

Test with tox:

tox -v -ejjb

Submit the change to gerrit:

git review -v

Follow the link to gerrit https://gerrit.opnfv.org/gerrit/51 in a few moments the verify job will have completed and you will see Verified +1 jenkins-ci in the gerrit ui.

If the changes pass the verify job https://build.opnfv.org/ci/view/builder/job/builder-verify-jjb/ , the patch can be submitited by a committer.

Job Types

  • Verify Job
    • Trigger: recheck or reverify
  • Merge Job
    • Trigger: remerge
  • Experimental Job
    • Trigger: check-experimental

The verify and merge jobs are retriggerable in Gerrit by simply leaving a comment with one of the keywords listed above. This is useful in case you need to re-run one of those jobs in case if build issues or something changed with the environment.

The experimental jobs are not triggered automatically. You need to leave a comment with the keyword list above to trigger it manually. It is useful for trying out experimental features.

Note that, experimental jobs skip vote for verified status, which means it will reset the verified status to 0. If you want to keep the verified status, use recheck-experimental in commit message to trigger both verify and experimental jobs.

You can add below persons as reviewers to your patch in order to get it reviewed and submitted.

Or Add the group releng-contributors

Or just email a request for submission to opnfv-helpdesk@rt.linuxfoundation.org

The Current merge and verify jobs for jenkins job builder can be found in releng-jobs.yaml.

Jenkins Node Labels

TBD

Source Control and Code Review

Gerrit

Gerrit User Guide

Artifact and Image Repositories

Google Storage & Docker Hub

Artifact Repository

TBD

Docker Hub

TBD

Issue and Bug Tracking

JIRA