Cross Community Continuous Integration

Contents:

1. Cross Community Continuous Integration

1.1. Introduction

OPNFV develops, operates, and maintains its Continuous Integration (CI) process used by OPNFV community to develop, integrate, test and release the integrated reference platform for NFV.

During past releases, OPNFV released different flavors (scenarios) of the platform in an entirely automated fashion, resulting in feedback to OPNFV itself and other communities OPNFV works with. This enabled the community to implement new features, identify/fix bugs directly in corresponding upstream communities.

The development and release model employed by OPNFV during the past releases used stable versions of upstream components. This helps developers and users who are focused on stability however requires a lot of time for development, integration, and testing, thus resulting in a slower pace of innovation.

In order to increase the speed of development and evolution of the platform, OPNFV needs to provide means for its developers and users. One of the ways to achieve this is to ensure developers have access to the latest versions of the upstream components they are developing, integrating, and testing.

Based on this need, OPNFV Infrastructure Working Group (Infra WG) started the Cross Community Continuous Integration (XCI) initiative, bringing in a new development and release model into OPNFV which is based on Continuous Delivery and DevOps practices and principles.

1.2. Focus Areas

1.2.1. Enabling Continuous Delivery

By definition, XCI focuses on master branches in order to

• shorten the time it takes to introduce new features
• make it easier to identify and fix bugs
• ease the effort to develop, integrate, and test the reference platform
• establish additional feedback loops within OPNFV, towards the users and between the communities OPNFV works with
• increase the visibility regarding the state of things at all times

XCI aims to enable this by applying basic CI/CD & DevOps principles and following best practices such as

• fail fast, fix fast
• always have working software
• small and frequent commits
• work against the trunk, shortening development time
• fast and tailored feedback
• everything is visible to everyone all the time
• and others

By doing this, the overall quality of the platform components provided by the upstream communities will increase greatly, making it easier for anyone to consume them when they need them with less trouble, helping to find and fix bugs much earlier than what it is today and develop new features.

How good this can work depends on the nature of the changes. If the changes are atomic and more importantly complete, the value added by XCI will increase significantly.

1.2.2. Putting Users and Developers First

Apart from applying the principles and following the best practices, XCI puts the users and developers first by pushing for

• less burden on developers as they do not need to be aware of all details that are not of their interest/concern
• reduced complexity
• an easy way to try and develop things
• speed, helping developers to bring their scenarios to OPNFV faster
• a real scenario ownership

The proof of the user and developer centric approach is that the first thing XCI made available is the sandbox for users and developers to try things out.

1.2.3. Keeping Quality, Confidence and Predictability High

Another and perhaps the most crucial concern for XCI is to keep the quality high, increase the confidence, have predictability, and have the availability of the latest versions earlier. Some of the prerequisites to fulfill these goals are

• Test early and often
• Know the quality at all times
• Make the platform available early so people have time to develop, integrate, and test their work
• Avoid big bang uplifts
• Avoid surprises

1.2.4. Source Based Deployments

CI starts on developer workstation and this is the fastest feedback a developer can get. In order to ensure developers can apply this principle, they need the tools and ways to enable fast development and test cycles that are repeatable as many times as necessary without hassle.

One way to achieve this is to bring developers closer to the source and remove anything between them. This means that what XCI brings is not only deploying from upstream master branches but doing that from source with no intermediaries in between.

A simple scenario that demostrates the value of bringing capability of source based deployments to OPNFV can be seen on the diagram below.

As you can see on the diagram, XCI provides tools and ways for developers to

• patch the source code on their laptop
• get the patch deployed to the stack with a single command
• run the tests
• fix if something is broken
• repeat the cycle until they are satisfied with it and have confidence in it
• send the patch for review and CI

This does not mean XCI will completely skip using artifacts. Artifact based deployments will be available in later CI loops such as daily/weekly, but the developer near loops will be run using source code.

1.2.5. Multi-distro Support

Giving choice and not imposing things on developers and users are two of the important aspects of XCI. This means that if they want to have all in one deployments, they should be able to do that by using different flavors provided by XCI.

Multi-distro support falls into same category for XCI; giving choice and making sure people can pick and choose what Linux distribution they want to use.

XCI currently supports Ubuntu 16.04, CentOS 7, and openSUSE Leap 42.3 and the choice is entirely left to the user.

Feature parity between the OPNFV scenarios on different Linux distributions that are supported by XCI may vary and it is possible for OPNFV community to work on bringing them to the same level.

1.3. XCI Pipelines

Providing timely and tailored feedback is one of the most important things about CI. It is important to make the feedback easily accessible and consumable for the community so the issues can be analysed as quickly as possible and fixes can be issued appropriately.

XCI focuses on feedback aspects of the CI and ensures that whatever feedback provided to community makes sense rather than just pointing to some logs. In order to achieve this, XCI enhances existing feedback loops and establishes new ones based on who needs the feedback. XCI does this by its pipelines as listed below.

1.3.1. Pipelines for Upstream Projects

OPNFV work upstream first which means that majority of the work is done in upstream projects. The upstream projects OPNFV works with have CI pipelines for the code contributed but the pipelines generally lack the testing that is important for the OPNFV community.

XCI aims to provide patch level feedback and feedback based on the tip of the master branches for the upstream projects. This means that if an OPNFV developer contributes to an upstream project, it will be possible for developer to get additional feedback from OPNFV XCI in order to ensure the contribution works in OPNFV context as well. The level of testing will be adjusted based on the community needs and it is important not to duplicate the testing done by the upstream communities in their CI pipelines.

Apart from providing feedback to the developers, these pipelines will be used for finding working versions of upstream components from their master branches to pin them for development purposes.

Please note that the types and scope of testing done by XCI vary for different projects and the work for enabling pipelines is still in progress which heavily depends on the readiness of the upstream projects.

1.3.2. Pipelines for OPNFV Scenarios

OPNFV CI has pipelines for scenarios constructed by the OPNFV projects. However the existing pipelines have number of areas that require improvements.

The existing pipelines lack the granularity one might expect. This means that the changes to the scenarios are either not tested properly or tested together with unrelated scenarios, resulting in lack of testing or taking too much time to get feedback.

Apart from the test coverage and the time it takes to test, scenarios are generally tested on a daily basis on baremetal even if it may not be necessary and there are no changes.

XCI will change how the feedback is provided for the scenarios by pushing scenario ownership to corresponding projects and establishing loops for patchset verification, daily and weekly tests. This means that if a scenario changes in a project repo, verification for that scenario will directly be triggered and testing will be done using virtual deployments, providing feedback to the project.

Daily and weekly loops will be run on baremetal if and only if the scenario is worth testing on baremetal. This will be achieved by applying promotion concepts; if a scenario passes virtual deployments, it will be tested by daily loops on baremetal and by weekly loops later on.

1.3.3. Pipelines for OPNFV Test Projects

OPNFV Test Projects generally lack the CI coverage they need. Most of the test projects only have unit tests, resulting in faults slipping into platform testing, making it harder for community to understand what really went wrong; for example, is there a problem with the scenario itself or is the problem with the test framework/cases?

XCI aims to establish proper CI pipelines for the test projects by employing virtual deployments, so any change that is done to test frameworks/cases themselves will be tested against a real virtual deployment. The deployments will be brought up using a verified version of the relevant scenario and via snapshots so the patch verification will be relevant and quick. If the testing at this level fails, it is most probably due to the patch itself rather than the scenario, preventing faulty code from slipping into master branch of the test project. Further feedback loops, such as post-merge, can be established depending on the needs of the community.

1.3.4. Pipelines for XCI Framework and Sandbox

XCI itself needs to be tested properly in order to ensure the changes to the framework or the sandbox do not break anything for the community.

1.3.5. Putting All Together

All the pipelines explained in earlier sections run in parallel and as independently from each other as possible, providing feedback to relevant communities and people so they can get the feedback that fit their purposes.

Output of these pipelines (verdicts, artifacts, and so on) are also used by each of the pipelines appropriately, ensuring that the pipeline uses well tested versions of artifacts they need.

An example of this could be the pipelines for OPNFV test projects and the pipelines for the OPNFV scenarios. Pipelines for OPNFV projects need verified versions of the scenarios to gate changes coming to test project repositories so they can be tested in isolation. This means that whatever goes wrong during the gate is probably due to change itself and not because of the scenario since the scenario is tested long before and promoted to test pipeline to be used for gating.

Similar thing is valid for the OPNFV scenarios as well; the pipelines need verified versions of test frameworks/cases so when scenario is put on a baremetal and tested, only thing that really changed and possibily of causing a failure is the scenario itself.

2. Sandbox and User Guide

2.1. The Sandbox

Users and developers need to have an easy way to bring up an environment that fits their purpose in a simple way so they can spend time on features they are developing, bugs they are fixing, trying things out, for learning purposes or just for fun rather than dealing with tools and mechanisms used for creating and provisioning nodes, installing different components they do not intend to touch, etc.

However, we also have to deal with reality. For example, not all users and developers have full Pharos baremetal PODs or powerful machines available for their work or they may want to use different Linux distributions for different reasons. It is important to take these into account and provide different configuration options for the sandbox based on the requirements that people have on the environment they will be using.

Based on the observations we made and the feedback we received from the OPNFV users and developers, XCI Team has created a sandbox that is highly configurable, simple and at the same time capable of providing a realistic environment for people to do their work. The sandbox makes it possible to bring up the complete environment with a single command and offers a variety of options to change how the stack should be deployed. The configuration of the sandbox is as easy as setting a few environment variables.

The sandbox provides

• automated way to bring up and tear down a complete stack
• various flavors to pick and use
• support for different Linux distributions
• multiple OPNFV scenarios to install
• ability to select different versions of upstream components to base the work on
• ability to enable additional OpenStack services or disable others

One last point to highlight here is that the XCI itself uses the sandbox for development and test purposes so it is continuously tested to ensure it works for XCI and for users and developers who are using it for different purposes.

2.2. Components of the Sandbox

The sandbox uses OpenStack projects for VM node creation, provisioning and OpenStack installation. XCI Team provides playbooks, roles, and scripts to ensure the components utilized by the sandbox work in a way that serves the users in the best possible way.

• openstack/bifrost: Bifrost (pronounced bye-frost) is a set of Ansible playbooks that automates the task of deploying a base image onto a set of known hardware using Ironic. It provides modular utility for one-off operating system deployment with as few operational requirements as reasonably possible. Bifrost supports different operating systems such as Ubuntu, CentOS, and openSUSE. More information about this project can be seen on Bifrost documentation.
• openstack/openstack-ansible: OpenStack-Ansible is an official OpenStack project which aims to deploy production environments from source in a way that makes it scalable while also being simple to operate, upgrade, and grow. More information about this project can be seen on OpenStack Ansible documentation.
• opnfv/releng-xci: OPNFV Releng Project provides additional scripts, Ansible playbooks and configuration options in order for developers to have an easy way of using openstack/bifrost and openstack/openstack-ansible by just setting a couple of environment variables and executing a single script. More infromation about this project can be seen on OPNFV Releng documentation.

2.3. Sandbox Flavors

XCI Developer Sandbox provides 4 different configurations (flavors) that can be deployed using VM nodes.

Available flavors are listed on the table below.

Flavor	Number of VM Nodes	VM Specs Per Node	Time Estimates
All in One (aio)	1 VM Node controller & compute on single/same node 1 compute node	vCPUs: 8 RAM: 12GB Disk: 80GB NICs: 1	Provisioning: 10 mins Deployment: 90 mins Total: 100 mins
Mini	3 VM Nodes 1 deployment node 1 controller node 1 compute node	vCPUs: 6 RAM: 12GB Disk: 80GB NICs: 1	Provisioning: 12 mins Deployment: 65 mins Total: 77 mins
No HA	4 VM Nodes 1 deployment node 1 controller node 2 compute nodes	vCPUs: 6 RAM: 12GB Disk: 80GB NICs: 1	Provisioning: 12 mins Deployment: 70 mins Total: 82 mins
HA	6 VM Nodes 1 deployment node 3 controller nodes 2 compute nodes	vCPUs: 6 RAM: 12GB Disk: 80GB NICs: 1	Provisioning: 15 mins Deployment: 105 mins Total: 120 mins

The specs for VMs are configurable and the more vCPU/RAM the better.

Estimated times listed above are provided as a guidance and they might vary depending on

• the physical (or virtual) host where the sandbox is run
• the specs of the VM nodes
• the Linux distribution
• whether the boot images are recreated or not
• installed/activated OpenStack services
• internet connection bandwidth

2.3.1. Flavor Layouts

All flavors are created and deployed based on the upstream OpenStack Ansible (OSA) guidelines.

Network configuration on the nodes are same no matter which flavor is used. The VMs are attached to default libvirt network and has single NIC where VLANs are created on. Different Linux bridges for management, storage and tunnel networks are created on these VLANs.

Use of more production-like network setup with multiple interfaces is in our backlog. Enabling OVS as default is currently in progress.

For storage, Cinder with NFS backend is used. Work to enable CEPH is currently ongoing.

The differences between the flavors are documented below.

All in One

As shown on the table in the previous section, this flavor consists of a single node. All the OpenStack services, including compute run on the same node.

The flavor All in One (aio) is deployed based on the process described in the upstream documentation. Please check OpenStack Ansible Developer Quick Start for details.

Mini/No HA/HA

These flavors consist of multiple nodes.

• opnfv: This node is used for driving the installation towards target nodes in order to ensure the deployment process is isolated from the physical host and always done on a clean machine.
• controller: OpenStack control plane runs on this node.
• compute: OpenStack compute service runs on this node.

Please see the diagram below for the host and service layout for these flavors.

2.4. User Guide

2.4.1. Prerequisites

• A machine with sufficient CPU/RAM/Disk based on the chosen flavor
• Ubuntu 16.04, OpenSUSE Leap 42.3, or CentOS 7
• CPU/motherboard that supports hardware-assisted virtualization
• Passwordless sudo
• An SSH key generated for your user (ie ~/.ssh/id_rsa)
• Packages to install
  • git
  • python 2.7
  • pip
  • libvirt

2.4.2. How to Use

Basic Usage

1. If you don’t have one already, generate an SSH key in $HOME/.ssh

   ssh-keygen -t rsa

2. Clone OPNFV releng-xci repository

   git clone https://gerrit.opnfv.org/gerrit/releng-xci.git

3. Change into directory where the sandbox script is located

   cd releng-xci/xci

4. Execute the sandbox script

   ./xci-deploy.sh

Issuing above command will start the sandbox deployment using the default flavor aio and the verified versions of upstream components. (pinned-versions). The sandbox should be ready between 1,5 and 2 hours depending on the host machine.

After the script finishes execution, you can login to opnfv host and start using your new deployment.

The openrc file will be available on opnfv host in $HOME.

Advanced Usage

The flavor to deploy and the versions of upstream components to use can be configured by the users by setting certain environment variables. Below example deploys noha flavor using the latest of openstack-ansible master branch and stores logs in different location than what is set as default.

1. If you don’t have one already, generate an SSH key in $HOME/.ssh

   ssh-keygen -t rsa

2. Clone OPNFV releng-xci repository

   git clone https://gerrit.opnfv.org/gerrit/releng-xci.git

3. Change into directory where the sandbox script is located

   cd releng-xci/xci

4. Set the sandbox flavor

   export XCI_FLAVOR=noha

5. Set the version to use for openstack-ansible

   export OPENSTACK_OSA_VERSION=master

6. Set where the logs should be stored

   export LOG_PATH=/home/jenkins/xcilogs

7. Execute the sandbox script

   ./xci-deploy.sh

Please note that changing the version to use may result in unexpected behaviors, especially if it is changed to master. If you are not sure about how good the version you intend to use is, it is advisable to use the pinned versions instead.

Verifying the Basic Operation

You can verify the basic operation using the commands below.

1. Login to opnfv host

   ssh root@192.168.122.2

2. Source openrc file

   source openrc

3. Issue OpenStack commands

   openstack service list

You can also access the Horizon UI by using the URL, username, and the password displayed on your console upon the completion of the deployment.

Debugging Tips

If xci-deploy.sh fails midway through and you happen to fix whatever problem caused the failure in the first place, please run the script again. Do not attempt to continue the deployment using helper scripts such as bifrost-provision.sh.

Look at various logs in $LOG_PATH directory. (default one is /tmp/.xci-deploy-env/opnfv/logs)

2.4.3. Behind the Scenes

Here are steps that take place upon the execution of the sandbox script xci-deploy.sh:

1. Sources environment variables in order to set things up properly.
2. Installs ansible on the host where sandbox script is executed.
3. Creates and provisions VM nodes based on the flavor chosen by the user.
4. Configures the host where the sandbox script is executed.
5. Configures the deployment host which the OpenStack installation will be driven from.
6. Configures the target hosts where OpenStack will be installed.
7. Configures the target hosts as controller(s) and compute(s) nodes.
8. Starts the OpenStack installation.

2.4.4. User Variables

All user variables can be set from command line by exporting them before executing the script. The current user variables can be seen from user-vars file located in releng-xci repository.

The variables can also be set directly within the file before executing the sandbox script. If you do this, you need to set $OPNFV_RELENG_DEV_PATH environment variable where the releng-xci repo is located on your host which you modified the files in.

export OPNFV_RELENG_DEV_PATH=/path/to/releng-xci/

2.4.5. Pinned Versions

As explained earlier, the users can pick and choose which versions to use. If you want to be on the safe side, you can use the pinned versions the sandbox provides. They can be seen from pinned-versions.

OPNFV runs periodic jobs against upstream projects openstack/bifrost and openstack/openstack-ansible using the latest on master branch, continuously chasing upstream to find a well working version.

Once a working version is identified, the versions of the upstream components are then bumped in releng-xci repo.

2.4.6. Further Information

If you intend to use the sandbox in more advanced ways or if you are developing XCI itself or an OPNFV scenario, please refer to XCI Developer Guide.

2.5. Limitations, Known Issues, and Improvements

The complete list can be seen using this link.

2.6. Changelog

Changelog can be seen using this link.

2.7. Testing

Sandbox is continuously tested by OPNFV XCI to ensure the changes do not impact users. In fact, OPNFV XCI itself uses the sandbox to ensure it is always in working state.

2.8. Support

OPNFV XCI issues are tracked in OPNFV JIRA Releng project. If you encounter an issue or identify a bug, please submit an issue to JIRA using this link. Please label the issue you are submitting with xci label.

If you have questions or comments, you can ask them on the #opnfv-pharos IRC channel on Freenode.

2.9. References

• Bifrost Documentation
• OpenStack Ansible Documentation
• OPNFV Releng Documentation

3. XCI Developer Guide

This document will contain the developer guide.

3.1. Introduction

4. XCI Way of Working

This document will contain the onboarding of new projects to XCI, way of working, and release model.

4.1. Introduction

Indices and Tables

• Search Page