OpenStack-based cloud infrastructure Testing Cookbook

Introduction

Define the purpose of the chapter which is to:

• Identify Framework Needs, Goals, and Dependencies

• Define Opensource Integration (OVP, Functest, CVC, others)

• Provide Automation Toolchain (list, topology, flow)

Relevant Community Projects and Initiatives

Functest

Functest was initially created to verify OPNFV Installers and Scenarios and then to publish fair, trustable and public results regarding the status of the different opensource technologies, especially for Neutron backends (e.g. Neutron agents, OpenDaylight, OVN, etc.). It has been continuously updated to offer the best testing coverage for any kind of OpenStack and Kubernetes deployments including production environments. It also ensures that the platforms meet Network Functions Virtualization requirements by running and testing VNFs amongst all tests available.

Functest is driven by a true verification of the platform under test as opposed to the interoperability programs such as RefStack or OPNFV Verification Program which select a small subset of Functional tests passing in many different opensource software combinations:

• tests are skipped if an optional support is missing (e.g. Barbican or networking features such as BGPVPN interconnection or Service Function Chaining)

• tests are parameterized (e.g. shared vs non-shared live migration)

• blacklist mechanisms are available if needed

It should be noted that the RefStack lists are included as they are in Functest in the next 3 dedicated testcases:

• refstack_compute (OpenStack Powered Compute)

• refstack_object (OpenStack Powered Object Storage)

• refstack_platform (OpenStack Powered Platform)

Functest also integrates Kubernetes End-to-end tests and allows verifying Kubernetes Conformance (see k8s-conformance).

Dovetail (OVP) mostly leverages on Functest but only runs a small part of Functest (~15% of all functional tests, no benchmarking tests, no VNF deployment and testing). It’s worth mentioning that Functest is patched to disable API verification which has differed from OpenStack rules for years.

Then Functest conforms with the upstream rules (versions, code quality, etc.) and especially their gates (a.k.a. the automatic verification prior to any code review) to preserve the quality between code and deployment. In that case, Functest can be considered as a smooth and lightweight integration of tests developed upstream (and the Functest team directly contributes in these projects: Rally, Tempest, etc.). It’s worth mentioning that, as opposed to the OpenStack Gates leveraging on DevStack, it can check the same already deployed SUT over and over even from a Raspberry PI. Here the testcases can be executed in parallel vs the same deployment instead of being executed vs different pools of virtual machines.

Here are the functional tests (>2000) running in OpenStack gates integrated in Functest Smoke (see Functest daily jobs for more details):

Testcases	Gates
tempest_neutron	Neutron
tempest_cinder	Cinder
tempest_keystone	Keystone
rally_sanity	General
refstack_defcore	General
tempest_full	General
tempest_slow	General
tempest_scenario	General
patrole	Patrole
tempest_barbican	Barbican
networking-bgpvpn	Networking BGP VPN
networking-sfc	Networking SFC

To complete functional testing, Functest also integrates a few performance tools (2-3 hours) as proposed by OpenStack:

Testcases	Benchmarking
rally_full	Control Plane (API) testing
rally_jobs	Control Plane (API) testing
vmtp	Data Plane testing
shaker	Data Plane testing

And VNFs automatically deployed and tested :

Testcases	Benchmarking
cloudify	Cloudify deployment
cloudify_ims	Clearwater IMS deployed via Coudify
heat_ims	Clearwater IMS deployed via Heat
vyos_vrouter	VyOS deployed via Cloudify
juju_epc	OAI deployed via Juju

Functest should be considered as a whole as it meets multiple objectives about the reference implementation:

• verify all APIs (services, advances, features, etc.) exposed by the reference implementation

• compare the reference implementation and local deployments from a functional standpoint and from OpenStack control plane and dataplane capabilities

It’s worth mentioning that Functest already takes into account the first Anuket profiles. Anuket should simply add the next Functest inputs according the reference implementation:

• Functest inputs

• tempest specific configuration

Additional links:

• Homepage

• Run Alpine Functest containers (Iruya)

• Deploy your own Functest CI/CD toolchains

• Functest gates

Yardstick

Bottlenecks

Test Tools

1. Shaker: https://pyshaker.readthedocs.io/en/latest/ (The distributed data-plane testing tool built for OpenStack)

2. Sonubuoy: https://sonobuoy.io/ It is a diagnostic tool that makes it easier to understand the state of a Kubernetes cluster by running a set of plugins (including Kubernetes conformance tests) in an accessible and non-destructive manner.

Scenario Descriptor File (SDF)

As defined by Anuket, Scenario Descriptor File’s (SDF) will be utilized to relay information from the Scenario Designer (or Test Manager), to Release Managers, CI Pipeline Owners, and Installer Agents, to define test scenario content, and specifications.

SDF’s will contain, but not limited to, the following Metadata, Components, Deployment Options, Deployment Tools, and Hardware prerequistes:

• Metadata

  • Name

  • History

  • Purpose

  • Owner

• Components

  • e.g. SDN controllers

  • Versions

  • Optional features, e.g. NFV features

• Deployment Options

  • Hardware types

  • Virtual deploy

  • HA, NUMA

• Deployment Tools

  • Supporting installers.

  • Valid options per installer.

OpenStack Testing Cookbook

At the time of writing, the CI description file is hosted in Functest and only runs the containers listed in RM/RA-1 Requirements. It will be completed by the next Anuket mandatory test cases and then a new CI description file will be proposed in CIRV tree.

Please note the next two points depending on the GNU/Linux distributions and the network settings:

• SELinux: you may have to add –system-site-packages when creating the virtualenv (“Aborting, target uses selinux but python bindings (libselinux-python) aren’t installed!”)

• Proxy: you may set your proxy in env for Ansible and in systemd for Docker https://docs.docker.com/config/daemon/systemd/#httphttps-proxy

To deploy your own CI toolchain running Anuket Compliance:

virtualenv functest --system-site-packages
. functest/bin/activate
pip install ansible
ansible-galaxy install collivier.xtesting
ansible-galaxy collection install ansible.posix community.general community.grafana kubernetes.core community.docker community.postgresql
git clone https://gerrit.opnfv.org/gerrit/functest functest-src
(cd functest-src && git checkout -b stable/wallaby origin/stable/wallaby)
ansible-playbook functest-src/ansible/site.cntt.yml

OpenStack API testing configuration

Here is the default Functest tree as proposed in Functest Wallaby:

• /home/opnfv/functest/openstack.creds

• /home/opnfv/functest/images

Download the images and fill /home/opnfv/functest/openstack.creds as proposed in Functest Wallaby

You may have to modify a few Functest env vars according to the SUT (see env in Functest Wallaby). Be free to modify functest-src/ansible/host_vars/127.0.0.1 at your convenience and then to reconfigure the toolchain:

ansible-playbook functest-src/ansible/site.cntt.yml

Run Anuket OpenStack Testing

Open http://127.0.0.1:8080/job/functest-wallaby-daily/ in a web browser, login as admin/admin and click on “Build with Parameters” (keep the default build_tag value).

If the System under test (SUT) is Anuket compliant, a link to the full archive containing all test results and artifacts will be printed in functest-wallaby-zip’s console. Be free to download it and then to send it to any reviewer committee.

To clean your working dir:

deactivate
rm -rf functest-src functest