Overview of the Functest suitesΒΆ
Functest is the OPNFV project primarily targeting function testing. In the Continuous Integration pipeline, it is launched after an OPNFV fresh installation to validate and verify the basic functions of the infrastructure.
The current list of test suites can be distributed over 4 main domains: VIM (Virtualised Infrastructure Manager), Controllers (i.e. SDN Controllers), Features and VNF (Virtual Network Functions).
| Domain | Tier | Test case | Comments |
|---|---|---|---|
| VIM | healthcheck | healthcheck | Verify basic operation in VIM |
| smoke | vPing_SSH | NFV “Hello World” using an SSH connection to a destination VM over a created floating IP address on the SUT Public / External network. Using the SSH connection a test script is then copied to the destination VM and then executed via SSH. The script will ping another VM on a specified IP address over the SUT Private Tenant network. | |
| vPing_userdata | Uses Ping with given userdata to test intra-VM connectivity over the SUT Private Tenant network. The correct operation of the NOVA Metadata service is also verified in this test. | ||
| tempest_smoke _serial | Generate and run a relevant Tempest Test Suite in smoke mode. The generated test set is dependent on the OpenStack deployment environment. | ||
| rally_sanity | Run a subset of the OpenStack Rally Test Suite in smoke mode | ||
| openstack | tempest_full _parallel | Generate and run a full set of the OpenStack Tempest Test Suite. See the OpenStack reference test suite [2]. The generated test set is dependent on the OpenStack deployment environment. | |
| rally_full | Run the OpenStack testing tool benchmarking OpenStack modules See the Rally documents [3]. | ||
| Controllers | sdn_suites | odl | Opendaylight Test suite Limited test suite to check the basic neutron (Layer 2) operations mainly based on upstream testcases. See below for details |
| onos | Test suite of ONOS L2 and L3 functions. See ONOSFW User Guide for details. | ||
| Features | features | Promise | Resource reservation and management project to identify NFV related requirements and realize resource reservation for future usage by capacity management of resource pools regarding compute, network and storage. See Promise User Guide for details. |
| Doctor | Doctor platform, as of Colorado release, provides the three features: * Immediate Notification * Consistent resource state awareness for compute host down * Valid compute host status given to VM owner See Doctor User Guide for details | ||
| bgpvpn | Implementation of the OpenStack bgpvpn API from the SDNVPN feature project. It allows for the creation of BGP VPNs. See SDNVPN User Guide for details | ||
| security_scan | Implementation of a simple security scan. (Currently available only for the Apex installer environment) | ||
| onos-sfc | SFC testing for onos scenarios TODO See for details | ||
| odl-sfc | SFC testing for odl scenarios TODO See for details | ||
| domino | Domino provides TOSCA template distribution service for network service and VNF descriptors among MANO components e.g., NFVO, VNFM, VIM, SDN-C, etc., as well as OSS/BSS functions. See Domino User Guide for details | ||
| copper | Deployment policy TODO See for details | ||
| multisites | Multisites TODO See for details | ||
| moon | Security management system TODO See for details | ||
| VNF | vnf | vims | Example of a real VNF deployment to show the NFV capabilities of the platform. The IP Multimedia Subsytem is a typical Telco test case, referenced by ETSI. It provides a fully functional VoIP System |
| parser | Parser is an integration project which aims to provide placement/deployment templates translation for OPNFV platform, including TOSCA -> HOT, POLICY -> TOSCA and YANG -> TOSCA. See Parser User Guide for details |
As shown in the above table, Functest is structured into different ‘domains’, ‘tiers’ and ‘test cases’. Each ‘test case’ usually represents an actual ‘Test Suite’ comprised -in turn- of several test cases internally.
Test cases also have an implicit execution order. For example, if the early ‘healthcheck’ Tier testcase fails, or if there are any failures in the ‘smoke’ Tier testcases, there is little point to launch a full testcase execution round.
An overview of the Functest Structural Concept is depicted graphically below:
Some of the test cases are developed by Functest team members, whereas others are integrated from upstream communities or other OPNFV projects. For example, Tempest is the OpenStack integration test suite and Functest is in charge of the selection, integration and automation of those tests that fit suitably to OPNFV.
The Tempest test suite is the default OpenStack smoke test suite but no new test cases have been created in OPNFV Functest.
The results produced by the tests run from CI are pushed and collected into a NoSQL database. The goal is to populate the database with results from different sources and scenarios and to show them on a Functest Dashboard. A screenshot of a live Functest Dashboard is shown below:
There is no real notion of Test domain or Test coverage. Basic components (VIM, SDN controllers) are tested through their own suites. Feature projects also provide their own test suites with different ways of running their tests.
vIMS test case was integrated to demonstrate the capability to deploy a relatively complex NFV scenario on top of the OPNFV infrastructure.
Functest considers OPNFV as a black box. As of Colorado release the OPNFV offers a lot of potential combinations:
- 3 controllers (OpenDaylight, ONOS, OpenContrail)
- 4 installers (Apex, Compass, Fuel, Joid)
Most of the tests are runnable by any combination, but some tests might have restrictions imposed by the utilized installers or due to the available deployed features. The system uses the environment variables (INSTALLER_IP and DEPLOY_SCENARIO) to automatically determine the valid test cases; for each given environment.
In the Colorado OPNFV System release a convenience Functest CLI utility is also introduced to simplify setting up the Functest evironment, management of the OpenStack environment (e.g. resource clean-up) and for executing tests. The Functest CLI organised the testcase into logical Tiers, which contain in turn one or more testcases. The CLI allow execution of a single specified testcase, all test cases in a specified Tier, or the special case of execution of ALL testcases. The Functest CLI is introduced in more detail in the section `Executing the functest suites`_ of this document.
