Functest Developer Guide¶

Introduction¶

Functest is a project dealing with functional testing. The project produces its own internal test cases but can also be considered as a framework to support feature and VNF onboarding project testing.

Therefore there are many ways to contribute to Functest. You can:

Develop new internal test cases

Integrate the tests from your feature project

Develop the framework to ease the integration of external test cases

Additional tasks involving Functest but addressing all the test projects may also be mentioned:

The API / Test collection framework

The dashboards

The automatic reporting portals

The testcase catalog

This document describes how, as a developer, you may interact with the Functest project. The first section details the main working areas of the project. The Second part is a list of “How to” to help you to join the Functest family whatever your field of interest is.

Functest developer areas¶

Functest High level architecture¶

Functest is a project delivering test containers dedicated to OPNFV. It includes the tools, the scripts and the test scenarios. Until Danube, Functest produced 2 docker files based on Ubuntu 14.04:

x86 Functest: https://hub.docker.com/r/opnfv/functest/

aarch64 Functest: https://hub.docker.com/r/opnfv/functest_aarch64/

In Euphrates Alpine containers have been introduced in order to lighten the container and manage testing slicing. The new containers are created according to the different tiers:

functest-core: https://hub.docker.com/r/opnfv/functest-core/

functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/

functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/

functest-features: https://hub.docker.com/r/opnfv/functest-features/

functest-components: https://hub.docker.com/r/opnfv/functest-components/

functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/

functest-parser: https://hub.docker.com/r/opnfv/functest-parser/

functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/

Standalone functest dockers are maintained for Euphrates but Alpine containers are recommended.

Functest can be described as follow:

+----------------------+
|                      |
|   +--------------+   |                  +-------------------+
|   |              |   |    Public        |                   |
|   | Tools        |   +------------------+      OPNFV        |
|   | Scripts      |   |                  | System Under Test |
|   | Scenarios    |   |                  |                   |
|   |              |   |                  |                   |
|   +--------------+   |                  +-------------------+
|                      |
|    Functest Docker   |
|                      |
+----------------------+

Functest internal test cases¶

The internal test cases in Euphrates are:

api_check

connection_check

snaps_health_check

vping_ssh

vping_userdata

odl

odl-netvirt

odl-fds

rally_full

rally_sanity

tempest_smoke_serial

tempest_full_parallel

cloudify_ims

By internal, we mean that this particular test cases have been developed and/or integrated by functest contributors and the associated code is hosted in the Functest repository. An internal case can be fully developed or a simple integration of upstream suites (e.g. Tempest/Rally developed in OpenStack, or odl suites are just integrated in Functest).

The structure of this repository is detailed in [1]. The main internal test cases are in the opnfv_tests subfolder of the repository, the internal test cases can be grouped by domain:

sdn: odl, odl_netvirt, odl_fds

openstack: api_check, connection_check, snaps_health_check, vping_ssh, vping_userdata, tempest_*, rally_*

vnf: cloudify_ims

If you want to create a new test case you will have to create a new folder under the testcases directory (See next section for details).

Functest external test cases¶

The external test cases are inherited from other OPNFV projects, especially the feature projects.

The external test cases are:

barometer

bgpvpn

doctor

domino

fds

parser

promise

refstack_defcore

snaps_smoke

functest-odl-sfc

orchestra_clearwaterims

orchestra_openims

cloudify_vrouter

juju_vepc

External test cases integrated in previous versions but not released in Euphrates:

copper

moon

netready

security_scan

The code to run these test cases is hosted in the repository of the project. Please note that orchestra test cases are hosted in Functest repository and not in orchestra repository. Cloudify_vrouter and juju_vepc code is also hosted in functest as there are no dedicated projects.

Functest framework¶

Functest is a framework.

Historically Functest is released as a docker file, including tools, scripts and a CLI to prepare the environment and run tests. It simplifies the integration of external test suites in CI pipeline and provide commodity tools to collect and display results.

Since Colorado, test categories also known as tiers have been created to group similar tests, provide consistent sub-lists and at the end optimize test duration for CI (see How To section).

The definition of the tiers has been agreed by the testing working group.

The tiers are:

healthcheck
smoke
features
components
vnf

Functest abstraction classes¶

In order to harmonize test integration, abstraction classes have been introduced:

testcase: base for any test case

unit: run unit tests as test case

feature: abstraction for feature project

vnf: abstraction for vnf onboarding

The goal is to unify the way to run tests in Functest.

Feature, unit and vnf_base inherit from testcase:

            +-----------------------------------------+
            |                                         |
            |         TestCase                        |
            |                                         |
            |         - init()                        |
            |         - run()                         |
            |         - push_to_db()                  |
            |         - is_successful()               |
            |                                         |
            +-----------------------------------------+
               |               |                   |
               V               V                   V
+--------------------+   +--------------+   +--------------------------+
|                    |   |              |   |                          |
|    feature         |   |    unit      |   |      vnf                 |
|                    |   |              |   |                          |
|                    |   |              |   |  - prepare()             |
|  - execute()       |   |              |   |  - deploy_orchestrator() |
| BashFeature class  |   |              |   |  - deploy_vnf()          |
|                    |   |              |   |  - test_vnf()            |
|                    |   |              |   |  - clean()               |
+--------------------+   +--------------+   +--------------------------+

Testcase¶

functest.core.testcase module — OPNFV Functest master documentation

functest.core.testcase module¶

Define the parent class of all Functest TestCases.

class functest.core.testcase.OSGCTestCase(**kwargs)¶

Bases: functest.core.testcase.TestCase

Model for single test case which requires an OpenStack Garbage Collector.

clean()¶: Clean the OpenStack resources.

create_snapshot()¶

Create a snapshot listing the OpenStack resources.

Returns:: TestCase.EX_OK if os_snapshot.main() returns 0. TestCase.EX_RUN_ERROR otherwise.

class functest.core.testcase.TestCase(**kwargs)¶

Bases: object

Base model for single test case.

EX_OK = 0¶: everything is OK

EX_PUSH_TO_DB_ERROR = 69¶: push_to_db() failed

EX_RUN_ERROR = 70¶: run() failed

EX_TESTCASE_FAILED = 68¶: results are false

clean()¶

Clean the resources.

It can be overriden if resources must be deleted after running the test case.

create_snapshot()¶

Save the testing environment before running test.

It can be overriden if resources must be listed running the test case.

Returns:: TestCase.EX_OK

get_duration()¶

Return the duration of the test case.

Returns:: duration if start_time and stop_time are set “XX:XX” otherwise.

is_successful()¶

Interpret the result of the test case.

It allows getting the result of TestCase. It completes run() which only returns the execution status.

It can be overriden if checking result is not suitable.

Returns:: TestCase.EX_OK if result is ‘PASS’. TestCase.EX_TESTCASE_FAILED otherwise.

push_to_db()¶

Push the results of the test case to the DB.

It allows publishing the results and to check the status.

It could be overriden if the common implementation is not suitable. The following attributes must be set before pushing the results to DB:

project_name,

case_name,

result,

start_time,

stop_time.

Returns:: TestCase.EX_OK if results were pushed to DB. TestCase.EX_PUSH_TO_DB_ERROR otherwise.

run(**kwargs)¶

Run the test case.

It allows running TestCase and getting its execution status.

The subclasses must override the default implementation which is false on purpose.

The new implementation must set the following attributes to push the results to DB:

result,

start_time,

stop_time.

Args:: kwargs: Arbitrary keyword arguments.
Returns:: TestCase.EX_RUN_ERROR.

Feature¶

functest.core.feature module — OPNFV Functest master documentation

functest.core.feature module¶

Define the parent classes of all Functest Features.

Feature is considered as TestCase offered by Third-party. It offers helpers to run any python method or any bash command.

class functest.core.feature.BashFeature(**kwargs)¶

Bases: functest.core.feature.Feature

Class designed to run any bash command.

execute(**kwargs)¶

Execute the cmd passed as arg

Args:: kwargs: Arbitrary keyword arguments.
Returns:: 0 if cmd returns 0, -1 otherwise.

class functest.core.feature.Feature(**kwargs)¶

Bases: functest.core.testcase.TestCase

Base model for single feature.

execute(**kwargs)¶

Execute the Python method.

The subclasses must override the default implementation which is false on purpose.

The new implementation must return 0 if success or anything else if failure.

Args:: kwargs: Arbitrary keyword arguments.
Returns:: -1.

run(**kwargs)¶

Run the feature.

It allows executing any Python method by calling execute().

It sets the following attributes required to push the results to DB:

result,

start_time,

stop_time.

It doesn’t fulfill details when pushing the results to the DB.

Args:: kwargs: Arbitrary keyword arguments.
Returns:: TestCase.EX_OK if execute() returns 0, TestCase.EX_RUN_ERROR otherwise.

Unit¶

functest.core.unit module — OPNFV Functest master documentation

functest.core.unit module¶

Define the parent class to run unittest.TestSuite as TestCase.

class functest.core.unit.Suite(**kwargs)¶

Bases: functest.core.testcase.TestCase

Base model for running unittest.TestSuite.

run(**kwargs)¶

Run the test suite.

It allows running any unittest.TestSuite and getting its execution status.

By default, it runs the suite defined as instance attribute. It can be overriden by passing name as arg. It must conform with TestLoader.loadTestsFromName().

It sets the following attributes required to push the results to DB:

result,

start_time,

stop_time,

details.

Args:: kwargs: Arbitrary keyword arguments.
Returns:: TestCase.EX_OK if any TestSuite has been run, TestCase.EX_RUN_ERROR otherwise.

VNF¶

functest.core.vnf module — OPNFV Functest master documentation

functest.core.vnf module¶

Define the parent class of all VNF TestCases.

exception functest.core.vnf.OrchestratorDeploymentException¶

Bases: exceptions.Exception

Raise when orchestrator cannot be deployed.

exception functest.core.vnf.VnfDeploymentException¶

Bases: exceptions.Exception

Raise when VNF cannot be deployed.

class functest.core.vnf.VnfOnBoarding(**kwargs)¶

Bases: functest.core.testcase.TestCase

Base model for VNF test cases.

clean()¶

Clean VNF test case.

It is up to the test providers to delete resources used for the tests. By default we clean:

the user,

the tenant

deploy_orchestrator()¶

Deploy an orchestrator (optional).

If this method is overriden then raise orchestratorDeploymentException if error during orchestrator deployment

deploy_vnf()¶

Deploy the VNF

This function MUST be implemented by vnf test cases. The details section MAY be updated in the vnf test cases.

The deployment can be executed via a specific orchestrator or using build-in orchestrators such as heat, OpenBaton, cloudify, juju, onap, ...

Returns:: True if the VNF is properly deployed False if the VNF is not deployed

Raise VnfDeploymentException if error during VNF deployment

prepare()¶

Prepare the environment for VNF testing:

Creation of a user,

Creation of a tenant,

Allocation admin role to the user on this tenant

Returns base.TestCase.EX_OK if preparation is successfull

Raise VnfPreparationException in case of problem

run(**kwargs)¶

Run of the VNF test case:

Deploy an orchestrator if needed (e.g. heat, cloudify, ONAP,...),

Deploy the VNF,

Perform tests on the VNF

A VNF test case is successfull when the 3 steps are PASS If one of the step is FAIL, the test case is FAIL

Returns:: TestCase.EX_OK if result is ‘PASS’. TestCase.EX_TESTCASE_FAILED otherwise.

test_vnf()¶

Test the VNF

This function MUST be implemented by vnf test cases. The details section MAY be updated in the vnf test cases.

Once a VNF is deployed, it is assumed that specific test suite can be run to validate the VNF. Please note that the same test suite can be used on several test case (e.g. clearwater test suite can be used whatever the orchestrator used for the deployment)

Returns:: True if VNF tests are PASS False if test suite is FAIL

Raise VnfTestException if error during VNF test

exception functest.core.vnf.VnfPreparationException¶

Bases: exceptions.Exception

Raise when VNF preparation cannot be executed.

exception functest.core.vnf.VnfTestException¶

Bases: exceptions.Exception

Raise when VNF cannot be tested.

see [5] to get code samples

Functest util classes¶

In order to simplify the creation of test cases, Functest develops also some functions that are used by internal test cases. Several features are supported such as logger, configuration management and Openstack capabilities (snapshot, clean, tacker,..). These functions can be found under <repo>/functest/utils and can be described as follows:

functest/utils/
|-- config.py
|-- constants.py
|-- decorators.py
|-- env.py
|-- functest_utils.py
|-- openstack_clean.py
|-- openstack_snapshot.py
|-- openstack_tacker.py
`-- openstack_utils.py

It is recommended to use the SNAPS-OO library for deploying OpenStack instances. SNAPS [4] is an OPNFV project providing OpenStack utils.

TestAPI¶

Functest is using the Test collection framework and the TestAPI developed by the OPNFV community. See [6] for details.

Reporting¶

A web page is automatically generated every day to display the status based on jinja2 templates [3].

Dashboard¶

Additional dashboarding is managed at the testing group level, see [7] for details.

How TOs¶

See How to section on Functest wiki [8]

References¶

[1]: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide

[2]: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide

[3]: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting

[4]: https://git.opnfv.org/snaps/

[5] : http://testresults.opnfv.org/functest/framework/index.html

[6]: http://docs.opnfv.org/en/latest/testing/testing-dev.html

[7]: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba

[8]: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932

IRC support chan: #opnfv-functest

Functest Developer Guide¶

Introduction¶

Functest developer areas¶

Functest High level architecture¶

Functest internal test cases¶

Functest external test cases¶

Functest framework¶

Functest abstraction classes¶

Testcase¶

functest.core.testcase module¶

Related Topics

This Page

Feature¶

functest.core.feature module¶

Related Topics

This Page

Unit¶

functest.core.unit module¶

Related Topics

This Page

VNF¶

functest.core.vnf module¶

Related Topics

This Page

Functest util classes¶

TestAPI¶

Reporting¶

Dashboard¶

How TOs¶

References¶

Table Of Contents

This Page