=========================================== OPNFV Brahmaputra release note for Functest =========================================== Abstract ======== This document describes the release note of Functest project. License ======= OPNFV Brahmaputra release note for Functest Docs (c) by Morgan Richomme (Orange) OPNFV Brahmaputra release note for Functest Docs are licensed under a Creative Commons Attribution 4.0 International License. You should have received a copy of the license along with this. If not, see . Version history =============== +------------+----------+------------------+------------------------+ | **Date** | **Ver.** | **Author** | **Comment** | | | | | | +------------+----------+------------------+------------------------+ | 2016-02-25 | 1.0.0 | Morgan Richomme | Functest for B release | | | | (Orange) | | +------------+----------+------------------+------------------------+ OPNFV Brahmaputra Release ========================= Functest deals with functional testing of the OPNFV solution. It includes test cases developed within the project and test cases developed in other OPNFV projects and other upstream communities. The internal test cases are: * vPing ssh * vPing userdata * Tempest * Rally * vIMS * ODL The OPNFV projects integrated into Functest framework for automation are: * Promise * Doctor * ONOSFW Release Data ============ +--------------------------------------+--------------------------------------+ | **Project** | functest | | | | +--------------------------------------+--------------------------------------+ | **Repo/tag** | brahmaputra.1.0 | | | | +--------------------------------------+--------------------------------------+ | **Release designation** | Brahmaputra base release | | | | +--------------------------------------+--------------------------------------+ | **Release date** | February 26 2016 | | | | +--------------------------------------+--------------------------------------+ | **Purpose of the delivery** | Brahmaputra base release | | | | +--------------------------------------+--------------------------------------+ Deliverables ============ Software -------- - The Functest Docker image Documents --------- - Installation/configuration guide - User Guide - Developer Guide - Test results per scenario Version change ============== Feature evolution ----------------- - dockerization of Functest - renaming of vPing into vPing userdata - Tempest update, use custom list of test cases - Rally update, use global scenario - Update jenkins logs - support of the different scenarios New features ------------ - introduction of a new vPing test case vPing ssh - introduction of vIMS test case - support of Promise - support of Doctor - support of ONOSFW - scenario management system - creation of a Test collection API - creation of the Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary - creation of Functest dashboard: http://testresults.opnfv.org/dashboard/ Scenario Matrix =============== For Brahmaputra, Functest supports the following scenarios: +----------------+---------+---------+---------+---------+ | Scenario | Apex | Compass | Fuel | Joid | +================+=========+=========+=========+=========+ | odl_l2 | X | X | X | X | +----------------+---------+---------+---------+---------+ | onos | | X | | | +----------------+---------+---------+---------+---------+ | nosdn | | X | X | | +----------------+---------+---------+---------+---------+ | ovs (dpdk) | | | X | | +----------------+---------+---------+---------+---------+ Functest defines the success criteria when having at least 4 consecutive successful runs of a given scenario from the Continuous Integration. The success criteria is defined as follows: * vPing SSH 100% OK * vPing userdata 100% OK * Tempest success rate > 90% * Rally success rate > 90% * ODL success rate = 100% * ONOSFW success rate = 100% * Promise success rate = 100% * vIMS: deployement of the orchestrator and the vIMS VNF successful Other scenarios are currently available but did not meet success criteria for the release but might be added in the incremental scenario update of the release. Brahmaputra limitations ----------------------- - Fuel and Apex Tempest success rate was below 90% but above 80%. Some of the error causes were identified (workers, lack of IP) - vIMS failed in CI for joid/odl_l2 scenario - vPing userdata and vIMS excluded from onos scenario - None of the odl_l3 scenarios has been successful due to vPing ssh issue. - joid/nosdn successful but the complete scenario not run 4 times in a raw - apex/nosdn never run (not a target scenario) but probably succesful See known issues section for details Test and installer/scenario dependencies ======================================== It is not always possible to run all the test cases on all the scenarios. The following table details the dependencies of the test cases per scenario. +----------------+-------------+-------------+-------------+-------------+ | Test cases | Apex | Compass | Fuel | Joid | +================+=============+=============+=============+=============+ | vPing SSH | all | all | all | all | +----------------+-------------+-------------+-------------+-------------+ | vPing userdata | all except | all except | all except | all except | | | ONOS | ONOS | ONOS | ONOS | +----------------+-------------+-------------+-------------+-------------+ | Tempest | all | all | all | all | +----------------+-------------+-------------+-------------+-------------+ | Rally | all | all | all | all | +----------------+-------------+-------------+-------------+-------------+ | ODL | all ODL | all ODL | all ODL | all ODL | +----------------+-------------+-------------+-------------+-------------+ | ONOS | ONOS | ONOS | ONOS | ONOS | +----------------+-------------+-------------+-------------+-------------+ | Promise | no | no | all | all | +----------------+-------------+-------------+-------------+-------------+ | vIMS | all except | all except | all except | all except | | | ONOS | ONOS | ONOS | ONOS | +----------------+-------------+-------------+-------------+-------------+ | Doctor | all | no | no | no | +----------------+-------------+-------------+-------------+-------------+ Test results ============ Test results are available in: - test results document: http://artifacts.opnfv.org/functest/docs/results/index.html - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/ - Test dashboards: http://testresults.opnfv.org/dashboard Known issues ------------ - nova metadata service not supported in ONOS: - it excludes vPing userdata and vIMS test cases - Rally worker issues: - A workaround has been implemented to run Tempest test cases sequentially - It may explain why Tempest scenarios (mainly on apex) do not run the 210 tests - https://bugs.launchpad.net/testrepository/+bug/1538941 - IPv6 issues in tempest suite: - tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools - Tempest code which doesn't reserve big enough allocation - https://bugs.launchpad.net/tempest/+bug/1514457 - Lack of IP addresses available lead to several errors in different test cases - vIMS: - the VM needs to have access to OpenStack API. - Technical architecture may not allow this access (for security reasons) - Orchestrator can be deployed but the vIMS VNF cannot - That is the reason why it fails on joid/odl_l2 scenario on Orange POD 2 Open JIRA tickets ================= +------------------+-----------------------------------------+ | JIRA | Description | +==================+=========================================+ | FUNCTEST-139 | prepare_env failed due to | | | https://pypi.python.org/samples is not | | | accessible | +------------------+-----------------------------------------+ | FUNCTEST-137 | Tempest success rate below 90 on apex | +------------------+-----------------------------------------+ | FUNCTEST-136 | Tempest success rate below 90 on fuel | +------------------+-----------------------------------------+ | FUNCTEST-135 | vPing scenario failed in odl_l3 scenario| +------------------+-----------------------------------------+ | FUNCTEST-124 | odl test suite troubleshooting | +------------------+-----------------------------------------+ Useful links ============ - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing - Functest repo: https://git.opnfv.org/cgit/functest - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/ - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611 - Wiki page for B Release: https://wiki.opnfv.org/functest_release_2 - Functest IRC chan: #opnfv-testperf - Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary - Functest dashboard: http://testresults.opnfv.org/dashboard .. Revision: 9c3f0aa7789e5c68b3e986a5f9d758849a509596 Build date: 2016-04-11