Dovetail Overview¶

Dovetail Tool¶

What is Dovetail tool¶

A toolset for providing an evaluation of a deployed cloud environment for OPNFV compliance.

Overview¶

Dovetail tool intends on being THE source of tools to provide evaluation of the use of OPNFV trademarks.

Dovetail tool provides users with a wrapper which leverage other test projects, such as Functest, Yardstick, etc, that helps to provide evaluation of the use of OPNFV trademarks. It does so by validating any NFV(Network Function Virtualization) implementation against the test cases in Dovetail.

Get involved!¶

Wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Home
Git Repo: https://git.opnfv.org/dovetail

Dovetail Tool Installation¶

Abstract¶

The Dovetail tool supports installation either directly on the Linux host, or within a Docker container. The detailed installation procedure on the Linux host or via the Docker container are shown in the following sections.

There is a need to config the following parameters before Dovetail tool running on the SUT (System Under Test):

SUT_TYPE
    SUT type, e.g., apex, compass, fuel, joid, etc
SUT_IP
    SUT external network IP, e.g., 192.168.200.2
NODE_NAME
    this can be shown in the test result for users to see which pod the dovetail tool runs
DEPLOY_SCENARIO
    deployment scenario, e.g., os-nosdn-nofeature-ha
BUILD_TAG
    this can be shown in the test result for users to identify logs
CI_DEBUG
    true for debug information printed and false for not printed
DEPLOY_TYPE
    baremetal or virtual

The above configuration can be achieved by

modifying the environment variables in files which live under /dovetail/conf/ directory
set and use Linux environment variables using export command
set and use these variables when using dovetail run command line, for details see the `Dovetail Command-line Interface`_ section
enable the OpenStack credential file, which can be achieved by using dovetail run --creds </path/creds>

Dovetail tool installation on local Linux host environment¶

In order to install Dovetail into a local Linux host environment, the following steps should be performed:

Downloading Dovetail tool¶

Source code downloading:

git clone https://git.opnfv.org/dovetail

Environment preparation¶

Dovetail configuration:

cd {dovetail_path}/dovetail/conf
vim functest_config.yml
vim yardstick_config.yml

To modify the envs parameters, e.g., SUT_TYPE, SUT_IP, NODE_NAME, DEPLOY_SCENARIO, BUILD_TAG, CI_DEBUG, DEPLOY_TYPE, in the yml files.

System dependencies and project configuration information are listed in requirements.txt, setup.py and setup.cfg, in Ubuntu environment, to run

apt-get -y install python-pip
pip install -e .

in centos and redhat environment, to run

yum -y install python-pip
pip install -e .

in fedora environment, to run

dnf -y install python-pip
pip install -e .

then the setup requirements are installed and dovetail commmand-line can be used, by using dovetail --help, dovetail command-line details are shown.

The tool tox is used, which configuration information is defined in tox.ini, unitttest, flake8-checking, coverage check, etc, can be done by running:

pip install tox
tox

Compliance and certification test cases¶

The compliance and certification test cases can be defined under the /dovetail/compliance directory, which is defined in yaml format. A sample file named compliance_set.yml is provided as follows:

compliance_set:
  name: compliance_set
  testcases_list:
    - dovetail.ipv6.tc001

The testcase listed here is dovetail.ipv6.tc001, defined within dovetail/testcase.

Running Dovetail tool¶

After environment preparation is complete and test cases added, the Dovetail tool can be run with

dovetail run --testsuite compliance_set

The value compliance_set passed to the testsuite flag can be replaced with the testsuite yaml file name which want to be run. If no argument is given, the compliance_set testsuite will be run as the default.

Moreover, the testcases in given testarea can be run with testarea command line argument, such as testarea ipv6 in compliance_set

dovetail run --testsuite compliance_set --testarea ipv6

Dovetail provides some sets, debug, proposed_tests and compliance_set, debug is used for locally and Continuous Integration(CI) developing purpose, which provides typical testcase examples, feel free to edit it when develops locally, such as only to run a testcase which only takes minutes. proposed_tests is the testcase candidate which mainly comes from the wiki link [1]. compliance_set is used for compliance. Moreover, dovetail tool can be easily extended to support more complicated compliance requirements, such as feature set based or scenario based compliance.

If you want to run the debug testsuite, just run with

dovetail run --testsuite debug

Running Dovetail in a Docker container¶

The Dovetail tool can be run in a Docker container by utilizing the following steps:

Pull Dovetail Docker image from public Dockerhub¶

sudo docker pull opnfv/dovetail:<Tag>

<Tag> here is the version, ‘latest’ is used for the master branch.

(Optional) Create Docker image from Dockerfile¶

Instead of an official Docker image, you can build your own Docker image from Dockerfile(s) located under the docker/ directory. The Dockerfile is based on Ubuntu and the Dockerfile.centos7 file is for a CentOS based Docker image.

cd {dovetail_path}/dovetail/docker
docker build --no-cache -t opnfv/dovetail:<Tag> --build-arg BRANCH=master .

Dovetail Docker container creation¶

Next, create the dovetail-docker-env file to define the environment parameters

INSTALLER_TYPE=compass
INSTALLER_IP=192.168.200.2
DEPLOY_TYPE=baremetal
DEPLOY_SCENARIO=ha-nosdn
CI_DEBUG=true

or if an OpenStack credential file is provided.

Then to instantiate the Dovetail Docker container, execute:

sudo docker run --privileged=true --rm -t \
     --env-file dovetail-docker-env OR </path/creds> \
     -v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \
     -v /var/run/docker.sock:/var/run/docker.sock \
     --name <Dovetail_Container_Name> (optional) \
     opnfv/dovetail:<Tag> /bin/bash

To attach dovetail container and Running test cases¶

Before connecting to the container, you can check the container status by running

sudo docker ps -a

Attach to the container by starting it and obtaining a bash prompt with

sudo docker exec -it <Dovetail_Container_Name>/<Container_Id> bash

Inside the container the following commands can be executed to trigger the testing

dovetail run --testsuite compliance_set

Offline Support¶

There are some SUTs that are isolated from the public internet, so offline support is needed. The idea is to provide all of the packages of dovetail release in http://artifacts.opnfv.org, then the user can download and transfer to their inner development environment.

The packages are shown in [2]

TO DO: to introduce more when it is mature enough.

Results Output¶

The running log is stored in /home/opnfv/dovetail/results/dovetail.log. The compliance report is stored in /home/opnfv/dovetail/results/dovetail_report.txt.

Dovetail Version and Release¶

Dovetail version information is defined in setup.cfg. At the time of release, it is the dovetail team’s responsibility to set the version value in setup.cfg.

[1]	https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases.

[2]	http://artifacts.opnfv.org/dovetail.html.

Template Syntax¶

Testcase Template Syntax¶

The testcases used for compliance and certification are defined in the dovetail/testcase directory, which are written in yaml format. Take the testcase ipv6.tc001.yml as an example. It is shown as:

---
dovetail.ipv6.tc001:
  name: dovetail.ipv6.tc001
  objective: Bulk creation and deletion of IPv6 networks, ports and subnets
  validate:
    type: functest
    testcase: tempest_smoke_serial
    pre_condition:
      - 'echo test for precondition in testcase'
    cmds:
      - 'functest env prepare'
      - 'functest testcase run {{validate_testcase}}'
    post_condition:
      - 'echo test for precondition in testcase'
  report:
    sub_testcase_list:
      - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
      - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
      - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet

At least three sections named name, objective, validate, report must be included
Section name distinguishes different test cases used for compliance, it is composed of 3 parts, dovetail., belongs to which test area, and the serial number
Section objective briefly describes what this testcase does
Section validate defines the scripts and configurations for the validation of the test case. type defines which method is used to validate, 3 ways, i.e., functest, yardstick and shell is supported currently. testcase represents the testcases in slicing/tier.
Section report defines the sub_testcases to be run. For yardstick, since it is not sliced by now, sub_testcase_list is not needed, only to edit the testcase part in section validate, such as yardstick_tc027

Config Template Syntax¶

For Dovetail tool, the config files are located in dovetail/dovetail/conf, which are written in yaml format. As both functest and yardstick are utilized by Dovetail, their configuration files should be configured as follows, within the functest_config.yml and yardstick_config.yml files, respectively.

Functest config template syntax¶

An example of functest configuration is shown as follows:

functest:
  image_name: opnfv/functest
  docker_tag: latest
  envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
         -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
         -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
  opts: '-id --privileged=true'
  pre_condition:
    cmds:
      - 'echo test for precondition'
  testcase:
    cmds:
      - 'python /home/opnfv/repos/functest/ci/prepare_env.py start'
      - 'python /home/opnfv/repos/functest/ci/run_tests.py -t {{script_testcase}} -r'
  post_condition:
    cmds:
      - ''
  result:
    dir: '/home/opnfv/functest/results'
    store_type: 'file'
    file_path: 'tempest/tempest.log'
    db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'

image_name and docker_tag sections define the docker image pulled from the public dockerhub.
envs should be correctly configed according to the SUT(System Under Test).
pre_condition represents some cleanups or preparations. testcase represents the testcases running cmds. post_condition represents some cleanups needed after all testcases finished.
result section gives the directory of the dovetail tool test result. db_url gives the database URL of the dovetail results to be stored.

Yardstick config template syntax¶

The configuration in yardstick_config.yml is similiar to functest_config.yml, and an example is shown as follows:

yardstick:
  image_name: opnfv/yardstick
  docker_tag: latest
  envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
         -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
         -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal
         -e EXTERNAL_NETWORK=ext-net'
  opts: '-id --privileged=true'
  pre_condition:
    cmds:
      - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
         source /home/opnfv/repos/yardstick/tests/ci/clean_images.sh && cleanup'
      - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
         cd /home/opnfv/repos/yardstick && source tests/ci/load_images.sh'
  testcase:
    cmds:
      - 'mkdir -p /home/opnfv/yardstick/results/'
      - 'cd /home/opnfv/repos/yardstick && source tests/ci/prepare_env.sh &&
         yardstick task start tests/opnfv/test_cases/{{script_testcase}}.yaml
         --output-file /home/opnfv/yardstick/results/{{script_testcase}}.out &>
         /home/opnfv/yardstick/results/yardstick.log'
  post_condition:
    cmds:
      - ''
  result:
    dir: '/home/opnfv/yardstick/results'
    store_type: 'file'
    file_path: 'yardstick.log'
    db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'

The main differences between yardstick_config.yml and functest_config.yml are the cmds subsection.

Jinja2 template syntax¶

Jinja2 module can be used to config the {{script_testcase}}. The parameters are defined in dovetail/dovetail/conf/dovetail_config.yml:

parameters:
- name: testcase
  path: '("name",)'
- name: script_testcase
  path: '("scripts", "testcase")'

Here path is the path defined in the testcase configuration files. Take script_testcase as an example. For testcase dovetail.ipv6.tc001:

dovetail.ipv6.tc001:
  name: dovetail.ipv6.tc001
  objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation
  scripts:
    type: functest
    testcase: tempest_smoke_serial
    sub_testcase_list:
      - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
      - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
      - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet

The path (“scripts”, “testcase”) means ‘testcase’ is the subsection of ‘scripts’ section. From above, by following the path (“scripts”, “testcase”) we can get the value of {{script_testcase}} is ‘tempest_smoke_serial’.

Dovetail Command-line Interface¶

Command Line Introduction¶

The Dovetail command-line interface provides a method for interacting with Dovetail from the console. For help on the dovetail command, enter:

dovetail --help

dovetail optional arguments:

--version
    show program's version number and exit
list <testsuite_name>
    list the testsuite details
show <testcase_name>
    show the testcase details
run <arguments>
    run the testcases

For dovetail list, if the <testsuite_name> is omitted, all the testsuites defined under /dovetail/compliance directory and related testcases will be listed, otherwise, the test area and test cases defined in <testsuite_name> are listed.

For dovetail show, the <testcase_name> is required, the contents defined in <testcase_name>.yml is shown.

For dovetail run, by running dovetail run --help, the dovetail run usage is shown as:

Options
-t, –SUT_TYPE	Installer type of the system under test (SUT).
–creds	Openstack credential file location
-i, –SUT_IP	IP of the system under test (SUT).
-d, –debug	True for showing debug log on screen.
-f, –func_tag	Overwrite tag for functest docker container (e.g. stable or latest)
-y, –yard_tag	Overwrite tag for yardstick docker container (e.g. stable or latest)
–testarea	compliance testarea within testsuite
–testsuite	compliance testsuite.
-h, –help	Show this message and exit.

If no arguments are given, the default testsuite will be performed, i.e., the compliance_set testsuite with default configurations.

For more information about dovetail command-line interface, please refer to the wiki page [3]_

Parameters definition with config file¶

The default dovetail run parameters can be modified in dovetail/dovetail/conf/cmd_config.yml, which is shown as:

cli:
  arguments:
    config:
      # This is a simple example of arguments.
      # Dovetail has no need of this kind of parameters currently.
      # The arguments must be given orderly at the run-time.
      #
      # docker_tag:
      #   flags: 'docker_tag'
      #   path:
      #     - 'functest/docker_tag'
      #     - 'yardstick/docker_tag'
    control:

  options:
    config:
      SUT_TYPE:
        flags:
          - '--SUT_TYPE'
          - '-t'
        path:
          - 'functest/envs'
          - 'yardstick/envs'
        help: 'Installer type of the system under test (SUT).'
      yard_tag:
        flags:
          - '--yard_tag'
          - '-y'
        path:
          - 'yardstick/docker_tag'
        help: 'Overwrite tag for yardstick docker container (e.g. stable or latest)'
      func_tag:
        flags:
          - '--func_tag'
          - '-f'
        path:
          - 'functest/docker_tag'
        help: 'Overwrite tag for functest docker container (e.g. stable or latest)'
    control:
      testsuite:
        flags:
          - '--testsuite'
        default: 'compliance_set'
        help: 'compliance testsuite.'
      testarea:
        flags:
          - '--testarea'
        default: 'full'
        help: 'compliance testarea within testsuite'

The Click module is used to parse parameters defined in the above config file, two subsections are included in this file, arguments and options, which corresponds to two types of parameters in Click.

Arguments and Options¶

Only options is used currently, which means parameters can be given (or not) without sequence restriction.

Config and control¶

All options/arguments are divided into two parts: config and control. The config ones are used for updating functest or yardstick config files according to the path given. For example, functest’s config file is dovetail/dovetail/conf/functest_config.yml, following is a simple example:

docker_tag: latest
envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
       -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
       -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'

If running with the command python run.py --SUT_TYPE fuel -f stable, then the configs will be changed into

docker_tag: stable
envs: '-e INSTALLER_TYPE=fuel -e INSTALLER_IP=192.168.200.2
       -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
       -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'

The config options/arguments can be added or deleted by modifying cmd_config.yml rather than changing the source code. However, for control command, besides adding it into cmd_config.yml, some other operations about the source code are also needed.

. [3] https://wiki.opnfv.org/display/dovetail/Dovetail+Command+Line