Dovetail - user guide¶

Dovetail Introduction¶

Welcome to Dovetail’s documentation !

Dovetail is an OPNFV Project.

This project intends to define and provide a set of OPNFV related validation criteria that will provide input for the evaluation of the use of OPNFV trademarks. The dovetail project will be executed with the guidance and oversight of the Compliance and Certification committee and work to secure the goals of the C&C committee for each release.

This project intends to incrementally define qualification criteria that establish the foundations of how we are able to measure the ability to utilize the OPNFV platform, how the platform itself should behave, and how applications may be deployed on the platform.

Contact Dovetail¶

Feedback? Contact us

Revision: 6fc0897edb21e13718f5254b5bb28597191a2efb Build date: 2016-10-25

Config Template Syntax¶

Dovetail uses Functest/Yardstick Docker container to run its testcases. So you need to give configurations for building the container and the commands it needs to do. In dovetail, all of these are defined in config yaml files dovetail/dovetail/conf/functest_config.yml and dovetail/dovetail/conf/yardstick_config.yml.

Functest template syntax¶

For example, you can define your functest_config.yml as:

functest:
  image_name: opnfv/functest
  docker_tag: latest
  envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
         -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
         -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'
  opts: '-id --privileged=true'
  pre_condition:
    cmds:
      - 'echo test for precondition'
  testcase:
    cmds:
      - 'python /home/opnfv/repos/functest/ci/prepare_env.py start'
      - 'python /home/opnfv/repos/functest/ci/run_tests.py -t {{script_testcase}} -r'
  post_condition:
    cmds:
      - ''
  result:
    dir: '/home/opnfv/functest/results'
    store_type: 'file'
    file_path: 'tempest/tempest.log'
    db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'

First, you need to give the image that you want to use for building functest/yardstick container. Besides, there also need some envirnment parameters such as INSTALLER_TYPE and INSTALLER_IP and the options for you container. Then the functest/yardstick container can be build with your settings.

Second, there need three kinds of commands, pre_condition, testcase and post_condition. If you want to do some cleanups or preparations, the commands can be put into pre_condition section orderly. All commands in this section will just be executed once in the begining. The testcase section does the main jobs of the testing. All functest testcases will use the container to execute these commands one by one. After finishing that, the test is accomplished and the results are stored in files or uploaded to database. The post_condition section does some work such as clean Docker images or something else after all testcases finished. All commands in this section will just execute once.

Besides, there need a result section and it gives the directory of the functest/yardstick results. The store_type should be the same with the cmds in testcase. That means if the test results are stored in files, then store_type need to be file and the file_path is also needed. If the test results are uploaded to database, then a db_url is needed for acquiring the results.

Yardstick template syntax¶

The framework of yardstick_config.yml is almost the same as functest_config.yml.

yardstick:
  image_name: opnfv/yardstick
  docker_tag: latest
  envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
         -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
         -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal
         -e EXTERNAL_NETWORK=ext-net'
  opts: '-id --privileged=true'
  pre_condition:
    cmds:
      - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
         source /home/opnfv/repos/yardstick/tests/ci/clean_images.sh && cleanup'
      - 'source /home/opnfv/repos/yardstick/tests/ci/prepare_env.sh &&
         cd /home/opnfv/repos/yardstick && source tests/ci/load_images.sh'
  testcase:
    cmds:
      - 'mkdir -p /home/opnfv/yardstick/results/'
      - 'cd /home/opnfv/repos/yardstick && source tests/ci/prepare_env.sh &&
         yardstick task start tests/opnfv/test_cases/{{script_testcase}}.yaml
         --output-file /home/opnfv/yardstick/results/{{script_testcase}}.out &>
         /home/opnfv/yardstick/results/yardstick.log'
  post_condition:
    cmds:
      - ''
  result:
    dir: '/home/opnfv/yardstick/results'
    store_type: 'file'
    file_path: 'yardstick.log'
    db_url: 'http://testresults.opnfv.org/test/api/v1/results?case=%s&last=1'

The main differences between yardstick_config.yml and functest_config.yml are the commands.

Jinja2 template syntax¶

Note that you can use jinja2 template for your parameters such as {{script_testcase}}. The parameters are defined in dovetail/dovetail/conf/dovetail_config.yml:

parameters:
- name: testcase
  path: '("name",)'
- name: script_testcase
  path: '("scripts", "testcase")'

Here path is the path in testcase config files that you can find the value of parameters. Take script_testcase as the example. For testcase dovetail.ipv6.tc001:

dovetail.ipv6.tc001:
  name: dovetail.ipv6.tc001
  objective: VIM ipv6 operations, to create/delete network, port and subnet in bulk operation
  scripts:
    type: functest
    testcase: tempest_smoke_serial
    sub_testcase_list:
      - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_network
      - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_port
      - tempest.api.network.test_networks.BulkNetworkOpsIpV6Test.test_bulk_create_delete_subnet

The path (“scripts”, “testcase”) means ‘testcase’ subsection of ‘scripts’ section. So follow the path (“scripts”, “testcase”) we can get the value of {{script_testcase}} that is ‘tempest_smoke_serial’.

Command Line Interface¶

Dovetail supports modifying some parameters at the run-time by using the command line interface. The commands can be defined through a config file by developers easily and be used when running certification manually. It now fits three kinds of running, directly running the python script, running after setup and running with docker container.

Define CLI with config file¶

For easy to be modified, dovetail provides dovetail/dovetail/conf/cmd_config.yml to define CLI dynamically.

cli:
  scenario:
    flags:
      - '--scenario'
    default: 'basic'
    help: 'certification scenario.'
  SUT_TYPE:
    flags:
      - '--SUT_TYPE'
      - '-TYPE'
    help: 'Give SUT_TYPE.'
  SUT_IP:
    flags:
      - '--SUT_IP'
      - '-IP'
    help: 'Give SUT_IP.'

The basic yaml file shown above contains three commands named scenario, SUT_TYPE and SUT_IP, and all of them are options. Dovetail uses click module in python to parse these commands, so for each command its keys should be consistent with click’s keys with the exception of flags. There is a list in flags such as ['--SUT_TYPE', '-TYPE'] defining the full name and short name of the command. The full name is necessary but the short name is optional. As you can see, the full name --SUT_TYPE should be the same with this block’s name SUT_TYPE. Other keys such as default and help are the same as defined in python module click.

The first command scenario is used for choosing the scenario to be run, and the cmd config file needs to include this command at least. All other commands are the environment paramenters that will be used to run functest/yardstick containers. For example, the default envs for functest container are given in functest’s config file dovetail/dovetail/conf/functest_config.yml:

envs: '-e INSTALLER_TYPE=compass -e INSTALLER_IP=192.168.200.2
       -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
       -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'

If running with the command python run.py --SUT_TYPE fuel, then the envs will be changed into

envs: '-e INSTALLER_TYPE=fuel -e INSTALLER_IP=192.168.200.2
       -e NODE_NAME=dovetail-pod -e DEPLOY_SCENARIO=ha_nosdn
       -e BUILD_TAG=dovetail -e CI_DEBUG=true -e DEPLOY_TYPE=baremetal'

The envs commands can be added or deleted just by modifying cmd_config.yml rather than changing the source code. Now it only can dynamically modify envs commands. If there is a need for adding non-envs command, besides adding it into cmd_config.yml, some other operations about the source code are also needed.

Run certification with CLI¶

For users, they can use CLI to input their own envs at the run-time instead of modifying the config files of functest and yardstick. So dovetail can supports different environments more flexible with CLI. Dovetail now can be run with three methods, directly running run.py script, running after setup and running in docker container. The uses of CLI are almost the same for these three methods and here takes the first one as the example.

All commands offered by dovetail can be listed by using help command --help.

root@90256c4efd05:~/dovetail/dovetail$ python run.py --help
Usage: run.py [OPTIONS]

Dovetail certification test entry!

Options:
  -TYPE, --SUT_TYPE TEXT  Give SUT_TYPE.
  --DEPLOY_SCENARIO TEXT  Give DEPLOY_SCENARIO.
  --DEPLOY_TYPE TEXT      Give DEPLOY_TYPE.
  -IP, --SUT_IP TEXT      Give SUT_IP.
  --CI_DEBUG TEXT         Give CI_DEBUG.
  --scenario TEXT         certification scenario.
  --help                  Show this message and exit.

All options listed can be used to input special environment values at the run-time. For example:

python run.py --SUT_TYPE compass --SUT_IP 192.168.200.2

There is no need to give all these commands. If it is not given by CLI, it will be set with the system’s environment value. If it is not included in system’s environment variables, it will be set with the default value in functest/yardstick config file.