2021-04-29 13:57:41,246 - xtesting.ci.run_tests - INFO - Deployment description:

+-------------------------+------------------------------------------------------------+
|         ENV VAR         |                           VALUE                            |
+-------------------------+------------------------------------------------------------+
|         CI_LOOP         |                           daily                            |
|          DEBUG          |                            true                            |
|     DEPLOY_SCENARIO     |                            ovs                             |
|      INSTALLER_TYPE     |                          airship                           |
|        BUILD_TAG        |                        13FPWFFUM0G0                        |
|        NODE_NAME        |                        intel-pod17                         |
|       TEST_DB_URL       |      http://testresults.opnfv.org/test/api/v1/results      |
|     TEST_DB_EXT_URL     |      http://testresults.opnfv.org/test/api/v1/results      |
|     S3_ENDPOINT_URL     |               https://storage.googleapis.com               |
|        S3_DST_URL       |     s3://artifacts.opnfv.org/airship/13FPWFFUM0G0/airs     |
|                         |     hip-opnfv-functest-benchmarking-latest-rally_full-     |
|                         |                          run-214                           |
|       HTTP_DST_URL      |     http://artifacts.opnfv.org/airship/13FPWFFUM0G0/ai     |
|                         |         rship-opnfv-functest-benchmarking-latest-          |
|                         |                     rally_full-run-214                     |
+-------------------------+------------------------------------------------------------+

2021-04-29 13:57:41,264 - xtesting.ci.run_tests - INFO - Loading test case 'rally_full'...
2021-04-29 13:58:31,235 - xtesting.ci.run_tests - INFO - Running test case 'rally_full'...
2021-04-29 13:59:16,568 - functest.opnfv_tests.openstack.rally.rally - INFO - Creating Rally environment...
2021-04-29 13:59:23,130 - functest.opnfv_tests.openstack.rally.rally - INFO - rally deployment create --fromenv --name opnfv-rally
+--------------------------------------+----------------------------+-------------+------------------+--------+
| uuid                                 | created_at                 | name        | status           | active |
+--------------------------------------+----------------------------+-------------+------------------+--------+
| 475eaff9-f5cd-46ab-8b1c-6c36dbc836ee | 2021-04-29T13:59:22.868277 | opnfv-rally | deploy->finished |        |
+--------------------------------------+----------------------------+-------------+------------------+--------+
Using deployment: 475eaff9-f5cd-46ab-8b1c-6c36dbc836ee
~/.rally/openrc was updated

HINTS:

* To use standard OpenStack clients, set up your env by running:
	source ~/.rally/openrc
  OpenStack clients are now configured, e.g run:
	openstack image list

2021-04-29 13:59:28,102 - functest.opnfv_tests.openstack.rally.rally - INFO - rally deployment check
--------------------------------------------------------------------------------
Platform openstack:
--------------------------------------------------------------------------------

Available services:
+-------------+----------------+-----------+
| Service     | Service Type   | Status    |
+-------------+----------------+-----------+
| __unknown__ | placement      | Available |
| cinder      | volume         | Available |
| cinder      | volumev2       | Available |
| cinder      | volumev3       | Available |
| cloud       | cloudformation | Available |
| glance      | image          | Available |
| heat        | orchestration  | Available |
| keystone    | identity       | Available |
| neutron     | network        | Available |
| nova        | compute        | Available |
| swift       | object-store   | Available |
+-------------+----------------+-----------+



2021-04-29 13:59:55,918 - functest.opnfv_tests.openstack.rally.rally - WARNING - No tenant network created. Trying EXTERNAL_NETWORK as a fallback
2021-04-29 13:59:56,125 - functest.opnfv_tests.openstack.rally.rally - INFO - Starting test scenario "authenticate" ...
2021-04-29 14:22:19,504 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task list --tag authenticate --uuids-only: 646f92ba-234b-483d-8b54-4ff0c46860bc
2021-04-29 14:22:20,804 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task detailed --uuid 646f92ba-234b-483d-8b54-4ff0c46860bc

--------------------------------------------------------------------------------
Task 646f92ba-234b-483d-8b54-4ff0c46860bc: finished
--------------------------------------------------------------------------------

test scenario Authenticate.keystone
args position 0
args values:
{
  "args": {},
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 646f92ba-234b-483d-8b54-4ff0c46860bc has 0 error(s)
--------------------------------------------------------------------------------

+--------------------------------------------------------------------------------------------------------------------------+
|                                                   Response Times (sec)                                                   |
+-----------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+-----------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| authenticate.keystone | 0.781     | 1.117        | 2.007        | 2.229        | 2.452     | 1.301     | 100.0%  | 10    |
| total                 | 0.781     | 1.117        | 2.007        | 2.229        | 2.452     | 1.301     | 100.0%  | 10    |
|  -> duration          | 0.781     | 1.117        | 2.007        | 2.229        | 2.452     | 1.301     | 100.0%  | 10    |
|  -> idle_duration     | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 10    |
+-----------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 3.92205
Full duration: 215.438277
--------------------------------------------------------------------------------

test scenario Authenticate.validate_cinder
args position 0
args values:
{
  "args": {
    "repetitions": 2
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 646f92ba-234b-483d-8b54-4ff0c46860bc has 0 error(s)
--------------------------------------------------------------------------------

+---------------------------------------------------------------------------------------------------------------------------------+
|                                                      Response Times (sec)                                                       |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                       | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| authenticate.validate_cinder | 0.16      | 0.42         | 0.968        | 1.03         | 1.092     | 0.51      | 100.0%  | 10    |
| total                        | 0.646     | 1.311        | 1.984        | 2.075        | 2.166     | 1.322     | 100.0%  | 10    |
|  -> duration                 | 0.646     | 1.311        | 1.984        | 2.075        | 2.166     | 1.322     | 100.0%  | 10    |
|  -> idle_duration            | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 10    |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 3.481599
Full duration: 321.241811
--------------------------------------------------------------------------------

test scenario Authenticate.validate_glance
args position 0
args values:
{
  "args": {
    "repetitions": 2
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 646f92ba-234b-483d-8b54-4ff0c46860bc has 0 error(s)
--------------------------------------------------------------------------------

+-----------------------------------------------------------------------------------------------------------+
|                                           Response Times (sec)                                            |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| total  | n/a       | n/a          | n/a          | n/a          | n/a       | n/a       | n/a     | 0     |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 0.0
Full duration: 50.199313
--------------------------------------------------------------------------------

test scenario Authenticate.validate_heat
args position 0
args values:
{
  "args": {
    "repetitions": 2
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 646f92ba-234b-483d-8b54-4ff0c46860bc has 0 error(s)
--------------------------------------------------------------------------------

+-------------------------------------------------------------------------------------------------------------------------------+
|                                                     Response Times (sec)                                                      |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                     | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| authenticate.validate_heat | 0.166     | 0.282        | 0.521        | 0.527        | 0.534     | 0.315     | 100.0%  | 10    |
| total                      | 0.613     | 0.989        | 1.394        | 1.599        | 1.803     | 1.04      | 100.0%  | 10    |
|  -> duration               | 0.613     | 0.989        | 1.394        | 1.599        | 1.803     | 1.04      | 100.0%  | 10    |
|  -> idle_duration          | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 10    |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 2.831736
Full duration: 211.211516
--------------------------------------------------------------------------------

test scenario Authenticate.validate_neutron
args position 0
args values:
{
  "args": {
    "repetitions": 2
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 646f92ba-234b-483d-8b54-4ff0c46860bc has 0 error(s)
--------------------------------------------------------------------------------

+----------------------------------------------------------------------------------------------------------------------------------+
|                                                       Response Times (sec)                                                       |
+-------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                        | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+-------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| authenticate.validate_neutron | 0.385     | 0.997        | 1.235        | 1.412        | 1.59      | 0.91      | 100.0%  | 10    |
| total                         | 0.803     | 1.813        | 2.217        | 2.219        | 2.221     | 1.728     | 100.0%  | 10    |
|  -> duration                  | 0.803     | 1.813        | 2.217        | 2.219        | 2.221     | 1.728     | 100.0%  | 10    |
|  -> idle_duration             | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 10    |
+-------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 4.908613
Full duration: 227.519012
--------------------------------------------------------------------------------

test scenario Authenticate.validate_nova
args position 0
args values:
{
  "args": {
    "repetitions": 2
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 646f92ba-234b-483d-8b54-4ff0c46860bc has 0 error(s)
--------------------------------------------------------------------------------

+-------------------------------------------------------------------------------------------------------------------------------+
|                                                     Response Times (sec)                                                      |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                     | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| authenticate.validate_nova | 0.162     | 0.23         | 0.638        | 1.011        | 1.383     | 0.387     | 100.0%  | 10    |
| total                      | 0.599     | 1.334        | 2.374        | 2.74         | 3.105     | 1.499     | 100.0%  | 10    |
|  -> duration               | 0.599     | 1.334        | 2.374        | 2.74         | 3.105     | 1.499     | 100.0%  | 10    |
|  -> idle_duration          | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 10    |
+----------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 4.383903
Full duration: 242.250658

HINTS:
* To plot HTML graphics with this data, run:
	rally task report 646f92ba-234b-483d-8b54-4ff0c46860bc --out output.html

* To generate a JUnit report, run:
	rally task export 646f92ba-234b-483d-8b54-4ff0c46860bc --type junit --to output.xml

* To get raw JSON output of task results, run:
	rally task report 646f92ba-234b-483d-8b54-4ff0c46860bc --json --out output.json

At least one workload did not pass SLA criteria.


2021-04-29 14:22:23,787 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task report --json --uuid 646f92ba-234b-483d-8b54-4ff0c46860bc --out /home/opnfv/functest/results/rally_full/authenticate.json

2021-04-29 14:22:23,796 - functest.opnfv_tests.openstack.rally.rally - INFO - Test scenario: "authenticate" Failed.
2021-04-29 14:22:23,828 - functest.opnfv_tests.openstack.rally.rally - WARNING - No tenant network created. Trying EXTERNAL_NETWORK as a fallback
2021-04-29 14:22:24,262 - functest.opnfv_tests.openstack.rally.rally - INFO - Starting test scenario "glance" ...
2021-04-29 15:06:43,654 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task list --tag glance --uuids-only: 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d
2021-04-29 15:06:44,847 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task detailed --uuid 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d

--------------------------------------------------------------------------------
Task 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d: aborted
--------------------------------------------------------------------------------

test scenario GlanceImages.create_and_delete_image
args position 0
args values:
{
  "args": {
    "container_format": "bare",
    "disk_format": "qcow2",
    "image_location": "/home/opnfv/functest/images/cirros-0.4.0-x86_64-disk.img"
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d has 0 error(s)
--------------------------------------------------------------------------------

+---------------------------------------------------------------------------------------------------------------------------------+
|                                                      Response Times (sec)                                                       |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                       | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| glance_v2.create_image       | 19.506    | 28.862       | 35.219       | 35.385       | 35.552    | 29.105    | 100.0%  | 10    |
|  -> glance_v2.get_image (x2) | 0.097     | 0.111        | 0.129        | 0.132        | 0.135     | 0.114     | 100.0%  | 10    |
|  -> glance_v2.upload_data    | 10.075    | 19.488       | 22.493       | 22.669       | 22.845    | 18.378    | 100.0%  | 10    |
| glance_v2.delete_image       | 8.278     | 20.295       | 25.467       | 26.077       | 26.687    | 19.653    | 100.0%  | 10    |
| total                        | 28.798    | 48.366       | 60.686       | 61.463       | 62.239    | 48.758    | 100.0%  | 10    |
|  -> duration                 | 28.798    | 48.366       | 60.686       | 61.463       | 62.239    | 48.758    | 100.0%  | 10    |
|  -> idle_duration            | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 10    |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 137.90488
Full duration: 419.298322
--------------------------------------------------------------------------------

test scenario GlanceImages.create_and_list_image
args position 0
args values:
{
  "args": {
    "container_format": "bare",
    "disk_format": "qcow2",
    "image_location": "/home/opnfv/functest/images/cirros-0.4.0-x86_64-disk.img"
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d has 0 error(s)
--------------------------------------------------------------------------------

+-----------------------------------------------------------------------------------------------------------+
|                                           Response Times (sec)                                            |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| total  | n/a       | n/a          | n/a          | n/a          | n/a       | n/a       | n/a     | 0     |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 0.0
Full duration: 160.421989
--------------------------------------------------------------------------------

test scenario GlanceImages.list_images
args position 0
args values:
{
  "args": {},
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d has 0 error(s)
--------------------------------------------------------------------------------

+--------------------------------------------------------------------------------------------------------------------------+
|                                                   Response Times (sec)                                                   |
+-----------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+-----------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| glance_v2.list_images | 0.167     | 0.194        | 0.629        | 0.972        | 1.314     | 0.352     | 100.0%  | 10    |
| total                 | 0.167     | 0.194        | 0.629        | 0.972        | 1.314     | 0.353     | 100.0%  | 10    |
|  -> duration          | 0.167     | 0.194        | 0.629        | 0.972        | 1.314     | 0.353     | 100.0%  | 10    |
|  -> idle_duration     | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 100.0%  | 10    |
+-----------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 4.861771
Full duration: 211.993183
--------------------------------------------------------------------------------

test scenario GlanceImages.create_image_and_boot_instances
args position 0
args values:
{
  "args": {
    "container_format": "bare",
    "disk_format": "qcow2",
    "image_location": "/home/opnfv/functest/images/cirros-0.4.0-x86_64-disk.img",
    "flavor": {
      "name": "rally_full-flavor_f304654f-f1b7-402f-b023-816517b42bc7"
    },
    "number_instances": 2,
    "boot_server_kwargs": {
      "nics": [
        {
          "net-id": "bc94259e-dafc-43a4-bfb4-7cbcbff98a2b"
        }
      ]
    }
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    },
    "quotas": {
      "nova": {
        "cores": -1,
        "floating_ips": -1,
        "instances": -1,
        "ram": -1,
        "security_group_rules": -1,
        "security_groups": -1
      }
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d has 5 error(s)
--------------------------------------------------------------------------------

TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_WEsKS38J_1:60b2249b-d702-44f0-9eca-1b3212d87f75 to become ('ACTIVE') current status BUILD

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/glance/images.py", line 250, in run
    **boot_server_kwargs)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in _boot_servers
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in <listcomp>
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally/task/utils.py", line 252, in wait_for_status
    timeout=timeout)
rally.exceptions.TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_WEsKS38J_1:60b2249b-d702-44f0-9eca-1b3212d87f75 to become ('ACTIVE') current status BUILD

--------------------------------------------------------------------------------
TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_QglK9k0n_1:cd0e53bf-90ae-421b-b4df-7d1406bb8b64 to become ('ACTIVE') current status BUILD

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/glance/images.py", line 250, in run
    **boot_server_kwargs)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in _boot_servers
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in <listcomp>
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally/task/utils.py", line 252, in wait_for_status
    timeout=timeout)
rally.exceptions.TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_QglK9k0n_1:cd0e53bf-90ae-421b-b4df-7d1406bb8b64 to become ('ACTIVE') current status BUILD

--------------------------------------------------------------------------------
TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_r3Eo2NLE_1:ba6b3fa0-df6f-4e24-aa11-e12b7105a8f3 to become ('ACTIVE') current status BUILD

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/glance/images.py", line 250, in run
    **boot_server_kwargs)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in _boot_servers
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in <listcomp>
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally/task/utils.py", line 252, in wait_for_status
    timeout=timeout)
rally.exceptions.TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_r3Eo2NLE_1:ba6b3fa0-df6f-4e24-aa11-e12b7105a8f3 to become ('ACTIVE') current status BUILD

--------------------------------------------------------------------------------
TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_KCpajm4v_1:3d6fa598-8ed5-48f8-9b4a-62900a083ffc to become ('ACTIVE') current status BUILD

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/glance/images.py", line 250, in run
    **boot_server_kwargs)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in _boot_servers
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in <listcomp>
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally/task/utils.py", line 252, in wait_for_status
    timeout=timeout)
rally.exceptions.TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_KCpajm4v_1:3d6fa598-8ed5-48f8-9b4a-62900a083ffc to become ('ACTIVE') current status BUILD

--------------------------------------------------------------------------------
TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_tCxUyc23_1:2b07857e-9e2c-40bf-aab7-00f2837d47ba to become ('ACTIVE') current status BUILD

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/glance/images.py", line 250, in run
    **boot_server_kwargs)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in _boot_servers
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 587, in <listcomp>
    ) for server in servers]
  File "/usr/lib/python3.6/site-packages/rally/task/utils.py", line 252, in wait_for_status
    timeout=timeout)
rally.exceptions.TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_a7cf835c_tCxUyc23_1:2b07857e-9e2c-40bf-aab7-00f2837d47ba to become ('ACTIVE') current status BUILD

--------------------------------------------------------------------------------
+---------------------------------------------------------------------------------------------------------------------------------+
|                                                      Response Times (sec)                                                       |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                       | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| glance_v2.create_image       | 40.236    | 41.461       | 49.919       | 52.558       | 55.197    | 43.884    | 100.0%  | 5     |
|  -> glance_v2.get_image (x2) | 0.103     | 0.11         | 0.124        | 0.126        | 0.128     | 0.113     | 100.0%  | 5     |
|  -> glance_v2.upload_data    | 23.721    | 25.055       | 28.235       | 29.156       | 30.076    | 25.644    | 100.0%  | 5     |
| nova.boot_servers            | 324.973   | 329.686      | 339.094      | 342.058      | 345.022   | 331.385   | 0.0%    | 5     |
| total                        | 365.495   | 371.147      | 389.014      | 394.616      | 400.219   | 375.268   | 0.0%    | 5     |
|  -> duration                 | 364.495   | 370.147      | 388.014      | 393.616      | 399.219   | 374.268   | 0.0%    | 5     |
|  -> idle_duration            | 1.0       | 1.0          | 1.0          | 1.0          | 1.0       | 1.0       | 0.0%    | 5     |
+------------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 766.069717
Full duration: 1776.721975

HINTS:
* To plot HTML graphics with this data, run:
	rally task report 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d --out output.html

* To generate a JUnit report, run:
	rally task export 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d --type junit --to output.xml

* To get raw JSON output of task results, run:
	rally task report 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d --json --out output.json

At least one workload did not pass SLA criteria.


2021-04-29 15:06:47,927 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task report --json --uuid 4b6a76ad-58b9-4e60-b294-9ca6b0dc3a0d --out /home/opnfv/functest/results/rally_full/glance.json

2021-04-29 15:06:47,938 - functest.opnfv_tests.openstack.rally.rally - INFO - Test scenario: "glance" Failed.
2021-04-29 15:06:47,984 - functest.opnfv_tests.openstack.rally.rally - WARNING - No tenant network created. Trying EXTERNAL_NETWORK as a fallback
2021-04-29 15:06:48,479 - functest.opnfv_tests.openstack.rally.rally - INFO - Starting test scenario "cinder" ...
2021-04-29 15:30:56,196 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task list --tag cinder --uuids-only: 0abb981b-0dc5-4055-94cb-a0facd865f16
2021-04-29 15:30:57,404 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task detailed --uuid 0abb981b-0dc5-4055-94cb-a0facd865f16

--------------------------------------------------------------------------------
Task 0abb981b-0dc5-4055-94cb-a0facd865f16: aborted
--------------------------------------------------------------------------------

test scenario CinderVolumes.create_and_attach_volume
args position 0
args values:
{
  "args": {
    "flavor": {
      "name": "rally_full-flavor_f304654f-f1b7-402f-b023-816517b42bc7"
    },
    "image": {
      "name": "rally_full-img_f304654f-f1b7-402f-b023-816517b42bc7"
    },
    "size": 1,
    "nics": [
      {
        "net-id": "bc94259e-dafc-43a4-bfb4-7cbcbff98a2b"
      }
    ]
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    },
    "quotas": {
      "cinder": {
        "gigabytes": -1,
        "snapshots": -1,
        "volumes": -1
      }
    },
    "api_versions": {
      "cinder": {
        "version": 3,
        "service_type": "volumev3"
      }
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 0abb981b-0dc5-4055-94cb-a0facd865f16 has 3 error(s)
--------------------------------------------------------------------------------

TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_b8bfb3fb_6o8hIZXY:3532b792-b53a-408f-96ce-a9a672f85f79 to become ('ACTIVE') current status BUILD

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally/common/logging.py", line 329, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/cinder/volumes.py", line 458, in run
    server = self._boot_server(image, flavor, **create_vm_params)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 88, in _boot_server
    check_interval=CONF.openstack.nova_server_boot_poll_interval
  File "/usr/lib/python3.6/site-packages/rally/task/utils.py", line 252, in wait_for_status
    timeout=timeout)
rally.exceptions.TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_b8bfb3fb_6o8hIZXY:3532b792-b53a-408f-96ce-a9a672f85f79 to become ('ACTIVE') current status BUILD

--------------------------------------------------------------------------------
TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_b8bfb3fb_nuofxbrK:57be8a05-05ef-40d7-b93f-027765839fe7 to become ('ACTIVE') current status BUILD

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally/common/logging.py", line 329, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/cinder/volumes.py", line 458, in run
    server = self._boot_server(image, flavor, **create_vm_params)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 88, in _boot_server
    check_interval=CONF.openstack.nova_server_boot_poll_interval
  File "/usr/lib/python3.6/site-packages/rally/task/utils.py", line 252, in wait_for_status
    timeout=timeout)
rally.exceptions.TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_b8bfb3fb_nuofxbrK:57be8a05-05ef-40d7-b93f-027765839fe7 to become ('ACTIVE') current status BUILD

--------------------------------------------------------------------------------
TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_b8bfb3fb_al2YfvJZ:a5e407e7-ccb7-47a7-9f51-24f277d07d40 to become ('ACTIVE') current status BUILD

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally/common/logging.py", line 329, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/cinder/volumes.py", line 458, in run
    server = self._boot_server(image, flavor, **create_vm_params)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/nova/utils.py", line 88, in _boot_server
    check_interval=CONF.openstack.nova_server_boot_poll_interval
  File "/usr/lib/python3.6/site-packages/rally/task/utils.py", line 252, in wait_for_status
    timeout=timeout)
rally.exceptions.TimeoutException: Rally tired waiting 300.00 seconds for Server s_rally_b8bfb3fb_al2YfvJZ:a5e407e7-ccb7-47a7-9f51-24f277d07d40 to become ('ACTIVE') current status BUILD

--------------------------------------------------------------------------------
+----------------------------------------------------------------------------------------------------------------------------+
|                                                    Response Times (sec)                                                    |
+-------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                  | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+-------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| nova.boot_server        | 257.096   | 310.199      | 310.986      | 311.053      | 311.119   | 291.231   | 40.0%   | 5     |
| cinder_v3.create_volume | 115.324   | 121.357      | 126.183      | 126.786      | 127.39    | 121.357   | 100.0%  | 2     |
| nova.attach_volume      | 127.452   | 142.383      | 154.327      | 155.82       | 157.313   | 142.383   | 100.0%  | 2     |
| nova.detach_volume      | 63.537    | 69.611       | 74.469       | 75.076       | 75.684    | 69.611    | 100.0%  | 2     |
| cinder_v3.delete_volume | 100.599   | 124.176      | 143.037      | 145.395      | 147.752   | 124.176   | 100.0%  | 2     |
| nova.delete_server      | 57.072    | 60.748       | 63.689       | 64.057       | 64.424    | 60.748    | 100.0%  | 2     |
| total                   | 310.2     | 311.119      | 782.711      | 787.536      | 792.361   | 498.541   | 40.0%   | 5     |
|  -> duration            | 309.2     | 310.119      | 781.711      | 786.536      | 791.361   | 497.541   | 40.0%   | 5     |
|  -> idle_duration       | 1.0       | 1.0          | 1.0          | 1.0          | 1.0       | 1.0       | 40.0%   | 5     |
+-------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 791.361036
Full duration: 1361.721297

HINTS:
* To plot HTML graphics with this data, run:
	rally task report 0abb981b-0dc5-4055-94cb-a0facd865f16 --out output.html

* To generate a JUnit report, run:
	rally task export 0abb981b-0dc5-4055-94cb-a0facd865f16 --type junit --to output.xml

* To get raw JSON output of task results, run:
	rally task report 0abb981b-0dc5-4055-94cb-a0facd865f16 --json --out output.json

At least one workload did not pass SLA criteria.


2021-04-29 15:31:00,342 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task report --json --uuid 0abb981b-0dc5-4055-94cb-a0facd865f16 --out /home/opnfv/functest/results/rally_full/cinder.json

2021-04-29 15:31:00,347 - functest.opnfv_tests.openstack.rally.rally - INFO - Test scenario: "cinder" Failed.
2021-04-29 15:31:00,382 - functest.opnfv_tests.openstack.rally.rally - WARNING - No tenant network created. Trying EXTERNAL_NETWORK as a fallback
2021-04-29 15:31:00,810 - functest.opnfv_tests.openstack.rally.rally - INFO - Starting test scenario "heat" ...
2021-04-29 15:57:42,207 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task list --tag heat --uuids-only: 9a9b4606-9f2a-45b6-b29d-fe107594440a
2021-04-29 15:57:43,432 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task detailed --uuid 9a9b4606-9f2a-45b6-b29d-fe107594440a

--------------------------------------------------------------------------------
Task 9a9b4606-9f2a-45b6-b29d-fe107594440a: aborted
--------------------------------------------------------------------------------

test scenario HeatStacks.create_and_list_stack
args position 0
args values:
{
  "args": {
    "template_path": "/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/rally/scenario/templates/default.yaml.template"
  },
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    },
    "roles": [
      "heat_stack_owner"
    ]
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 9a9b4606-9f2a-45b6-b29d-fe107594440a has 6 error(s)
--------------------------------------------------------------------------------

HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/stacks.py", line 50, in run
    files, environment)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/utils.py", line 65, in _create_stack
    stack_id = self.clients("heat").stacks.create(**kw)["stack"]["id"]
  File "/usr/lib/python3.6/site-packages/heatclient/v1/stacks.py", line 171, in create
    data=kwargs, headers=headers)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 381, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 323, in request
    raise exc.from_response(resp)
heatclient.exc.HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

--------------------------------------------------------------------------------
HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/stacks.py", line 50, in run
    files, environment)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/utils.py", line 65, in _create_stack
    stack_id = self.clients("heat").stacks.create(**kw)["stack"]["id"]
  File "/usr/lib/python3.6/site-packages/heatclient/v1/stacks.py", line 171, in create
    data=kwargs, headers=headers)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 381, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 323, in request
    raise exc.from_response(resp)
heatclient.exc.HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

--------------------------------------------------------------------------------
HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/stacks.py", line 50, in run
    files, environment)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/utils.py", line 65, in _create_stack
    stack_id = self.clients("heat").stacks.create(**kw)["stack"]["id"]
  File "/usr/lib/python3.6/site-packages/heatclient/v1/stacks.py", line 171, in create
    data=kwargs, headers=headers)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 381, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 323, in request
    raise exc.from_response(resp)
heatclient.exc.HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

--------------------------------------------------------------------------------
HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/stacks.py", line 50, in run
    files, environment)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/utils.py", line 65, in _create_stack
    stack_id = self.clients("heat").stacks.create(**kw)["stack"]["id"]
  File "/usr/lib/python3.6/site-packages/heatclient/v1/stacks.py", line 171, in create
    data=kwargs, headers=headers)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 381, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 323, in request
    raise exc.from_response(resp)
heatclient.exc.HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

--------------------------------------------------------------------------------
HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/stacks.py", line 50, in run
    files, environment)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/utils.py", line 65, in _create_stack
    stack_id = self.clients("heat").stacks.create(**kw)["stack"]["id"]
  File "/usr/lib/python3.6/site-packages/heatclient/v1/stacks.py", line 171, in create
    data=kwargs, headers=headers)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 381, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 323, in request
    raise exc.from_response(resp)
heatclient.exc.HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

--------------------------------------------------------------------------------
HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/stacks.py", line 50, in run
    files, environment)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/heat/utils.py", line 65, in _create_stack
    stack_id = self.clients("heat").stacks.create(**kw)["stack"]["id"]
  File "/usr/lib/python3.6/site-packages/heatclient/v1/stacks.py", line 171, in create
    data=kwargs, headers=headers)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 381, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3.6/site-packages/heatclient/common/http.py", line 323, in request
    raise exc.from_response(resp)
heatclient.exc.HTTPException: ERROR: b'<html>\r\n<head><title>504 Gateway Time-out</title></head>\r\n<body>\r\n<center><h1>504 Gateway Time-out</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

--------------------------------------------------------------------------------
+----------------------------------------------------------------------------------------------------------------------+
|                                                 Response Times (sec)                                                 |
+-------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action            | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+-------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| heat.create_stack | 60.65     | 60.967       | 64.378       | 65.653       | 66.929    | 62.0      | 0.0%    | 6     |
| total             | 60.65     | 60.967       | 64.378       | 65.654       | 66.929    | 62.0      | 0.0%    | 6     |
|  -> duration      | 60.65     | 60.967       | 64.378       | 65.654       | 66.929    | 62.0      | 0.0%    | 6     |
|  -> idle_duration | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 0.0%    | 6     |
+-------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 127.638959
Full duration: 1313.248278

HINTS:
* To plot HTML graphics with this data, run:
	rally task report 9a9b4606-9f2a-45b6-b29d-fe107594440a --out output.html

* To generate a JUnit report, run:
	rally task export 9a9b4606-9f2a-45b6-b29d-fe107594440a --type junit --to output.xml

* To get raw JSON output of task results, run:
	rally task report 9a9b4606-9f2a-45b6-b29d-fe107594440a --json --out output.json

At least one workload did not pass SLA criteria.


2021-04-29 15:57:46,398 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task report --json --uuid 9a9b4606-9f2a-45b6-b29d-fe107594440a --out /home/opnfv/functest/results/rally_full/heat.json

2021-04-29 15:57:46,404 - functest.opnfv_tests.openstack.rally.rally - INFO - Test scenario: "heat" Failed.
2021-04-29 15:57:46,440 - functest.opnfv_tests.openstack.rally.rally - WARNING - No tenant network created. Trying EXTERNAL_NETWORK as a fallback
2021-04-29 15:57:46,814 - functest.opnfv_tests.openstack.rally.rally - INFO - Starting test scenario "keystone" ...
2021-04-29 16:14:27,663 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task list --tag keystone --uuids-only: 3fcaca2f-d79b-4c22-b918-018bdeb74c78
2021-04-29 16:14:28,888 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task detailed --uuid 3fcaca2f-d79b-4c22-b918-018bdeb74c78

--------------------------------------------------------------------------------
Task 3fcaca2f-d79b-4c22-b918-018bdeb74c78: aborted
--------------------------------------------------------------------------------

test scenario KeystoneBasic.add_and_remove_user_role
args position 0
args values:
{
  "args": {},
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 3fcaca2f-d79b-4c22-b918-018bdeb74c78 has 0 error(s)
--------------------------------------------------------------------------------

+-----------------------------------------------------------------------------------------------------------+
|                                           Response Times (sec)                                            |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| total  | n/a       | n/a          | n/a          | n/a          | n/a       | n/a       | n/a     | 0     |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 0.0
Full duration: 167.407807
--------------------------------------------------------------------------------

test scenario KeystoneBasic.create_add_and_list_user_roles
args position 0
args values:
{
  "args": {},
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 3fcaca2f-d79b-4c22-b918-018bdeb74c78 has 0 error(s)
--------------------------------------------------------------------------------

+-----------------------------------------------------------------------------------------------------------+
|                                           Response Times (sec)                                            |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| total  | n/a       | n/a          | n/a          | n/a          | n/a       | n/a       | n/a     | 0     |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 0.0
Full duration: 163.613436
--------------------------------------------------------------------------------

test scenario KeystoneBasic.create_and_list_tenants
args position 0
args values:
{
  "args": {},
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 3fcaca2f-d79b-4c22-b918-018bdeb74c78 has 0 error(s)
--------------------------------------------------------------------------------

+-----------------------------------------------------------------------------------------------------------+
|                                           Response Times (sec)                                            |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| total  | n/a       | n/a          | n/a          | n/a          | n/a       | n/a       | n/a     | 0     |
+--------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 0.0
Full duration: 83.83953
--------------------------------------------------------------------------------

test scenario KeystoneBasic.create_and_delete_role
args position 0
args values:
{
  "args": {},
  "runner": {
    "concurrency": 4,
    "times": 10
  },
  "contexts": {
    "users": {
      "tenants": 3,
      "users_per_tenant": 2
    }
  },
  "sla": {
    "failure_rate": {
      "max": 0
    }
  },
  "hooks": []
}

--------------------------------------------------------------------------------
Task 3fcaca2f-d79b-4c22-b918-018bdeb74c78 has 1 error(s)
--------------------------------------------------------------------------------

BadGateway: Bad Gateway (HTTP 502)

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/rally/task/runner.py", line 72, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/scenarios/keystone/basic.py", line 210, in run
    role = self.admin_keystone.create_role()
  File "/usr/lib/python3.6/site-packages/rally/task/service.py", line 116, in wrapper
    return func(instance, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/services/identity/identity.py", line 162, in create_role
    return self._impl.create_role(name=name, domain_name=domain_name)
  File "/usr/lib/python3.6/site-packages/rally/task/service.py", line 116, in wrapper
    return func(instance, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/services/identity/keystone_v3.py", line 315, in create_role
    name, domain_name=domain_name))
  File "/usr/lib/python3.6/site-packages/rally/task/service.py", line 116, in wrapper
    return func(instance, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/rally_openstack/services/identity/keystone_v3.py", line 176, in create_role
    return self._clients.keystone("3").roles.create(name, domain=domain_id)
  File "/usr/lib/python3.6/site-packages/keystoneclient/v3/roles.py", line 144, in create
    **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
    return f(*args, **new_kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 386, in create
    self.key)
  File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 192, in _post
    resp, body = self.client.post(url, body=body, **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 381, in post
    return self.request(url, 'POST', **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 534, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 237, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 890, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.BadGateway: Bad Gateway (HTTP 502)

--------------------------------------------------------------------------------
+----------------------------------------------------------------------------------------------------------------------------+
|                                                    Response Times (sec)                                                    |
+-------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| Action                  | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count |
+-------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+
| keystone_v3.create_role | 9.44      | 24.516       | 44.695       | 46.997       | 49.3      | 28.5      | 80.0%   | 5     |
| keystone_v3.delete_role | 10.305    | 20.055       | 32.625       | 33.31        | 33.995    | 21.103    | 100.0%  | 4     |
| total                   | 20.123    | 49.3         | 64.647       | 68.215       | 71.783    | 45.382    | 80.0%   | 5     |
|  -> duration            | 20.123    | 49.3         | 64.647       | 68.215       | 71.783    | 45.382    | 80.0%   | 5     |
|  -> idle_duration       | 0.0       | 0.0          | 0.0          | 0.0          | 0.0       | 0.0       | 80.0%   | 5     |
+-------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+

Load duration: 102.962359
Full duration: 421.039414

HINTS:
* To plot HTML graphics with this data, run:
	rally task report 3fcaca2f-d79b-4c22-b918-018bdeb74c78 --out output.html

* To generate a JUnit report, run:
	rally task export 3fcaca2f-d79b-4c22-b918-018bdeb74c78 --type junit --to output.xml

* To get raw JSON output of task results, run:
	rally task report 3fcaca2f-d79b-4c22-b918-018bdeb74c78 --json --out output.json

At least one workload did not pass SLA criteria.


2021-04-29 16:14:31,999 - functest.opnfv_tests.openstack.rally.rally - INFO - rally task report --json --uuid 3fcaca2f-d79b-4c22-b918-018bdeb74c78 --out /home/opnfv/functest/results/rally_full/keystone.json

2021-04-29 16:14:32,005 - functest.opnfv_tests.openstack.rally.rally - INFO - Test scenario: "keystone" Failed.
2021-04-29 16:14:32,047 - functest.opnfv_tests.openstack.rally.rally - WARNING - No tenant network created. Trying EXTERNAL_NETWORK as a fallback
2021-04-29 16:14:32,485 - functest.opnfv_tests.openstack.rally.rally - INFO - Starting test scenario "neutron" ...
2021-04-29 18:14:32,512 - functest.opnfv_tests.openstack.rally.rally - ERROR - Error with run:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/rally/rally.py", line 697, in run
    self.run_tests(**kwargs)
  File "/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/rally/rally.py", line 547, in run_tests
    self.run_task(test)
  File "/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/rally/rally.py", line 438, in run_task
    stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
  File "/usr/lib/python3.6/subprocess.py", line 289, in call
    return p.wait(timeout=timeout)
  File "/usr/lib/python3.6/subprocess.py", line 1469, in wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['rally', 'task', 'start', '--tag', 'neutron', '--abort-on-sla-failure', '--task', '/home/opnfv/functest/data/rally/task/task.yaml', '--task-args', "{'service_list': ['neutron'], 'image_name': 'rally_full-img_f304654f-f1b7-402f-b023-816517b42bc7', 'flavor_name': 'rally_full-flavor_f304654f-f1b7-402f-b023-816517b42bc7', 'flavor_alt_name': 'rally_full-flavor_alt_f304654f-f1b7-402f-b023-816517b42bc7', 'glance_image_location': '/home/opnfv/functest/images/cirros-0.4.0-x86_64-disk.img', 'glance_image_format': 'qcow2', 'tmpl_dir': '/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/rally/scenario/templates', 'sup_dir': '/usr/lib/python3.6/site-packages/functest/opnfv_tests/openstack/rally/scenario/support', 'users_amount': 2, 'tenants_amount': 3, 'use_existing_users': False, 'iterations': 10, 'concurrency': 4, 'smoke': False, 'volume_version': 3, 'volume_service_type': 'volumev3', 'block_migration': 'false', 'floating_network': '', 'netid': 'bc94259e-dafc-43a4-bfb4-7cbcbff98a2b'}"]' timed out after 7200 seconds
2021-04-29 18:14:32,517 - xtesting.ci.run_tests - INFO - Test result:

+--------------------+------------------+------------------+----------------+
|     TEST CASE      |     PROJECT      |     DURATION     |     RESULT     |
+--------------------+------------------+------------------+----------------+
|     rally_full     |     functest     |      256:01      |      FAIL      |
+--------------------+------------------+------------------+----------------+