/usr/lib/python3.6/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.168.122.143: b'e82de6bfd441777e44005ab5ec367ba6' key.get_fingerprint()))) 2018-03-09 07:58:04,077 main.py 160 INFO doctor test starting....... 2018-03-09 07:58:04,077 apex.py 38 INFO Setup Apex installer start...... 2018-03-09 07:58:04,077 apex.py 52 INFO Get SSH keys from Apex installer...... 2018-03-09 07:58:04,114 apex.py 69 INFO Get controller ips from Apex installer...... 2018-03-09 07:58:08,239 apex.py 80 INFO Get controller_ips:['192.30.9.9'] from Apex installer 2018-03-09 07:58:08,826 apex.py 118 INFO Set apply patches start...... /usr/lib/python3.6/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.30.9.9: b'451219b29de78198f5ff7b4a972e3d77' key.get_fingerprint()))) 2018-03-09 07:58:09,335 apex.py 99 INFO Setup ssh stunnel in controller nodesin Apex installer...... 2018-03-09 07:58:09,341 image.py 48 INFO image create start...... 2018-03-09 07:58:09,899 image.py 68 INFO image create end...... 2018-03-09 07:58:09,899 user.py 69 INFO user create start...... 2018-03-09 07:58:09,984 user.py 93 INFO project doctor already created...... 2018-03-09 07:58:09,984 user.py 95 INFO test project 2018-03-09 07:58:10,021 user.py 111 INFO user doctor already created...... 2018-03-09 07:58:10,021 user.py 113 INFO test user 2018-03-09 07:58:10,041 user.py 127 INFO role _member_ already created...... 2018-03-09 07:58:10,041 user.py 128 INFO test role 2018-03-09 07:58:10,133 user.py 149 INFO Already grant a role:_member_ to user: doctor on project: doctor 2018-03-09 07:58:10,231 user.py 149 INFO Already grant a role:admin to user: admin on project: doctor 2018-03-09 07:58:10,232 user.py 77 INFO user create end...... 2018-03-09 07:58:10,232 main.py 96 INFO doctor fault management test starting....... 2018-03-09 07:58:10,232 user.py 181 INFO user quota update start...... 2018-03-09 07:58:10,873 user.py 197 INFO user quota update end...... 2018-03-09 07:58:10,874 network.py 41 INFO network create start....... 2018-03-09 07:58:11,999 network.py 47 INFO network create end....... 2018-03-09 07:58:11,999 network.py 49 INFO subnet create start....... 2018-03-09 07:58:12,057 network.py 58 INFO subn2018-03-09 07:58:17,230 instance.py 110 INFO wait for vm launch end...... 2018-03-09 07:58:17,230 alarm.py 45 INFO alarm create start...... 2018-03-09 07:58:18,737 alarm.py 81 INFO alarm create end...... 2018-03-09 07:58:18,737 sample.py 72 INFO sample inspector start...... 2018-03-09 07:58:19,749 sample.py 26 INFO sample consumer start...... * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit) * Running on http://0.0.0.0:12346/ (Press CTRL+C to quit) 2018-03-09 07:58:20,751 apex.py 84 INFO Get host ip from host name in Apex installer...... 2018-03-09 07:58:24,615 apex.py 95 INFO Get host_ip:['192.30.9.3'] from host_name:overcloud-novacompute-1.opnfvlf.org in Apex installer 2018-03-09 07:58:24,615 main.py 193 INFO Get host info(name:overcloud-novacompute-1.opnfvlf.org, ip:192.2018-03-09 07:58:25,388 apex.py 95 INFO Get host_ip:['192.30.9.3'] from host_name:overcloud-novacompute-1.opnfvlf.org in Apex installer 2018-03-09 07:58:25,388 main.py 193 INFO Get host info(name:overcloud-novacompute-1.opnfvlf.org, ip:2018-03-09 08:00:24,675 network_failure.py 39 INFO fault inject start...... 2018-03-09 08:00:24,677 apex.py 52 INFO Get SSH keys from Apex installer...... 2018-03-09 08:00:24,677 apex.py 55 INFO Already have SSH keys from Apex installer..2018-03-09 08:00:25,391 network_failure.py 39 INFO fault inject start...... 2018-03-09 08:00:25,391 apex.py 52 INFO Get SSH keys from Apex installer...... 2018-03-09 08:00:25,392 apex.py 55 INFO Already have SSH keys from Apex installer...... /usr/lib/python3.6/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.30.9.3: b'4689d992c1140fcec34570ac72e08397' key.get_fingerprint()))) 2018-03-09 08:00:25,456 utils.py 91 INFO Copy /src/doctor-tests/doctor_tests/disable_network.sh -> disable_network.sh 2018-03-09 08:00:25,613 utils.py 72 INFO Executing: bash disab2018-03-09 08:00:26,096 sample.py 97 INFO doctor monitor detected at 1520582426.0965865 2018-03-09 08:00:26,097 sample.py 41 INFO sample monitor report error...... 2018-03-09 08:00:26,104 sample.py 163 INFO event posted in sample inspecto2018-03-09 08:00:26,171 sample.py 97 INFO doctor monitor detected at 1520582426.1718097 2018-03-09 08:00:26,172 sample.py 41 INFO sample monitor report error...... 2018-03-09 08:00:26,180 sample.py 163 INFO event posted in sample inspector at 1520582426.1801836 2018-03-09 08:00:26,180 sample.py 164 INFO sample inspector = 2018-03-09 08:00:26,180 sample.py 166 INFO sample inspector received data = b'[{"time": "2018-03-09T08:00:26.172365", "type": "compute.host.down", "details": {"hostname": "overcloud-nova2018-03-09 08:00:26,261 sample.py 121 INFO doctor mark vm() error at 1520582426.261654 127.0.0.1 - - [09/Mar/2018 08:00:26] "PUT /events HTTP/1.1" 200 - 2018-03-09 08:00:26,264 sample.py 99 INFO ping timeout, quit monito2018-03-2018-03-09 08:00:35,029 main.py 133 ERROR doctor fault management test failed, Exception=Can not match notified time 2018-03-09 08:00:37,123 sample.py 78 INFO sample inspector stop...... 2018-03-09 08:00:37,167 sample.py 173 INFO shutdown inspector app server at 1520582437.1670215 127.0.0.1 - - [09/Mar/2018 08:00:37] "POST /event2018-03-09 08:00:37,788 sample.py 78 INFO sample inspector stop...... 2018-03-09 08:00:37,823 sample.py 173 INFO shutdown inspector app server at 1520582437.8236682 127.0.0.1 - - [09/Mar/2018 08:00:37] "POST /events/shutdown HTTP/1.1" 200 - 2018-03-09 08:00:37,826 sample.py 35 INFO sample monitor stop...... 2018-03-09 08:00:37,827 sample.py 106 INFO Stopping Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.3) 2018-03-09 08:00:37,827 sample.py 31 INFO sample consumer stop...... 2018-03-09 08:00:37,832 sample.py 64 INFO shutdown consumer app server at 1520582437.8323739 127.0.0.1 - - [09/Mar/2018 08:00:37] "POST /shutdown HTTP/1.1"2018-03-09 08:01:39,903 instance.py 89 INFO instance delete end....... 2018-03-09 08:01:39,903 network.py 61 INFO subnet delete start....... 2018-03-09 08:01:41,043 main.py 175 ERROR doctor test failed, Exception=Subnet ed31cecc-3a1c-413c-97c7-d24ed525f146 could not be found. Neutron server returns request_ids: ['req-3757757c-9f47-4d84-80b1-15414217b24f'] 2018-03-09 08:01:41,044 apex.py 127 INFO restore apply patches start...... 2018-03-09 08:01:41,647 image.py 71 INFO image delete start....... 2018-03-09 08:01:42,996 image.py 76 INFO image delete end....... 2018-03-09 08:01:42,996 user.py 156 INFO user delete start...... Traceback (most recent call last): File "/src/doctor-tests/doctor_tests/main.py", line 117, in test_fault_management notification_time = calculate_notification_time(LogFile) File "/src/doctor-tests/doctor_tests/scenario/common.py", line 23, in calculate_notification_time raise Exception('Can not match notified time') Exception: Can not match notified time During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/src/doctor-tests/doctor_tests/main.py", line 134, in test_fault_management sys.exit(1) SystemExit: 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/src/doctor-tests/doctor_tests/main.py", line 169, in run getattr(self, function)() File "/src/doctor-tests/doctor_tests/main.py", line 136, in test_fault_management self.cleanup_fault_management() File "/src/doctor-tests/doctor_tests/main.py", line 253, in cleanup_fault_management self.network.delete() File "/src/doctor-tests/doctor_tests/network.py", line 63, in delete self.neutron.delete_subnet(self.subnet['id']) File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 832, in delete_subnet return self.delete(self.subnet_path % (subnet)) File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 348, in delete headers=headers, params=params) File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 329, in retry_request headers=headers, params=params) File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 292, in do_request self._handle_fault_response(status_code, replybody, resp) File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 268, in _handle_fault_response exception_handler_v20(status_code, error_body) File "/usr/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20 request_ids=request_ids) neutronclient.common.exceptions.NotFound: Subnet ed31cecc-3a1c-413c-97c7-d24ed525f146 could not be found. Neutron server returns request_ids: ['req-3757757c-9f47-4d84-80b1-15414217b24f'] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/src/doctor-tests/doctor_tests/main.py", line 176, in run sys.exit(1) SystemExit: 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/doctor-test", line 10, in sys.exit(main()) File "/src/doctor-tests/doctor_tests/main.py", line 276, in main doctor.run() File "/src/doctor-tests/doctor_tests/main.py", line 178, in run self.cleanup() File "/src/doctor-tests/doctor_tests/main.py", line 259, in cleanup self.user.delete() File "/src/doctor-tests/doctor_tests/user.py", line 167, in delete project=project) File "/usr/lib/python3.6/site-packages/positional/__init__.py", line 108, in inner return wrapped(*args, **kwargs) File "/usr/lib/python3.6/site-packages/keystoneclient/v3/roles.py", line 358, in revoke **kwargs) File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 75, in func return f(*args, **new_kwargs) File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 419, in delete self.build_url(dict_args_in_out=kwargs)) File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 219, in _delete return self.client.delete(url, **kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 303, in delete return self.request(url, 'DELETE', **kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 447, in request resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 192, in request return self.session.request(url, method, **kwargs) File "/usr/lib/python3.6/site-packages/positional/__init__.py", line 108, in inner return wrapped(*args, **kwargs) File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 742, in request raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.NotFound: Could not find project: ce586134bc2b49c0b84a1a1cdb845d19. (HTTP 404) (Request-ID: req-c6b93987-dac8-4ce3-b2f3-c5ebc55a4b73) 2018-03-09 08:01:43,278 - xtesting.core.feature - ERROR - Execute command: doctor-test failed 2018-03-09 08:01:43,279 - xtesting.core.feature - INFO - Test result is stored in '/var/lib/xtesting/results/doctor-notification.log'