/usr/lib/python3.6/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.168.122.66: b'9f1e016fb1cce7bd681441baa970e29c'
  key.get_fingerprint())))
2018-06-05 00:43:12,808 main.py 106 INFO   doctor test starting.......
2018-06-05 00:43:12,808 apex.py 28 INFO   Setup Apex installer start......
2018-06-05 00:43:12,808 base.py 80 INFO   Get SSH keys from apex installer......
2018-06-05 00:43:13,198 apex.py 46 INFO   Get controller ips from Apex installer......
2018-06-05 00:43:13,198 base.py 98 INFO   Run command=source stackrc; nova list | grep ' overcloud-controller-[0-9] ' | sed -e 's/^.*ctlplane=//' |awk '{print $1}' in apex installer......
2018-06-05 00:43:16,949 base.py 107 INFO   Output=['192.30.9.3'] command=source stackrc; nova list | grep ' overcloud-controller-[0-9] ' | sed -e 's/^.*ctlplane=//' |awk '{print $1}' in apex installer
2018-06-05 00:43:16,950 apex.py 53 INFO   Get controller_ips:['192.30.9.3'] from Apex installer
2018-06-05 00:43:17,630 apex.py 66 INFO   Set apply patches start......
/usr/lib/python3.6/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.30.9.3: b'0d0b21d7d307d3b2d82d8c69ff4b162c'
  key.get_fingerprint())))
2018-06-05 00:43:22,137 base.py 60 INFO   Setup ssh stunnel in apex installer......
2018-06-05 00:43:22,144 image.py 48 INFO   image create start......
2018-06-05 00:43:27,509 image.py 68 INFO   image create end......
2018-06-05 00:43:27,509 user.py 69 INFO   user create start......
2018-06-05 00:43:27,535 user.py 85 INFO   create project......
2018-06-05 00:43:27,587 user.py 95 INFO   test project <Project description=, domain_id=default, enabled=True, id=d2c4fa2e664d4207a95f6604f0bdf4cd, is_domain=False, links={'self': 'http://192.30.9.9:35357/v3/projects/d2c4fa2e664d4207a95f6604f0bdf4cd'}, name=doctor, parent_id=default>
2018-06-05 00:43:27,676 user.py 103 INFO   create user......
2018-06-05 00:43:28,017 user.py 113 INFO   test user <User domain_id=default, enabled=True, id=8a23a273e53b462eb7d2a93443c3107c, links={'self': 'http://192.30.9.9:35357/v3/users/8a23a273e53b462eb7d2a93443c3107c'}, name=doctor, options={}, password_expires_at=None>
2018-06-05 00:43:28,183 user.py 127 INFO   role _member_ already created......
2018-06-05 00:43:28,183 user.py 128 INFO   test role <Role domain_id=None, id=9fe2ff9ee4384b1894a90878d3e92bab, links={'self': 'http://192.30.9.9:35357/v3/roles/9fe2ff9ee4384b1894a90878d3e92bab'}, name=_member_>
2018-06-05 00:43:28,756 user.py 77 INFO   user create end......
2018-06-05 00:43:28,756 main.py 53 INFO   doctor fault management test starting.......
2018-06-05 00:43:29,247 fault_management.py 65 INFO   fault management setup......
2018-06-05 00:43:29,248 user.py 181 INFO   user quota update start......
2018-06-05 00:43:29,920 user.py 197 INFO   user quota update end......
2018-06-05 00:43:29,920 network.py 41 INFO   network create start.......
2018-06-05 00:43:30,804 network.py 47 INFO   network create end.......
2018-06-05 00:43:30,804 network.py 49 INFO   subnet create start.......
2018-06-05 00:43:31,273 network.py 58 INFO   subnet create end.......
2018-06-05 00:43:31,274 instance.py 51 INFO   instance create start......
2018-06-05 00:43:33,473 instance.py 73 INFO   instance create end......
2018-06-05 00:43:33,473 instance.py 92 INFO   wait for vm launch start......
2018-06-05 00:43:38,105 instance.py 110 INFO   wait for vm launch end......
2018-06-05 00:43:38,105 alarm.py 45 INFO   alarm create start......
2018-06-05 00:43:39,620 alarm.py 81 INFO   alarm create end......
2018-06-05 00:43:39,621 sample.py 72 INFO   sample inspector start......
2018-06-05 00:43:40,463 sample.py 26 INFO   sample consumer start......
 * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit)
 * Running on http://0.0.0.0:12346/ (Press CTRL+C to quit)
2018-06-05 00:43:41,414 apex.py 58 INFO   Get host ip by hostname=overcloud-novacompute-1.opnfvlf.org from Apex installer......
2018-06-05 00:43:41,414 base.py 98 INFO   Run command=source stackrc; nova show overcloud-novacompute-1 | awk '/ ctlplane network /{print $5}' in apex installer......
2018-06-05 00:43:45,541 base.py 107 INFO   Output=['192.30.9.4'] command=source stackrc; nova show overcloud-novacompute-1 | awk '/ ctlplane network /{print $5}' in apex installer
2018-06-05 00:43:45,542 fault_management.py 118 INFO   Get host info(name:overcloud-novacompute-1.opnfvlf.org, ip:192.30.9.4) which vm(doctor_vm0) launched at
2018-06-05 00:43:45,542 sample.py 30 INFO   sample monitor start......
2018-06-05 00:43:45,543 sample.py 85 INFO   Starting Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.4)
2018-06-05 00:45:45,635 fault_management.py 89 INFO   fault management start......
2018-06-05 00:45:45,637 base.py 80 INFO   Get SSH keys from apex installer......
2018-06-05 00:45:45,637 base.py 84 INFO   Already have SSH keys from apex installer......
/usr/lib/python3.6/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.30.9.4: b'6cbd48c8bf66e4ca6a428087b800dc47'
  key.get_fingerprint())))
2018-06-05 00:45:45,703 utils.py 91 INFO   Copy /src/doctor-tests/doctor_tests/disable_network.sh -> disable_network.sh
2018-06-05 00:45:45,873 utils.py 72 INFO   Executing: bash disable_network.sh > disable_network.log 2>&1 &
2018-06-05 00:45:45,941 utils.py 86 INFO   *** SUCCESSFULLY run command bash disable_network.sh > disable_network.log 2>&1 &
2018-06-05 00:45:45,942 fault_management.py 91 INFO   fault management end......
2018-06-05 00:45:47,091 sample.py 98 INFO   doctor monitor detected at 1528159547.0909204
2018-06-05 00:45:47,091 sample.py 41 INFO   sample monitor report error......
2018-06-05 00:45:47,099 sample.py 168 INFO   event posted in sample inspector at 1528159547.0996966
2018-06-05 00:45:47,100 sample.py 169 INFO   sample inspector = <doctor_tests.inspector.sample.SampleInspector object at 0x7f338b27fc18>
2018-06-05 00:45:47,100 sample.py 171 INFO   sample inspector received data = b'[{"time": "2018-06-05T00:45:47.091631", "type": "compute.host.down", "details": {"hostname": "overcloud-novacompute-1.opnfvlf.org", "status": "down", "monitor": "monitor_sample", "monitor_event_id": "monitor_sample_event1"}}]'
2018-06-05 00:45:47,881 sample.py 115 INFO   doctor mark host(overcloud-novacompute-1.opnfvlf.org) down at 1528159547.8819
2018-06-05 00:45:48,030 sample.py 126 INFO   doctor mark vm(<Server: doctor_vm0>) error at 1528159548.0300255
127.0.0.1 - - [05/Jun/2018 00:45:48] "PUT /events HTTP/1.1" 200 -
2018-06-05 00:45:48,032 sample.py 101 INFO   ping timeout, quit monitoring...
2018-06-05 00:45:48,134 sample.py 58 INFO   doctor consumer notified at 1528159548.133995
2018-06-05 00:45:48,134 sample.py 61 INFO   sample consumer received data = {'severity': 'moderate', 'alarm_name': 'doctor_alarm0', 'current': 'alarm', 'alarm_id': '7eebb79c-fc6a-4a62-b88e-d53c07dbfc9d', 'reason': 'Event <id=f50f1430-2781-4445-8038-b1fc2112c8b1,event_type=compute.instance.update> hits the query <query=[{"field": "traits.instance_id", "op": "eq", "type": "", "value": "935900aa-b711-43e2-be56-2918ea7b0d31"}, {"field": "traits.state", "op": "eq", "type": "", "value": "error"}]>.', 'reason_data': {'type': 'event', 'event': {'event_type': 'compute.instance.update', 'traits': [['resource_id', 1, '935900aa-b711-43e2-be56-2918ea7b0d31'], ['ephemeral_gb', 2, 0], ['instance_type_id', 2, 78], ['user_id', 1, '8a23a273e53b462eb7d2a93443c3107c'], ['service', 1, 'compute'], ['state', 1, 'error'], ['old_state', 1, 'active'], ['project_id', 1, 'd2c4fa2e664d4207a95f6604f0bdf4cd'], ['launched_at', 4, '2018-06-05T00:36:04'], ['disk_gb', 2, 1], ['instance_id', 1, '935900aa-b711-43e2-be56-2918ea7b0d31'], ['host', 1, 'overcloud-controller-0.opnfvlf.org'], ['root_gb', 2, 1], ['tenant_id', 1, 'd2c4fa2e664d4207a95f6604f0bdf4cd'], ['memory_mb', 2, 512], ['instance_type', 1, 'm1.tiny'], ['vcpus', 2, 1], ['request_id', 1, 'req-478d154d-1a2e-403c-8dab-2ae40b731d88']], 'message_signature': 'df7b8c7d01a5499a833b79c44acb1f6cab8b005082145fb6a54526c56de6c6b2', 'raw': {}, 'generated': '2018-06-05T00:45:48.015987', 'message_id': 'f50f1430-2781-4445-8038-b1fc2112c8b1'}}, 'previous': 'insufficient data'}
127.0.0.1 - - [05/Jun/2018 00:45:48] "POST /failure HTTP/1.1" 200 -
2018-06-05 00:45:56,003 main.py 79 ERROR  doctor fault management test failed, Exception=doctor fault management test failed, notification_time=1.043074607849121
2018-06-05 00:45:56,004 fault_management.py 94 INFO   fault management cleanup......
2018-06-05 00:45:56,004 base.py 80 INFO   Get SSH keys from apex installer......
2018-06-05 00:45:56,004 base.py 84 INFO   Already have SSH keys from apex installer......
2018-06-05 00:45:57,072 utils.py 91 INFO   Copy disable_network.log -> /src/doctor-tests/doctor_tests/disable_network.log
2018-06-05 00:45:57,246 fault_management.py 155 INFO   Get the disable_netork.log fromdown_host(host_name:overcloud-novacompute-1.opnfvlf.org, host_ip:192.30.9.4)
2018-06-05 00:45:59,358 sample.py 78 INFO   sample inspector stop......
2018-06-05 00:45:59,400 sample.py 178 INFO   shutdown inspector app server at 1528159559.4004507
127.0.0.1 - - [05/Jun/2018 00:45:59] "POST /events/shutdown HTTP/1.1" 200 -
2018-06-05 00:45:59,403 sample.py 35 INFO   sample monitor stop......
2018-06-05 00:45:59,403 sample.py 108 INFO   Stopping Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.4)
2018-06-05 00:45:59,404 sample.py 31 INFO   sample consumer stop......
2018-06-05 00:45:59,409 sample.py 66 INFO   shutdown consumer app server at 1528159559.4094932
127.0.0.1 - - [05/Jun/2018 00:45:59] "POST /shutdown HTTP/1.1" 200 -
2018-06-05 00:45:59,412 alarm.py 84 INFO   alarm delete start.......
2018-06-05 00:45:59,469 alarm.py 93 INFO   alarm delete end.......
2018-06-05 00:45:59,470 instance.py 76 INFO   instance delete start.......
2018-06-05 00:46:01,007 instance.py 89 INFO   instance delete end.......
2018-06-05 00:46:01,007 network.py 61 INFO   subnet delete start.......
2018-06-05 00:46:01,539 network.py 64 INFO   subnet delete end.......
2018-06-05 00:46:01,539 network.py 66 INFO   network delete start.......
2018-06-05 00:46:02,224 network.py 69 INFO   network delete end.......
2018-06-05 00:46:02,224 apex.py 80 INFO   restore apply patches start......
2018-06-05 00:46:03,426 image.py 71 INFO   image delete start.......
2018-06-05 00:46:05,327 image.py 76 INFO   image delete end.......
2018-06-05 00:46:05,327 user.py 156 INFO   user delete start......
2018-06-05 00:46:05,806 user.py 178 INFO   user delete end......