/usr/lib/python3.6/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.168.122.30: b'cdd743e30c57bfd2fce6d180a0a7e48a'
  key.get_fingerprint())))
2018-07-09 12:34:41,864 main.py 106 INFO   doctor test starting.......
2018-07-09 12:34:41,864 apex.py 28 INFO   Setup Apex installer start......
2018-07-09 12:34:41,864 base.py 80 INFO   Get SSH keys from apex installer......
2018-07-09 12:34:42,207 apex.py 46 INFO   Get controller ips from Apex installer......
2018-07-09 12:34:42,207 base.py 98 INFO   Run command=source stackrc; nova list | grep ' overcloud-controller-[0-9] ' | sed -e 's/^.*ctlplane=//' |awk '{print $1}' in apex installer......
2018-07-09 12:34:46,046 base.py 107 INFO   Output=['192.30.9.8'] command=source stackrc; nova list | grep ' overcloud-controller-[0-9] ' | sed -e 's/^.*ctlplane=//' |awk '{print $1}' in apex installer
2018-07-09 12:34:46,047 apex.py 53 INFO   Get controller_ips:['192.30.9.8'] from Apex installer
2018-07-09 12:34:46,712 apex.py 66 INFO   Set apply patches start......
/usr/lib/python3.6/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.30.9.8: b'51eefc1655f362b0fcc6e8c91b0d39da'
  key.get_fingerprint())))
2018-07-09 12:34:51,087 base.py 60 INFO   Setup ssh stunnel in apex installer......
2018-07-09 12:34:51,093 image.py 48 INFO   image create start......
2018-07-09 12:34:56,664 image.py 68 INFO   image create end......
2018-07-09 12:34:56,665 user.py 69 INFO   user create start......
2018-07-09 12:34:56,756 user.py 85 INFO   create project......
2018-07-09 12:34:56,794 user.py 95 INFO   test project <Project description=, domain_id=default, enabled=True, id=4a0b32f2ee7c413ca6fa0894c0a84549, is_domain=False, links={'self': 'http://192.30.9.10:35357/v3/projects/4a0b32f2ee7c413ca6fa0894c0a84549'}, name=doctor, parent_id=default>
2018-07-09 12:34:56,884 user.py 103 INFO   create user......
2018-07-09 12:34:57,226 user.py 113 INFO   test user <User domain_id=default, enabled=True, id=723215d2fe8d4caf81f8f1696943cf41, links={'self': 'http://192.30.9.10:35357/v3/users/723215d2fe8d4caf81f8f1696943cf41'}, name=doctor, options={}, password_expires_at=None>
2018-07-09 12:34:57,301 user.py 127 INFO   role _member_ already created......
2018-07-09 12:34:57,302 user.py 128 INFO   test role <Role domain_id=None, id=9fe2ff9ee4384b1894a90878d3e92bab, links={'self': 'http://192.30.9.10:35357/v3/roles/9fe2ff9ee4384b1894a90878d3e92bab'}, name=_member_>
2018-07-09 12:34:57,844 user.py 77 INFO   user create end......
2018-07-09 12:34:57,845 main.py 53 INFO   doctor fault management test starting.......
2018-07-09 12:34:58,386 fault_management.py 65 INFO   fault management setup......
2018-07-09 12:34:58,386 user.py 181 INFO   user quota update start......
2018-07-09 12:34:59,066 user.py 197 INFO   user quota update end......
2018-07-09 12:34:59,066 network.py 41 INFO   network create start.......
2018-07-09 12:34:59,972 network.py 47 INFO   network create end.......
2018-07-09 12:34:59,972 network.py 49 INFO   subnet create start.......
2018-07-09 12:35:00,649 network.py 58 INFO   subnet create end.......
2018-07-09 12:35:00,649 instance.py 51 INFO   instance create start......
2018-07-09 12:35:02,702 instance.py 73 INFO   instance create end......
2018-07-09 12:35:02,702 instance.py 92 INFO   wait for vm launch start......
2018-07-09 12:35:11,716 instance.py 110 INFO   wait for vm launch end......
2018-07-09 12:35:11,716 alarm.py 45 INFO   alarm create start......
2018-07-09 12:35:13,106 alarm.py 81 INFO   alarm create end......
2018-07-09 12:35:13,106 sample.py 72 INFO   sample inspector start......
2018-07-09 12:35:13,922 sample.py 26 INFO   sample consumer start......
 * Running on http://0.0.0.0:12346/ (Press CTRL+C to quit)
 * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit)
2018-07-09 12:35:14,888 apex.py 58 INFO   Get host ip by hostname=overcloud-novacompute-1.opnfvlf.org from Apex installer......
2018-07-09 12:35:14,889 base.py 98 INFO   Run command=source stackrc; nova show overcloud-novacompute-1 | awk '/ ctlplane network /{print $5}' in apex installer......
2018-07-09 12:35:18,967 base.py 107 INFO   Output=['192.30.9.9'] command=source stackrc; nova show overcloud-novacompute-1 | awk '/ ctlplane network /{print $5}' in apex installer
2018-07-09 12:35:18,967 fault_management.py 118 INFO   Get host info(name:overcloud-novacompute-1.opnfvlf.org, ip:192.30.9.9) which vm(doctor_vm0) launched at
2018-07-09 12:35:18,968 sample.py 30 INFO   sample monitor start......
2018-07-09 12:35:18,968 sample.py 85 INFO   Starting Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.9)
2018-07-09 12:37:18,972 fault_management.py 89 INFO   fault management start......
2018-07-09 12:37:18,973 base.py 80 INFO   Get SSH keys from apex installer......
2018-07-09 12:37:18,973 base.py 84 INFO   Already have SSH keys from apex installer......
/usr/lib/python3.6/site-packages/paramiko/client.py:711: UserWarning: Unknown ssh-ed25519 host key for 192.30.9.9: b'e4be15f16f3fa1f1ed34ecc9643469af'
  key.get_fingerprint())))
2018-07-09 12:37:19,039 utils.py 91 INFO   Copy /src/doctor-tests/doctor_tests/disable_network.sh -> disable_network.sh
2018-07-09 12:37:19,207 utils.py 72 INFO   Executing: bash disable_network.sh > disable_network.log 2>&1 &
2018-07-09 12:37:19,275 utils.py 86 INFO   *** SUCCESSFULLY run command bash disable_network.sh > disable_network.log 2>&1 &
2018-07-09 12:37:19,276 fault_management.py 91 INFO   fault management end......
2018-07-09 12:37:20,404 sample.py 98 INFO   doctor monitor detected at 1531139840.404877
2018-07-09 12:37:20,405 sample.py 41 INFO   sample monitor report error......
2018-07-09 12:37:20,413 sample.py 168 INFO   event posted in sample inspector at 1531139840.4132502
2018-07-09 12:37:20,413 sample.py 169 INFO   sample inspector = <doctor_tests.inspector.sample.SampleInspector object at 0x7fc6e1df4be0>
2018-07-09 12:37:20,413 sample.py 171 INFO   sample inspector received data = b'[{"time": "2018-07-09T12:37:20.405619", "type": "compute.host.down", "details": {"hostname": "overcloud-novacompute-1.opnfvlf.org", "status": "down", "monitor": "monitor_sample", "monitor_event_id": "monitor_sample_event1"}}]'
2018-07-09 12:37:20,469 sample.py 115 INFO   doctor mark host(overcloud-novacompute-1.opnfvlf.org) down at 1531139840.4695518
2018-07-09 12:37:20,552 sample.py 126 INFO   doctor mark vm(<Server: doctor_vm0>) error at 1531139840.552722
127.0.0.1 - - [09/Jul/2018 12:37:20] "PUT /events HTTP/1.1" 200 -
2018-07-09 12:37:20,555 sample.py 101 INFO   ping timeout, quit monitoring...
2018-07-09 12:37:22,173 sample.py 58 INFO   doctor consumer notified at 1531139842.173668
2018-07-09 12:37:22,174 sample.py 61 INFO   sample consumer received data = {'severity': 'moderate', 'alarm_name': 'doctor_alarm0', 'current': 'alarm', 'alarm_id': '52427663-ffd3-493d-a591-6a7c6685698b', 'reason': 'Event <id=51daac1a-e5fd-463b-8661-8edb309ec84a,event_type=compute.instance.update> hits the query <query=[{"field": "traits.instance_id", "op": "eq", "type": "", "value": "08867186-06e4-4a97-b529-8fd8250d1e63"}, {"field": "traits.state", "op": "eq", "type": "", "value": "error"}]>.', 'reason_data': {'type': 'event', 'event': {'event_type': 'compute.instance.update', 'traits': [['resource_id', 1, '08867186-06e4-4a97-b529-8fd8250d1e63'], ['ephemeral_gb', 2, 0], ['instance_type_id', 2, 78], ['user_id', 1, '723215d2fe8d4caf81f8f1696943cf41'], ['service', 1, 'compute'], ['state', 1, 'error'], ['old_state', 1, 'active'], ['project_id', 1, '4a0b32f2ee7c413ca6fa0894c0a84549'], ['launched_at', 4, '2018-07-09T12:26:54'], ['disk_gb', 2, 1], ['instance_id', 1, '08867186-06e4-4a97-b529-8fd8250d1e63'], ['host', 1, 'overcloud-controller-0.opnfvlf.org'], ['root_gb', 2, 1], ['tenant_id', 1, '4a0b32f2ee7c413ca6fa0894c0a84549'], ['memory_mb', 2, 512], ['instance_type', 1, 'm1.tiny'], ['vcpus', 2, 1], ['request_id', 1, 'req-089415fb-d05f-4b1c-b92d-bb15b8632c75']], 'message_signature': '4ed78534190bf3d4e1a3ca41393763c9e115c12d1d3005c636f9037460d641c6', 'raw': {}, 'generated': '2018-07-09T12:37:20.548167', 'message_id': '51daac1a-e5fd-463b-8661-8edb309ec84a'}}, 'previous': 'insufficient data'}
127.0.0.1 - - [09/Jul/2018 12:37:22] "POST /failure HTTP/1.1" 200 -
2018-07-09 12:37:29,329 main.py 79 ERROR  doctor fault management test failed, Exception=doctor fault management test failed, notification_time=1.7687909603118896
2018-07-09 12:37:29,330 fault_management.py 94 INFO   fault management cleanup......
2018-07-09 12:37:29,330 base.py 80 INFO   Get SSH keys from apex installer......
2018-07-09 12:37:29,330 base.py 84 INFO   Already have SSH keys from apex installer......
2018-07-09 12:37:30,397 utils.py 91 INFO   Copy disable_network.log -> /src/doctor-tests/doctor_tests/disable_network.log
2018-07-09 12:37:30,572 fault_management.py 155 INFO   Get the disable_netork.log fromdown_host(host_name:overcloud-novacompute-1.opnfvlf.org, host_ip:192.30.9.9)
2018-07-09 12:37:32,669 sample.py 78 INFO   sample inspector stop......
2018-07-09 12:37:32,713 sample.py 178 INFO   shutdown inspector app server at 1531139852.7135344
127.0.0.1 - - [09/Jul/2018 12:37:32] "POST /events/shutdown HTTP/1.1" 200 -
2018-07-09 12:37:32,715 sample.py 35 INFO   sample monitor stop......
2018-07-09 12:37:32,716 sample.py 108 INFO   Stopping Pinger host_name(overcloud-novacompute-1.opnfvlf.org), host_ip(192.30.9.9)
2018-07-09 12:37:32,716 sample.py 31 INFO   sample consumer stop......
2018-07-09 12:37:32,720 sample.py 66 INFO   shutdown consumer app server at 1531139852.7205634
127.0.0.1 - - [09/Jul/2018 12:37:32] "POST /shutdown HTTP/1.1" 200 -
2018-07-09 12:37:32,722 alarm.py 84 INFO   alarm delete start.......
2018-07-09 12:37:32,769 alarm.py 93 INFO   alarm delete end.......
2018-07-09 12:37:32,769 instance.py 76 INFO   instance delete start.......
2018-07-09 12:37:34,498 instance.py 89 INFO   instance delete end.......
2018-07-09 12:37:34,498 network.py 61 INFO   subnet delete start.......
2018-07-09 12:37:35,032 network.py 64 INFO   subnet delete end.......
2018-07-09 12:37:35,033 network.py 66 INFO   network delete start.......
2018-07-09 12:37:35,766 network.py 69 INFO   network delete end.......
2018-07-09 12:37:35,766 apex.py 80 INFO   restore apply patches start......
2018-07-09 12:37:36,945 image.py 71 INFO   image delete start.......
2018-07-09 12:37:37,516 image.py 76 INFO   image delete end.......
2018-07-09 12:37:37,516 user.py 156 INFO   user delete start......
2018-07-09 12:37:37,996 user.py 178 INFO   user delete end......