2018-03-29 10:47:06,966 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-29 10:47:06,966 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-29 10:47:11,536 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-29 10:47:11,536 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.4', 'overcloud-controller-1: 192.30.9.7', 'overcloud-controller-2: 192.30.9.6'] 2018-03-29 10:47:11,536 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.3', 'overcloud-novacompute-1: 192.30.9.10'] 2018-03-29 10:47:11,536 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-29 10:47:25,795 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-29 10:47:25,837 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-29 10:47:39,326 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-29 10:47:39,582 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-29 10:47:39,583 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-29 10:49:27,942 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-29 10:49:41,581 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-29 10:49:41,581 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-29 10:51:03,095 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-29 10:51:29,285 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 10:52:07,316 - baro_tests.barometer - INFO - PASS 2018-03-29 10:52:07,316 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Hugepages. 2018-03-29 10:53:00,561 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-29 10:53:41,048 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 10:54:18,385 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-29 10:54:18,385 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-29 10:54:18,385 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-29 10:55:10,120 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-29 10:55:38,607 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 10:55:41,200 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-29 10:55:44,405 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-29 10:55:47,150 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-29 10:56:40,578 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-29 10:57:06,452 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 10:57:43,917 - baro_tests.barometer - INFO - PASS 2018-03-29 10:57:43,918 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on OVS events. 2018-03-29 10:58:39,271 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-29 10:59:04,741 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 10:59:42,083 - baro_tests.barometer - INFO - PASS 2018-03-29 10:59:42,084 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on OVS stats. 2018-03-29 11:00:34,583 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-29 11:01:00,506 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-29 11:01:20,117 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-29 11:01:20,118 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-29 11:02:11,666 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-29 11:02:38,213 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-29 11:02:40,585 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-29 11:02:43,056 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-29 11:02:45,512 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-29 11:02:45,512 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-29 11:02:45,512 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-29 11:02:57,930 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-29 11:03:08,289 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-29 11:03:18,578 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-29 11:04:20,469 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-29 11:04:45,405 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 088d9672-6436-4cf9-84ae-c23cf18b7f4e... 2018-03-29 11:04:45,406 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-29 11:04:45,406 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-29 11:04:45,406 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-29 11:04:59,027 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-29 11:04:59,028 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522321090.34 ... 2018-03-29 11:04:59,028 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:04:59,028 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522321090.34 ... 2018-03-29 11:04:59,029 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:04:59,029 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522321090.34 ... 2018-03-29 11:04:59,029 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:04:59,029 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522321090.34 ... 2018-03-29 11:04:59,030 - baro_tests.barometer - ERROR - Time of last two entries differ by 20.0, but interval is 10 2018-03-29 11:04:59,030 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Hugepages. 2018-03-29 11:05:50,159 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-29 11:06:28,314 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 088d9672-6436-4cf9-84ae-c23cf18b7f4e... 2018-03-29 11:06:28,315 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-29 11:06:28,317 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-29 11:06:28,317 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-29 11:06:40,952 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-29 11:06:40,954 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522321200.34 ... 2018-03-29 11:06:40,955 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:06:40,955 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522321200.34 ... 2018-03-29 11:06:40,956 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:06:40,956 - baro_tests.barometer - INFO - OK 2018-03-29 11:06:40,957 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-29 11:06:52,969 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-29 11:07:06,602 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-29 11:07:06,603 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522321200.336, 1522321210.335), ('intel_rdt-0-2', 'ipc', 1522321200.336, 1522321210.335)] 2018-03-29 11:07:06,603 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522321220.336, 1522321230.335), ('intel_rdt-0-2', 'ipc', 1522321220.336, 1522321230.335)] 2018-03-29 11:07:06,603 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522321200.34 - intel_rdt-0-2 bytes-llc 1522321220.34 ... 2018-03-29 11:07:06,604 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:07:06,604 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522321200.34 - intel_rdt-0-2 ipc 1522321220.34 ... 2018-03-29 11:07:06,604 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:07:06,605 - baro_tests.barometer - INFO - OK 2018-03-29 11:07:06,605 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-29 11:07:55,691 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-29 11:08:20,732 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 088d9672-6436-4cf9-84ae-c23cf18b7f4e... 2018-03-29 11:08:20,733 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-29 11:08:20,733 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-29 11:08:20,733 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-29 11:08:34,149 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-29 11:08:34,256 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-29 11:08:34,360 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-29 11:08:34,521 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-29 11:08:34,522 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-29 11:08:34,523 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-29 11:09:28,951 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-29 11:09:54,001 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 088d9672-6436-4cf9-84ae-c23cf18b7f4e... 2018-03-29 11:09:54,001 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-29 11:09:54,002 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-29 11:09:54,002 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-29 11:10:06,621 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-29 11:10:06,621 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522321416.34 ... 2018-03-29 11:10:06,622 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:10:06,622 - baro_tests.barometer - INFO - OK 2018-03-29 11:10:06,622 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-29 11:10:09,626 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-29 11:10:22,122 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-29 11:10:22,123 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522321416.336, 1522321418.337)] 2018-03-29 11:10:22,123 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522321432.337, 1522321433.337)] 2018-03-29 11:10:22,124 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522321416.34 - ovs_events-br0 gauge-link_status 1522321432.34 ... 2018-03-29 11:10:22,124 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:10:22,125 - baro_tests.barometer - INFO - OK 2018-03-29 11:10:22,125 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-29 11:11:11,495 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-29 11:11:36,507 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 088d9672-6436-4cf9-84ae-c23cf18b7f4e... 2018-03-29 11:11:36,507 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-29 11:11:36,507 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-29 11:11:36,508 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-29 11:11:48,885 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-29 11:11:48,885 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522321518.34 ... 2018-03-29 11:11:48,886 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:11:48,886 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522321518.34 ... 2018-03-29 11:11:48,886 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:11:48,887 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522321518.34 ... 2018-03-29 11:11:48,887 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:11:48,887 - baro_tests.barometer - INFO - OK 2018-03-29 11:11:48,887 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-29 11:11:51,890 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-29 11:12:04,236 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-29 11:12:04,236 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522321518.336, 1522321519.336), ('ovs_stats-br0.br0', 'if_errors', 1522321518.336, 1522321519.336), ('ovs_stats-br0.br0', 'if_packets', 1522321518.336, 1522321519.336)] 2018-03-29 11:12:04,237 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522321534.336, 1522321535.336), ('ovs_stats-br0.br0', 'if_errors', 1522321534.336, 1522321535.336), ('ovs_stats-br0.br0', 'if_packets', 1522321534.336, 1522321535.336)] 2018-03-29 11:12:04,237 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522321518.34 - ovs_stats-br0.br0 if_dropped 1522321534.34 ... 2018-03-29 11:12:04,237 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:12:04,238 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522321518.34 - ovs_stats-br0.br0 if_errors 1522321534.34 ... 2018-03-29 11:12:04,238 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:12:04,238 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522321518.34 - ovs_stats-br0.br0 if_packets 1522321534.34 ... 2018-03-29 11:12:04,238 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:12:04,238 - baro_tests.barometer - INFO - OK 2018-03-29 11:12:04,238 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-29 11:12:17,140 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-29 11:12:29,235 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-29 11:12:29,235 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-29 11:13:45,285 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-29 11:14:10,798 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 11:14:48,126 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-29 11:14:48,126 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-29 11:14:48,126 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-03-29 11:15:41,325 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-29 11:16:20,738 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 11:16:58,051 - baro_tests.barometer - INFO - PASS 2018-03-29 11:16:58,052 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Intel RDT. 2018-03-29 11:17:48,456 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-29 11:18:12,064 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 11:18:14,705 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-29 11:18:17,275 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-29 11:18:19,986 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-29 11:19:10,897 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-29 11:19:36,429 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 11:20:13,617 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-29 11:20:13,617 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-29 11:20:13,618 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-03-29 11:21:04,516 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-29 11:21:29,598 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-29 11:22:07,194 - baro_tests.barometer - INFO - PASS 2018-03-29 11:22:07,196 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on OVS stats. 2018-03-29 11:23:00,036 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-29 11:23:24,513 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-29 11:23:43,484 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-29 11:23:43,485 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-29 11:24:34,625 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-29 11:25:00,753 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-29 11:25:03,125 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-29 11:25:05,373 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-29 11:25:07,696 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-29 11:25:07,696 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-29 11:25:07,696 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-29 11:25:19,464 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-29 11:25:31,146 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-29 11:25:41,453 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-29 11:26:42,397 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-29 11:27:07,545 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 13a49543-4cba-4a02-b43e-e65d1779f762... 2018-03-29 11:27:07,545 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-29 11:27:07,545 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-29 11:27:07,546 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-29 11:27:20,298 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-29 11:27:20,299 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522322445.55 ... 2018-03-29 11:27:20,300 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:27:20,302 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522322445.55 ... 2018-03-29 11:27:20,303 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:27:20,307 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522322445.55 ... 2018-03-29 11:27:20,307 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:27:20,307 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522322445.55 ... 2018-03-29 11:27:20,308 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:27:20,308 - baro_tests.barometer - INFO - OK 2018-03-29 11:27:20,308 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-29 11:27:32,320 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-29 11:27:45,648 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-29 11:27:45,651 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522322445.546, 1522322455.546), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522322445.546, 1522322455.546), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522322445.546, 1522322455.546), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522322445.546, 1522322455.546)] 2018-03-29 11:27:45,655 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522322475.546, 1522322485.546), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522322475.546, 1522322485.546), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522322475.546, 1522322485.546), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522322475.546, 1522322485.546)] 2018-03-29 11:27:45,656 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522322445.55 - hugepages-mm-2048Kb vmpage_number-free 1522322475.55 ... 2018-03-29 11:27:45,657 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:27:45,657 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522322445.55 - hugepages-mm-2048Kb vmpage_number-used 1522322475.55 ... 2018-03-29 11:27:45,657 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:27:45,657 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522322445.55 - hugepages-node0-2048Kb vmpage_number-free 1522322475.55 ... 2018-03-29 11:27:45,657 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:27:45,657 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522322445.55 - hugepages-node0-2048Kb vmpage_number-used 1522322475.55 ... 2018-03-29 11:27:45,657 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:27:45,658 - baro_tests.barometer - INFO - OK 2018-03-29 11:27:45,660 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-29 11:28:34,943 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-29 11:29:11,862 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 13a49543-4cba-4a02-b43e-e65d1779f762... 2018-03-29 11:29:11,863 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-29 11:29:11,863 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-29 11:29:11,863 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-29 11:29:24,652 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-29 11:29:24,654 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522322575.55 ... 2018-03-29 11:29:24,655 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:29:24,656 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522322575.55 ... 2018-03-29 11:29:24,660 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:29:24,662 - baro_tests.barometer - INFO - OK 2018-03-29 11:29:24,664 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-29 11:29:36,673 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-29 11:29:48,911 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-29 11:29:48,911 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522322575.546, 1522322585.546), ('intel_rdt-0-2', 'ipc', 1522322575.546, 1522322585.546)] 2018-03-29 11:29:48,912 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522322595.546, 1522322605.546), ('intel_rdt-0-2', 'ipc', 1522322595.546, 1522322605.546)] 2018-03-29 11:29:48,913 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522322575.55 - intel_rdt-0-2 bytes-llc 1522322595.55 ... 2018-03-29 11:29:48,915 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:29:48,916 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522322575.55 - intel_rdt-0-2 ipc 1522322595.55 ... 2018-03-29 11:29:48,916 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:29:48,916 - baro_tests.barometer - INFO - OK 2018-03-29 11:29:48,918 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-29 11:30:38,470 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-29 11:31:03,458 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 13a49543-4cba-4a02-b43e-e65d1779f762... 2018-03-29 11:31:03,458 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-29 11:31:03,459 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-29 11:31:03,459 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-29 11:31:15,337 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-29 11:31:15,469 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-29 11:31:15,580 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-29 11:31:15,711 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-29 11:31:15,711 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-29 11:31:15,713 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-29 11:32:04,885 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-29 11:32:30,155 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 13a49543-4cba-4a02-b43e-e65d1779f762... 2018-03-29 11:32:30,156 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-29 11:32:30,156 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-29 11:32:30,156 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-29 11:32:42,030 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-29 11:32:42,030 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522322784.55 ... 2018-03-29 11:32:42,031 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:32:42,031 - baro_tests.barometer - INFO - OK 2018-03-29 11:32:42,031 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-29 11:32:45,034 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-29 11:32:57,756 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-29 11:32:57,757 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522322784.547, 1522322785.547)] 2018-03-29 11:32:57,757 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522322800.547, 1522322801.547)] 2018-03-29 11:32:57,757 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522322784.55 - ovs_events-br0 gauge-link_status 1522322800.55 ... 2018-03-29 11:32:57,757 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:32:57,757 - baro_tests.barometer - INFO - OK 2018-03-29 11:32:57,758 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-29 11:33:48,762 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-29 11:34:13,761 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 13a49543-4cba-4a02-b43e-e65d1779f762... 2018-03-29 11:34:13,763 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-29 11:34:13,764 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-29 11:34:13,765 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-29 11:34:26,672 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-29 11:34:26,672 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522322888.55 ... 2018-03-29 11:34:26,673 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:34:26,673 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522322889.55 ... 2018-03-29 11:34:26,674 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:34:26,674 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522322889.55 ... 2018-03-29 11:34:26,675 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:34:26,676 - baro_tests.barometer - INFO - OK 2018-03-29 11:34:26,676 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-29 11:34:29,679 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-29 11:34:41,617 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-29 11:34:41,617 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522322888.547, 1522322890.547), ('ovs_stats-br0.br0', 'if_errors', 1522322889.547, 1522322890.547), ('ovs_stats-br0.br0', 'if_packets', 1522322889.547, 1522322890.547)] 2018-03-29 11:34:41,618 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522322903.547, 1522322904.547), ('ovs_stats-br0.br0', 'if_errors', 1522322903.547, 1522322904.547), ('ovs_stats-br0.br0', 'if_packets', 1522322903.547, 1522322904.547)] 2018-03-29 11:34:41,618 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522322888.55 - ovs_stats-br0.br0 if_dropped 1522322903.55 ... 2018-03-29 11:34:41,618 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:34:41,619 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522322889.55 - ovs_stats-br0.br0 if_errors 1522322903.55 ... 2018-03-29 11:34:41,619 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:34:41,619 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522322889.55 - ovs_stats-br0.br0 if_packets 1522322903.55 ... 2018-03-29 11:34:41,619 - baro_tests.barometer - DEBUG - OK 2018-03-29 11:34:41,619 - baro_tests.barometer - INFO - OK 2018-03-29 11:34:41,619 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-29 11:34:55,188 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-29 11:34:55,189 - baro_tests.barometer - INFO - ====================================================================== 2018-03-29 11:34:55,191 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-29 11:34:55,191 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-29 11:34:55,194 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-29 11:34:55,194 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-29 11:34:55,194 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-29 11:34:55,194 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-29 11:34:55,194 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | FAIL | 2018-03-29 11:34:55,195 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | PASS | 2018-03-29 11:34:55,195 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-29 11:34:55,195 - baro_tests.barometer - INFO - | IN:OVS events | PASS | FAIL | 2018-03-29 11:34:55,195 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-29 11:34:55,195 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-29 11:34:55,195 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-29 11:34:55,195 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-29 11:34:55,195 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-03-29 11:34:55,195 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-29 11:34:55,196 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-29 11:34:55,196 - baro_tests.barometer - INFO - | IN:Hugepages | FAIL | PASS | 2018-03-29 11:34:55,196 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-29 11:34:55,196 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-29 11:34:55,196 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-29 11:34:55,196 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-29 11:34:55,196 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-29 11:34:55,196 - baro_tests.barometer - INFO - ====================================================================== 2018-03-29 11:34:55,196 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-29 11:34:55,196 - baro_tests.barometer - ERROR - Overall Result is Fail