2018-03-27 11:44:39,897 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-27 11:44:39,897 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-27 11:44:44,817 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-27 11:44:44,817 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.8'] 2018-03-27 11:44:44,818 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.7', 'overcloud-novacompute-1: 192.30.9.10'] 2018-03-27 11:44:44,818 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-27 11:44:56,299 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-27 11:44:56,341 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-27 11:45:09,141 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-27 11:45:09,394 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-27 11:45:09,395 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-27 11:45:48,284 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-27 11:45:59,788 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-27 11:45:59,788 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-27 11:47:08,781 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-27 11:47:31,436 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 11:48:09,019 - baro_tests.barometer - INFO - PASS 2018-03-27 11:48:09,019 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Hugepages. 2018-03-27 11:48:56,924 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-27 11:49:32,225 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 11:50:09,224 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-27 11:50:09,224 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-27 11:50:09,225 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-27 11:50:57,099 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-27 11:51:20,338 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 11:51:23,002 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-27 11:52:11,049 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-27 11:52:33,950 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 11:53:11,114 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-27 11:53:11,115 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-27 11:53:11,115 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-03-27 11:53:57,440 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-27 11:54:20,854 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 11:54:57,766 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-27 11:54:57,766 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-27 11:54:57,767 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-03-27 11:55:44,965 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-27 11:56:08,457 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-27 11:56:27,257 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-27 11:56:27,258 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-27 11:57:14,005 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-27 11:57:37,277 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-27 11:57:39,583 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-27 11:57:39,584 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-27 11:57:39,584 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-27 11:57:50,987 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-27 11:58:02,424 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-27 11:58:12,723 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-27 11:59:10,978 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-27 11:59:33,841 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 1973427b-521a-44a4-86e9-8c1cbdb8f863... 2018-03-27 11:59:33,841 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-27 11:59:33,842 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-27 11:59:33,842 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-27 11:59:45,625 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 11:59:45,625 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522151587.0 ... 2018-03-27 11:59:45,626 - baro_tests.barometer - DEBUG - OK 2018-03-27 11:59:45,626 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522151587.0 ... 2018-03-27 11:59:45,626 - baro_tests.barometer - DEBUG - OK 2018-03-27 11:59:45,626 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522151587.0 ... 2018-03-27 11:59:45,626 - baro_tests.barometer - DEBUG - OK 2018-03-27 11:59:45,626 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522151587.0 ... 2018-03-27 11:59:45,627 - baro_tests.barometer - DEBUG - OK 2018-03-27 11:59:45,627 - baro_tests.barometer - INFO - OK 2018-03-27 11:59:45,627 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-27 11:59:57,639 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-27 12:00:09,705 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 12:00:09,705 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522151586.999, 1522151596.999), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522151586.999, 1522151596.999), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522151586.999, 1522151596.999), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522151586.999, 1522151596.999)] 2018-03-27 12:00:09,705 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522151606.999, 1522151616.999), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522151606.999, 1522151616.999), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522151606.999, 1522151616.999), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522151606.999, 1522151616.999)] 2018-03-27 12:00:09,706 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522151587.0 - hugepages-mm-2048Kb vmpage_number-free 1522151607.0 ... 2018-03-27 12:00:09,706 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:00:09,706 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522151587.0 - hugepages-mm-2048Kb vmpage_number-used 1522151607.0 ... 2018-03-27 12:00:09,706 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:00:09,706 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522151587.0 - hugepages-node0-2048Kb vmpage_number-free 1522151607.0 ... 2018-03-27 12:00:09,706 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:00:09,706 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522151587.0 - hugepages-node0-2048Kb vmpage_number-used 1522151607.0 ... 2018-03-27 12:00:09,707 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:00:09,707 - baro_tests.barometer - INFO - OK 2018-03-27 12:00:09,707 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-27 12:00:57,315 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-27 12:01:32,823 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 1973427b-521a-44a4-86e9-8c1cbdb8f863... 2018-03-27 12:01:32,825 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-27 12:01:32,826 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-27 12:01:32,829 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-27 12:01:45,273 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 12:01:45,273 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522151707.0 ... 2018-03-27 12:01:45,274 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:01:45,274 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522151707.0 ... 2018-03-27 12:01:45,274 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:01:45,275 - baro_tests.barometer - INFO - OK 2018-03-27 12:01:45,275 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-27 12:01:57,282 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-27 12:02:08,871 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 12:02:08,872 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522151706.999, 1522151716.999), ('intel_rdt-0-2', 'ipc', 1522151706.999, 1522151716.999)] 2018-03-27 12:02:08,872 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522151726.999, 1522151736.999), ('intel_rdt-0-2', 'ipc', 1522151726.999, 1522151736.999)] 2018-03-27 12:02:08,872 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522151707.0 - intel_rdt-0-2 bytes-llc 1522151727.0 ... 2018-03-27 12:02:08,872 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:02:08,873 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522151707.0 - intel_rdt-0-2 ipc 1522151727.0 ... 2018-03-27 12:02:08,873 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:02:08,873 - baro_tests.barometer - INFO - OK 2018-03-27 12:02:08,874 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-27 12:02:55,653 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-27 12:03:18,626 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 1973427b-521a-44a4-86e9-8c1cbdb8f863... 2018-03-27 12:03:18,627 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 12:03:18,627 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-27 12:03:18,627 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-27 12:03:29,237 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-27 12:03:29,364 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-27 12:03:29,465 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-27 12:03:29,597 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-27 12:03:29,598 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-27 12:03:29,598 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-27 12:04:13,580 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-27 12:04:35,387 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 1973427b-521a-44a4-86e9-8c1cbdb8f863... 2018-03-27 12:04:35,388 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 12:04:35,388 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-27 12:04:35,388 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-27 12:04:47,266 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 12:04:47,266 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522151900.0 ... 2018-03-27 12:04:47,267 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:04:47,267 - baro_tests.barometer - INFO - OK 2018-03-27 12:04:47,267 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-27 12:04:50,270 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-27 12:05:01,638 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 12:05:01,639 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522151900.0, 1522151901.001)] 2018-03-27 12:05:01,639 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522151914.0, 1522151915.0)] 2018-03-27 12:05:01,639 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522151900.0 - ovs_events-br0 gauge-link_status 1522151914.0 ... 2018-03-27 12:05:01,640 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:05:01,640 - baro_tests.barometer - INFO - OK 2018-03-27 12:05:01,640 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-27 12:05:47,529 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-27 12:06:09,579 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 1973427b-521a-44a4-86e9-8c1cbdb8f863... 2018-03-27 12:06:09,580 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 12:06:09,580 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-27 12:06:09,580 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-27 12:06:22,156 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 12:06:22,156 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522151995.0 ... 2018-03-27 12:06:22,156 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:06:22,156 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522151995.0 ... 2018-03-27 12:06:22,157 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:06:22,157 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522151995.0 ... 2018-03-27 12:06:22,157 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:06:22,157 - baro_tests.barometer - INFO - OK 2018-03-27 12:06:22,158 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-27 12:06:25,158 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-27 12:06:37,087 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 12:06:37,088 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522151995.0, 1522151996.0), ('ovs_stats-br0.br0', 'if_errors', 1522151995.0, 1522151996.0), ('ovs_stats-br0.br0', 'if_packets', 1522151995.0, 1522151996.0)] 2018-03-27 12:06:37,088 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522152010.0, 1522152011.0), ('ovs_stats-br0.br0', 'if_errors', 1522152010.0, 1522152011.0), ('ovs_stats-br0.br0', 'if_packets', 1522152010.0, 1522152011.0)] 2018-03-27 12:06:37,088 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522151995.0 - ovs_stats-br0.br0 if_dropped 1522152010.0 ... 2018-03-27 12:06:37,089 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:06:37,089 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522151995.0 - ovs_stats-br0.br0 if_errors 1522152010.0 ... 2018-03-27 12:06:37,089 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:06:37,090 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522151995.0 - ovs_stats-br0.br0 if_packets 1522152010.0 ... 2018-03-27 12:06:37,090 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:06:37,090 - baro_tests.barometer - INFO - OK 2018-03-27 12:06:37,090 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-27 12:06:48,361 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-27 12:06:59,021 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-27 12:06:59,021 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-27 12:08:11,177 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-27 12:08:33,941 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 12:09:10,948 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-27 12:09:10,949 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-27 12:09:10,949 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-03-27 12:09:57,699 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-27 12:10:30,853 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 12:11:07,856 - baro_tests.barometer - INFO - PASS 2018-03-27 12:11:07,857 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Intel RDT. 2018-03-27 12:11:55,109 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-27 12:12:17,808 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 12:12:20,329 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-27 12:13:05,938 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-27 12:13:27,869 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 12:13:30,522 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-03-27 12:14:16,570 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-27 12:14:38,621 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 12:15:15,910 - baro_tests.barometer - INFO - PASS 2018-03-27 12:15:15,910 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on OVS stats. 2018-03-27 12:16:01,125 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-27 12:16:22,725 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-27 12:16:41,375 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-27 12:16:41,378 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-27 12:17:26,552 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-27 12:17:48,912 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-27 12:17:51,274 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-27 12:17:51,274 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-27 12:17:51,274 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-27 12:18:02,503 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-27 12:18:14,302 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-27 12:18:24,610 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-27 12:19:20,705 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-27 12:19:42,106 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 3e57eb27-cae4-49ef-ad12-a563c5724237... 2018-03-27 12:19:42,106 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-27 12:19:42,107 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-27 12:19:42,107 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-27 12:19:53,393 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 12:19:53,393 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522152799.57 ... 2018-03-27 12:19:53,394 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:19:53,394 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522152799.57 ... 2018-03-27 12:19:53,394 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:19:53,394 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522152799.57 ... 2018-03-27 12:19:53,395 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:19:53,395 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522152799.57 ... 2018-03-27 12:19:53,395 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:19:53,395 - baro_tests.barometer - INFO - OK 2018-03-27 12:19:53,395 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-27 12:20:05,405 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-27 12:20:17,073 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 12:20:17,074 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522152799.574, 1522152809.574), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522152799.574, 1522152809.574), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522152799.574, 1522152809.574), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522152799.574, 1522152809.574)] 2018-03-27 12:20:17,074 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522152829.574, 1522152839.574), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522152829.574, 1522152839.574), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522152829.574, 1522152839.574), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522152829.574, 1522152839.574)] 2018-03-27 12:20:17,075 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522152799.57 - hugepages-mm-2048Kb vmpage_number-free 1522152829.57 ... 2018-03-27 12:20:17,075 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:20:17,075 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522152799.57 - hugepages-mm-2048Kb vmpage_number-used 1522152829.57 ... 2018-03-27 12:20:17,075 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:20:17,076 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522152799.57 - hugepages-node0-2048Kb vmpage_number-free 1522152829.57 ... 2018-03-27 12:20:17,076 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:20:17,076 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522152799.57 - hugepages-node0-2048Kb vmpage_number-used 1522152829.57 ... 2018-03-27 12:20:17,076 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:20:17,076 - baro_tests.barometer - INFO - OK 2018-03-27 12:20:17,076 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-27 12:21:01,221 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-27 12:21:35,542 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 3e57eb27-cae4-49ef-ad12-a563c5724237... 2018-03-27 12:21:35,542 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-27 12:21:35,543 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-27 12:21:35,543 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-27 12:21:48,101 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 12:21:48,102 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522152919.57 ... 2018-03-27 12:21:48,103 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:21:48,105 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522152919.57 ... 2018-03-27 12:21:48,106 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:21:48,107 - baro_tests.barometer - INFO - OK 2018-03-27 12:21:48,107 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-27 12:22:00,107 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-27 12:22:11,729 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 12:22:11,729 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522152919.574, 1522152929.574), ('intel_rdt-0-2', 'ipc', 1522152919.574, 1522152929.574)] 2018-03-27 12:22:11,729 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522152939.574, 1522152949.574), ('intel_rdt-0-2', 'ipc', 1522152939.574, 1522152949.574)] 2018-03-27 12:22:11,730 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522152919.57 - intel_rdt-0-2 bytes-llc 1522152939.57 ... 2018-03-27 12:22:11,730 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:22:11,730 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522152919.57 - intel_rdt-0-2 ipc 1522152939.57 ... 2018-03-27 12:22:11,731 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:22:11,731 - baro_tests.barometer - INFO - OK 2018-03-27 12:22:11,731 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-27 12:22:56,364 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-27 12:23:18,485 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 3e57eb27-cae4-49ef-ad12-a563c5724237... 2018-03-27 12:23:18,485 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 12:23:18,488 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-27 12:23:18,489 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-27 12:23:30,068 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-27 12:23:30,169 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-27 12:23:30,300 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-27 12:23:30,412 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-27 12:23:30,413 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-27 12:23:30,414 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-27 12:24:15,001 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-27 12:24:38,108 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 3e57eb27-cae4-49ef-ad12-a563c5724237... 2018-03-27 12:24:38,108 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 12:24:38,109 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-27 12:24:38,109 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-27 12:24:49,725 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 12:24:49,726 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522153113.58 ... 2018-03-27 12:24:49,727 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:24:49,727 - baro_tests.barometer - INFO - OK 2018-03-27 12:24:49,727 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-27 12:24:52,729 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-27 12:25:03,419 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 12:25:03,419 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522153113.575, 1522153114.575)] 2018-03-27 12:25:03,419 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522153126.575, 1522153127.575)] 2018-03-27 12:25:03,419 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522153113.58 - ovs_events-br0 gauge-link_status 1522153126.58 ... 2018-03-27 12:25:03,419 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:25:03,420 - baro_tests.barometer - INFO - OK 2018-03-27 12:25:03,420 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-27 12:25:49,238 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-27 12:26:10,990 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 3e57eb27-cae4-49ef-ad12-a563c5724237... 2018-03-27 12:26:10,991 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 12:26:10,991 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-27 12:26:10,991 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-27 12:26:21,750 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 12:26:21,751 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522153204.58 ... 2018-03-27 12:26:21,751 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:26:21,751 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522153205.58 ... 2018-03-27 12:26:21,752 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:26:21,752 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522153205.58 ... 2018-03-27 12:26:21,752 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:26:21,752 - baro_tests.barometer - INFO - OK 2018-03-27 12:26:21,752 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-27 12:26:24,756 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-27 12:26:36,574 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 12:26:36,575 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522153204.575, 1522153205.575), ('ovs_stats-br0.br0', 'if_errors', 1522153205.575, 1522153206.575), ('ovs_stats-br0.br0', 'if_packets', 1522153205.575, 1522153206.575)] 2018-03-27 12:26:36,575 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522153219.575, 1522153220.575), ('ovs_stats-br0.br0', 'if_errors', 1522153219.575, 1522153220.575), ('ovs_stats-br0.br0', 'if_packets', 1522153219.575, 1522153220.575)] 2018-03-27 12:26:36,575 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522153204.58 - ovs_stats-br0.br0 if_dropped 1522153219.58 ... 2018-03-27 12:26:36,576 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:26:36,576 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522153205.58 - ovs_stats-br0.br0 if_errors 1522153219.58 ... 2018-03-27 12:26:36,576 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:26:36,576 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522153205.58 - ovs_stats-br0.br0 if_packets 1522153219.58 ... 2018-03-27 12:26:36,576 - baro_tests.barometer - DEBUG - OK 2018-03-27 12:26:36,576 - baro_tests.barometer - INFO - OK 2018-03-27 12:26:36,577 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-27 12:26:48,071 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-27 12:26:48,071 - baro_tests.barometer - INFO - ====================================================================== 2018-03-27 12:26:48,071 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-27 12:26:48,072 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-27 12:26:48,072 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 12:26:48,072 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-27 12:26:48,072 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 12:26:48,072 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-27 12:26:48,072 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | FAIL | 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | PASS | 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | PASS | 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 12:26:48,073 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-27 12:26:48,074 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-27 12:26:48,074 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-27 12:26:48,074 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-27 12:26:48,074 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-27 12:26:48,074 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-27 12:26:48,074 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 12:26:48,074 - baro_tests.barometer - INFO - ====================================================================== 2018-03-27 12:26:48,074 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-27 12:26:48,074 - baro_tests.barometer - ERROR - Overall Result is Fail