2018-03-27 05:23:54,960 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-27 05:23:54,960 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-27 05:23:58,942 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-27 05:23:58,943 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.9', 'overcloud-controller-1: 192.30.9.10', 'overcloud-controller-2: 192.30.9.3'] 2018-03-27 05:23:58,943 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.4', 'overcloud-novacompute-1: 192.30.9.5'] 2018-03-27 05:23:58,943 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-27 05:24:12,049 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-27 05:24:12,091 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-27 05:24:25,180 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-27 05:24:25,428 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-27 05:24:25,429 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-27 05:26:14,870 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-27 05:26:28,450 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-27 05:26:28,450 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-27 05:27:48,460 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-27 05:28:15,885 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:28:54,003 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-27 05:28:54,004 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-27 05:28:54,004 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Hugepages. 2018-03-27 05:29:47,962 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-27 05:30:25,250 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:30:28,159 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 05:30:31,268 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-27 05:30:34,458 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-27 05:31:30,405 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-27 05:31:56,873 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:31:59,575 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 05:32:02,339 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-27 05:32:04,918 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-27 05:32:58,821 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-27 05:33:27,609 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:33:30,316 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 05:33:32,959 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-27 05:33:35,554 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-03-27 05:34:29,601 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-27 05:34:55,590 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:34:58,205 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 05:35:00,811 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-27 05:35:03,379 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-03-27 05:35:55,978 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-27 05:36:20,683 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-27 05:36:39,534 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-27 05:36:39,535 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-27 05:37:33,160 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-27 05:38:01,596 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-27 05:38:03,935 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-27 05:38:06,774 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-27 05:38:09,448 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-27 05:38:09,449 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-27 05:38:09,449 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-27 05:38:22,473 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-27 05:38:33,296 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-27 05:38:45,165 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-27 05:39:47,339 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-27 05:40:14,052 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node e22a5499-d5b1-4fe0-89db-0d7214c91689... 2018-03-27 05:40:14,053 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-27 05:40:14,054 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-27 05:40:14,054 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-27 05:40:27,149 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 05:40:27,150 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522128840.28 ... 2018-03-27 05:40:27,150 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:40:27,150 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522128840.28 ... 2018-03-27 05:40:27,151 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:40:27,151 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522128840.28 ... 2018-03-27 05:40:27,151 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:40:27,151 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522128840.28 ... 2018-03-27 05:40:27,151 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:40:27,151 - baro_tests.barometer - INFO - OK 2018-03-27 05:40:27,151 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-27 05:40:39,163 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-27 05:40:53,310 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 05:40:53,310 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522128840.275, 1522128850.275), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522128840.275, 1522128850.275), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522128840.275, 1522128850.275), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522128840.275, 1522128850.275)] 2018-03-27 05:40:53,311 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522128860.275, 1522128870.275), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522128860.275, 1522128870.275), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522128860.275, 1522128870.275), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522128860.275, 1522128870.275)] 2018-03-27 05:40:53,311 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522128840.28 - hugepages-mm-2048Kb vmpage_number-free 1522128860.28 ... 2018-03-27 05:40:53,311 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:40:53,311 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522128840.28 - hugepages-mm-2048Kb vmpage_number-used 1522128860.28 ... 2018-03-27 05:40:53,311 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:40:53,311 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522128840.28 - hugepages-node0-2048Kb vmpage_number-free 1522128860.28 ... 2018-03-27 05:40:53,312 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:40:53,312 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522128840.28 - hugepages-node0-2048Kb vmpage_number-used 1522128860.28 ... 2018-03-27 05:40:53,312 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:40:53,312 - baro_tests.barometer - INFO - OK 2018-03-27 05:40:53,312 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-27 05:41:46,917 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-27 05:42:27,862 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node e22a5499-d5b1-4fe0-89db-0d7214c91689... 2018-03-27 05:42:27,862 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-27 05:42:27,862 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-27 05:42:27,863 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-27 05:42:40,273 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 05:42:40,273 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522128970.28 ... 2018-03-27 05:42:40,273 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:42:40,273 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522128970.28 ... 2018-03-27 05:42:40,273 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:42:40,274 - baro_tests.barometer - INFO - OK 2018-03-27 05:42:40,274 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-27 05:42:52,286 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-27 05:43:05,169 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 05:43:05,169 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522128970.275, 1522128980.275), ('intel_rdt-0-2', 'ipc', 1522128970.275, 1522128980.275)] 2018-03-27 05:43:05,169 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522128990.275, 1522129000.275), ('intel_rdt-0-2', 'ipc', 1522128990.275, 1522129000.275)] 2018-03-27 05:43:05,169 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522128970.28 - intel_rdt-0-2 bytes-llc 1522128990.28 ... 2018-03-27 05:43:05,170 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:43:05,170 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522128970.28 - intel_rdt-0-2 ipc 1522128990.28 ... 2018-03-27 05:43:05,170 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:43:05,170 - baro_tests.barometer - INFO - OK 2018-03-27 05:43:05,171 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-27 05:43:55,185 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-27 05:44:19,565 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node e22a5499-d5b1-4fe0-89db-0d7214c91689... 2018-03-27 05:44:19,566 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 05:44:19,566 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-27 05:44:19,566 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-27 05:44:31,728 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-27 05:44:31,832 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-27 05:44:31,962 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-27 05:44:32,101 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-27 05:44:32,102 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-27 05:44:32,103 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-27 05:45:24,354 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-27 05:45:50,701 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node e22a5499-d5b1-4fe0-89db-0d7214c91689... 2018-03-27 05:45:50,702 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 05:45:50,704 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-27 05:45:50,705 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-27 05:46:04,242 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 05:46:04,242 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522129187.74 ... 2018-03-27 05:46:04,243 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:46:04,243 - baro_tests.barometer - INFO - OK 2018-03-27 05:46:04,243 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-27 05:46:07,244 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-27 05:46:19,997 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 05:46:19,997 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522129187.738, 1522129188.738)] 2018-03-27 05:46:19,998 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522129203.738, 1522129204.738)] 2018-03-27 05:46:19,998 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522129187.74 - ovs_events-br0 gauge-link_status 1522129203.74 ... 2018-03-27 05:46:19,998 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:46:19,999 - baro_tests.barometer - INFO - OK 2018-03-27 05:46:19,999 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-27 05:47:10,668 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-27 05:47:35,348 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node e22a5499-d5b1-4fe0-89db-0d7214c91689... 2018-03-27 05:47:35,349 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 05:47:35,349 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-27 05:47:35,349 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-27 05:47:47,685 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 05:47:47,686 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522129290.74 ... 2018-03-27 05:47:47,686 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:47:47,687 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522129290.74 ... 2018-03-27 05:47:47,687 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:47:47,688 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522129291.74 ... 2018-03-27 05:47:47,688 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:47:47,689 - baro_tests.barometer - INFO - OK 2018-03-27 05:47:47,691 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-27 05:47:50,695 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-27 05:48:03,886 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 05:48:03,887 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522129290.738, 1522129291.738), ('ovs_stats-br0.br0', 'if_errors', 1522129290.738, 1522129291.738), ('ovs_stats-br0.br0', 'if_packets', 1522129291.738, 1522129292.738)] 2018-03-27 05:48:03,887 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522129307.738, 1522129308.738), ('ovs_stats-br0.br0', 'if_errors', 1522129307.738, 1522129308.738), ('ovs_stats-br0.br0', 'if_packets', 1522129307.738, 1522129308.738)] 2018-03-27 05:48:03,887 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522129290.74 - ovs_stats-br0.br0 if_dropped 1522129307.74 ... 2018-03-27 05:48:03,888 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:48:03,888 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522129290.74 - ovs_stats-br0.br0 if_errors 1522129307.74 ... 2018-03-27 05:48:03,888 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:48:03,888 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522129291.74 - ovs_stats-br0.br0 if_packets 1522129307.74 ... 2018-03-27 05:48:03,888 - baro_tests.barometer - DEBUG - OK 2018-03-27 05:48:03,888 - baro_tests.barometer - INFO - OK 2018-03-27 05:48:03,889 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-27 05:48:15,900 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-27 05:48:29,101 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-27 05:48:29,103 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-27 05:49:50,546 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-27 05:50:16,692 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:50:54,011 - baro_tests.barometer - INFO - PASS 2018-03-27 05:50:54,012 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Hugepages. 2018-03-27 05:51:46,699 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-27 05:52:23,687 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:53:00,934 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-27 05:53:00,934 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-27 05:53:00,935 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Intel RDT. 2018-03-27 05:53:54,321 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-27 05:54:19,234 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:54:21,812 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-27 05:54:24,435 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-27 05:54:27,068 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-27 05:55:20,013 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-27 05:55:45,774 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:56:23,626 - baro_tests.barometer - INFO - PASS 2018-03-27 05:56:23,628 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on OVS events. 2018-03-27 05:57:17,276 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-27 05:57:42,982 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-27 05:58:20,526 - baro_tests.barometer - INFO - PASS 2018-03-27 05:58:20,527 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on OVS stats. 2018-03-27 05:59:13,513 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-27 05:59:39,530 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-27 05:59:58,308 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-27 05:59:58,309 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-27 06:00:48,097 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-27 06:01:12,259 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-27 06:01:14,681 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-27 06:01:17,050 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-27 06:01:19,396 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-27 06:01:19,397 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-27 06:01:19,397 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-27 06:01:31,924 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-27 06:01:42,928 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-27 06:01:54,038 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-27 06:02:56,913 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-27 06:03:22,683 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 2ddd9347-826a-4de1-8c93-6499ccf7cdff... 2018-03-27 06:03:22,683 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-27 06:03:22,683 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-27 06:03:22,684 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-27 06:03:35,714 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 06:03:35,714 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522130218.6 ... 2018-03-27 06:03:35,717 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:03:35,718 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522130218.6 ... 2018-03-27 06:03:35,719 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:03:35,719 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522130218.6 ... 2018-03-27 06:03:35,719 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:03:35,722 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522130218.6 ... 2018-03-27 06:03:35,723 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:03:35,724 - baro_tests.barometer - INFO - OK 2018-03-27 06:03:35,724 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-27 06:03:47,733 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-27 06:04:00,715 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 06:04:00,716 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522130218.599, 1522130228.598), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522130218.599, 1522130228.598), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522130218.599, 1522130228.599), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522130218.599, 1522130228.599)] 2018-03-27 06:04:00,716 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1522130238.599, 1522130248.598), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1522130238.599, 1522130248.598), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1522130238.599, 1522130248.599), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1522130238.599, 1522130248.599)] 2018-03-27 06:04:00,716 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1522130218.6 - hugepages-mm-2048Kb vmpage_number-free 1522130238.6 ... 2018-03-27 06:04:00,717 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:04:00,717 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1522130218.6 - hugepages-mm-2048Kb vmpage_number-used 1522130238.6 ... 2018-03-27 06:04:00,717 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:04:00,717 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1522130218.6 - hugepages-node0-2048Kb vmpage_number-free 1522130238.6 ... 2018-03-27 06:04:00,717 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:04:00,717 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1522130218.6 - hugepages-node0-2048Kb vmpage_number-used 1522130238.6 ... 2018-03-27 06:04:00,717 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:04:00,717 - baro_tests.barometer - INFO - OK 2018-03-27 06:04:00,718 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-27 06:04:50,480 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-27 06:05:28,531 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 2ddd9347-826a-4de1-8c93-6499ccf7cdff... 2018-03-27 06:05:28,533 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-27 06:05:28,535 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-27 06:05:28,535 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-27 06:05:42,018 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 06:05:42,019 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522130338.6 ... 2018-03-27 06:05:42,019 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:05:42,019 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522130338.6 ... 2018-03-27 06:05:42,020 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:05:42,020 - baro_tests.barometer - INFO - OK 2018-03-27 06:05:42,020 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-27 06:05:54,031 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-27 06:06:06,978 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 06:06:06,979 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522130338.599, 1522130348.599), ('intel_rdt-0-2', 'ipc', 1522130338.599, 1522130348.599)] 2018-03-27 06:06:06,979 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1522130368.598, 1522130378.599), ('intel_rdt-0-2', 'ipc', 1522130368.598, 1522130378.599)] 2018-03-27 06:06:06,980 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1522130338.6 - intel_rdt-0-2 bytes-llc 1522130368.6 ... 2018-03-27 06:06:06,980 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:06:06,980 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1522130338.6 - intel_rdt-0-2 ipc 1522130368.6 ... 2018-03-27 06:06:06,980 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:06:06,980 - baro_tests.barometer - INFO - OK 2018-03-27 06:06:06,980 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-27 06:06:57,164 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-27 06:07:22,081 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 2ddd9347-826a-4de1-8c93-6499ccf7cdff... 2018-03-27 06:07:22,082 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 06:07:22,082 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-27 06:07:22,082 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-27 06:07:35,039 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-27 06:07:35,154 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-27 06:07:35,271 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-27 06:07:35,378 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-27 06:07:35,380 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-27 06:07:35,384 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-27 06:08:26,474 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-27 06:08:52,099 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 2ddd9347-826a-4de1-8c93-6499ccf7cdff... 2018-03-27 06:08:52,100 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 06:08:52,100 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-27 06:08:52,101 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-27 06:09:04,165 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 06:09:04,166 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522130556.6 ... 2018-03-27 06:09:04,166 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:09:04,167 - baro_tests.barometer - INFO - OK 2018-03-27 06:09:04,168 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-27 06:09:07,171 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-27 06:09:19,604 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 06:09:19,604 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522130556.6, 1522130557.6)] 2018-03-27 06:09:19,605 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1522130572.6, 1522130573.6)] 2018-03-27 06:09:19,607 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1522130556.6 - ovs_events-br0 gauge-link_status 1522130572.6 ... 2018-03-27 06:09:19,608 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:09:19,610 - baro_tests.barometer - INFO - OK 2018-03-27 06:09:19,613 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-27 06:10:08,382 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-27 06:10:32,569 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 2ddd9347-826a-4de1-8c93-6499ccf7cdff... 2018-03-27 06:10:32,569 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-27 06:10:32,570 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-27 06:10:32,570 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-27 06:10:45,720 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-27 06:10:45,721 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522130658.6 ... 2018-03-27 06:10:45,721 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:10:45,724 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522130658.6 ... 2018-03-27 06:10:45,726 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:10:45,729 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522130658.6 ... 2018-03-27 06:10:45,729 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:10:45,730 - baro_tests.barometer - INFO - OK 2018-03-27 06:10:45,732 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-27 06:10:48,736 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-27 06:11:02,275 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-27 06:11:02,276 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522130658.6, 1522130659.6), ('ovs_stats-br0.br0', 'if_errors', 1522130658.6, 1522130659.6), ('ovs_stats-br0.br0', 'if_packets', 1522130658.6, 1522130659.6)] 2018-03-27 06:11:02,276 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1522130675.6, 1522130676.6), ('ovs_stats-br0.br0', 'if_errors', 1522130675.6, 1522130676.6), ('ovs_stats-br0.br0', 'if_packets', 1522130675.6, 1522130676.6)] 2018-03-27 06:11:02,276 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1522130658.6 - ovs_stats-br0.br0 if_dropped 1522130675.6 ... 2018-03-27 06:11:02,277 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:11:02,277 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1522130658.6 - ovs_stats-br0.br0 if_errors 1522130675.6 ... 2018-03-27 06:11:02,277 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:11:02,277 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1522130658.6 - ovs_stats-br0.br0 if_packets 1522130675.6 ... 2018-03-27 06:11:02,277 - baro_tests.barometer - DEBUG - OK 2018-03-27 06:11:02,277 - baro_tests.barometer - INFO - OK 2018-03-27 06:11:02,277 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-27 06:11:14,453 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-27 06:11:14,454 - baro_tests.barometer - INFO - ====================================================================== 2018-03-27 06:11:14,456 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-27 06:11:14,456 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-27 06:11:14,459 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 06:11:14,459 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-27 06:11:14,461 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 06:11:14,461 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-27 06:11:14,464 - baro_tests.barometer - INFO - | IN:Hugepages | FAIL | PASS | 2018-03-27 06:11:14,464 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | FAIL | 2018-03-27 06:11:14,465 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-27 06:11:14,465 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | PASS | 2018-03-27 06:11:14,466 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | PASS | 2018-03-27 06:11:14,467 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 06:11:14,470 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-27 06:11:14,470 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-27 06:11:14,471 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-03-27 06:11:14,471 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 06:11:14,471 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-27 06:11:14,471 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-27 06:11:14,471 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-27 06:11:14,472 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-27 06:11:14,472 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-27 06:11:14,472 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-27 06:11:14,473 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-27 06:11:14,473 - baro_tests.barometer - INFO - ====================================================================== 2018-03-27 06:11:14,474 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-27 06:11:14,476 - baro_tests.barometer - ERROR - Overall Result is Fail