2018-03-24 23:27:25,420 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-24 23:27:25,420 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-24 23:27:29,890 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-24 23:27:29,890 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.6', 'overcloud-controller-1: 192.30.9.4', 'overcloud-controller-2: 192.30.9.8'] 2018-03-24 23:27:29,890 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.5', 'overcloud-novacompute-1: 192.30.9.7'] 2018-03-24 23:27:29,890 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-24 23:27:44,155 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-24 23:27:44,197 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-24 23:27:57,653 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-24 23:27:57,905 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-24 23:27:57,906 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-24 23:29:47,303 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-24 23:30:01,532 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-24 23:30:01,533 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-24 23:31:20,050 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-24 23:31:46,806 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-24 23:32:24,468 - baro_tests.barometer - INFO - PASS 2018-03-24 23:32:24,469 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Hugepages. 2018-03-24 23:33:17,598 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-24 23:33:57,616 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-24 23:34:00,191 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-24 23:34:03,002 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-24 23:34:05,784 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-24 23:34:59,738 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-24 23:35:26,004 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-24 23:35:28,569 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-24 23:35:31,214 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-24 23:35:33,747 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-24 23:36:29,147 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-24 23:36:54,640 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-24 23:36:57,297 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-24 23:36:59,909 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-24 23:37:02,555 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-03-24 23:37:56,758 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-24 23:38:22,887 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-24 23:39:00,020 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-24 23:39:00,021 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-24 23:39:00,021 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-03-24 23:39:51,563 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-24 23:40:18,575 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-24 23:40:37,429 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-24 23:40:37,430 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-24 23:41:31,790 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-24 23:41:58,372 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-24 23:42:00,622 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-24 23:42:03,735 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-24 23:42:06,216 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-24 23:42:06,217 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-24 23:42:06,217 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-24 23:42:18,893 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-24 23:42:30,170 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-24 23:42:41,186 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-24 23:43:45,041 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-24 23:44:13,188 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node f0458dbb-8d59-4842-a6ca-7a3818f5709e... 2018-03-24 23:44:13,188 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-24 23:44:13,189 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-24 23:44:13,189 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-24 23:44:26,236 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-24 23:44:26,237 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521934679.63 ... 2018-03-24 23:44:26,237 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:44:26,237 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521934679.63 ... 2018-03-24 23:44:26,237 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:44:26,237 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521934679.63 ... 2018-03-24 23:44:26,237 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:44:26,238 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521934679.63 ... 2018-03-24 23:44:26,238 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:44:26,238 - baro_tests.barometer - INFO - OK 2018-03-24 23:44:26,238 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-24 23:44:38,248 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-24 23:44:50,912 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-24 23:44:50,912 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521934679.63, 1521934689.63), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521934679.63, 1521934689.63), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521934679.63, 1521934689.63), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521934679.63, 1521934689.63)] 2018-03-24 23:44:50,913 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521934699.63, 1521934709.63), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521934699.63, 1521934709.63), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521934699.63, 1521934709.63), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521934699.63, 1521934709.63)] 2018-03-24 23:44:50,916 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521934679.63 - hugepages-mm-2048Kb vmpage_number-free 1521934699.63 ... 2018-03-24 23:44:50,917 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:44:50,918 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521934679.63 - hugepages-mm-2048Kb vmpage_number-used 1521934699.63 ... 2018-03-24 23:44:50,919 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:44:50,920 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521934679.63 - hugepages-node0-2048Kb vmpage_number-free 1521934699.63 ... 2018-03-24 23:44:50,921 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:44:50,922 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521934679.63 - hugepages-node0-2048Kb vmpage_number-used 1521934699.63 ... 2018-03-24 23:44:50,922 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:44:50,923 - baro_tests.barometer - INFO - OK 2018-03-24 23:44:50,924 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-24 23:45:45,406 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-24 23:46:26,239 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node f0458dbb-8d59-4842-a6ca-7a3818f5709e... 2018-03-24 23:46:26,240 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-24 23:46:26,240 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-24 23:46:26,240 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-24 23:46:39,930 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-24 23:46:39,931 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521934809.63 ... 2018-03-24 23:46:39,931 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:46:39,932 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521934809.63 ... 2018-03-24 23:46:39,932 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:46:39,932 - baro_tests.barometer - INFO - OK 2018-03-24 23:46:39,933 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-24 23:46:51,945 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-24 23:47:05,992 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-24 23:47:05,993 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521934809.63, 1521934819.63), ('intel_rdt-0-2', 'ipc', 1521934809.63, 1521934819.63)] 2018-03-24 23:47:05,993 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521934839.63, 1521934849.63), ('intel_rdt-0-2', 'ipc', 1521934839.63, 1521934849.63)] 2018-03-24 23:47:05,994 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521934809.63 - intel_rdt-0-2 bytes-llc 1521934839.63 ... 2018-03-24 23:47:05,994 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:47:05,994 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521934809.63 - intel_rdt-0-2 ipc 1521934839.63 ... 2018-03-24 23:47:05,994 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:47:05,994 - baro_tests.barometer - INFO - OK 2018-03-24 23:47:05,995 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-24 23:47:59,718 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-24 23:48:24,331 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node f0458dbb-8d59-4842-a6ca-7a3818f5709e... 2018-03-24 23:48:24,332 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-24 23:48:24,332 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-24 23:48:24,332 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-24 23:48:37,119 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-24 23:48:37,278 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-24 23:48:37,405 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-24 23:48:37,508 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-24 23:48:37,508 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-24 23:48:37,509 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-24 23:49:28,386 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-24 23:49:53,399 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node f0458dbb-8d59-4842-a6ca-7a3818f5709e... 2018-03-24 23:49:53,399 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-24 23:49:53,400 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-24 23:49:53,400 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-24 23:50:06,426 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-24 23:50:06,427 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521935033.63 ... 2018-03-24 23:50:06,427 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:50:06,428 - baro_tests.barometer - INFO - OK 2018-03-24 23:50:06,428 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-24 23:50:09,431 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-24 23:50:22,228 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-24 23:50:22,229 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521935033.631, 1521935034.631)] 2018-03-24 23:50:22,230 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521935049.631, 1521935050.631)] 2018-03-24 23:50:22,230 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521935033.63 - ovs_events-br0 gauge-link_status 1521935049.63 ... 2018-03-24 23:50:22,231 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:50:22,231 - baro_tests.barometer - INFO - OK 2018-03-24 23:50:22,235 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-24 23:51:14,413 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-24 23:51:40,823 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node f0458dbb-8d59-4842-a6ca-7a3818f5709e... 2018-03-24 23:51:40,823 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-24 23:51:40,823 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-24 23:51:40,824 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-24 23:51:54,187 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-24 23:51:54,188 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521935140.63 ... 2018-03-24 23:51:54,188 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:51:54,188 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521935140.63 ... 2018-03-24 23:51:54,188 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:51:54,188 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521935140.63 ... 2018-03-24 23:51:54,189 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:51:54,189 - baro_tests.barometer - INFO - OK 2018-03-24 23:51:54,189 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-24 23:51:57,192 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-24 23:52:10,734 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-24 23:52:10,735 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521935140.631, 1521935141.631), ('ovs_stats-br0.br0', 'if_errors', 1521935140.631, 1521935141.631), ('ovs_stats-br0.br0', 'if_packets', 1521935140.631, 1521935142.631)] 2018-03-24 23:52:10,735 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521935157.631, 1521935158.631), ('ovs_stats-br0.br0', 'if_errors', 1521935157.631, 1521935158.631), ('ovs_stats-br0.br0', 'if_packets', 1521935157.631, 1521935158.631)] 2018-03-24 23:52:10,735 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521935140.63 - ovs_stats-br0.br0 if_dropped 1521935157.63 ... 2018-03-24 23:52:10,735 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:52:10,735 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521935140.63 - ovs_stats-br0.br0 if_errors 1521935157.63 ... 2018-03-24 23:52:10,736 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:52:10,736 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521935140.63 - ovs_stats-br0.br0 if_packets 1521935157.63 ... 2018-03-24 23:52:10,736 - baro_tests.barometer - DEBUG - OK 2018-03-24 23:52:10,736 - baro_tests.barometer - INFO - OK 2018-03-24 23:52:10,737 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-24 23:52:23,313 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-24 23:52:36,018 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-24 23:52:36,018 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-24 23:53:47,815 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-24 23:54:12,992 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-24 23:54:50,296 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-24 23:54:50,296 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-24 23:54:50,296 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-03-24 23:55:41,430 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-24 23:56:19,370 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-24 23:56:56,432 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-24 23:56:56,432 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-24 23:56:56,433 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Intel RDT. 2018-03-24 23:57:50,407 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-24 23:58:16,638 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-24 23:58:19,240 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-24 23:58:21,965 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-24 23:58:24,521 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-24 23:59:15,640 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-24 23:59:39,595 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-25 00:00:16,834 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-25 00:00:16,835 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-25 00:00:16,835 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-03-25 00:01:10,656 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-25 00:01:35,915 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-25 00:01:38,492 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 00:01:41,156 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-25 00:01:43,806 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS stats. 2018-03-25 00:02:35,988 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-25 00:03:02,258 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-25 00:03:21,028 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-25 00:03:21,029 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-25 00:04:12,173 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-25 00:04:37,270 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-25 00:04:56,004 - baro_tests.barometer - INFO - PASS 2018-03-25 00:04:56,006 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with AODH PASSED on OVS events. 2018-03-25 00:04:56,006 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-25 00:04:56,006 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-25 00:05:09,570 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-25 00:05:20,878 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-25 00:05:31,199 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-25 00:06:33,752 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-25 00:06:58,954 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 5169bf44-2c89-4292-8457-de142443d52d... 2018-03-25 00:06:58,954 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-25 00:06:58,955 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-25 00:06:58,955 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-25 00:07:11,865 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 00:07:11,865 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521936026.48 ... 2018-03-25 00:07:11,866 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:07:11,866 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521936036.48 ... 2018-03-25 00:07:11,866 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:07:11,866 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521936036.48 ... 2018-03-25 00:07:11,867 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:07:11,867 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521936036.48 ... 2018-03-25 00:07:11,867 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:07:11,867 - baro_tests.barometer - INFO - OK 2018-03-25 00:07:11,867 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-25 00:07:23,869 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-25 00:07:36,806 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 00:07:36,806 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521936026.48, 1521936036.48), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521936036.48, 1521936046.48), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521936036.48, 1521936046.48), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521936036.48, 1521936046.48)] 2018-03-25 00:07:36,807 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521936056.48, 1521936066.48), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521936056.48, 1521936066.48), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521936056.48, 1521936066.48), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521936056.48, 1521936066.48)] 2018-03-25 00:07:36,807 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521936026.48 - hugepages-mm-2048Kb vmpage_number-free 1521936056.48 ... 2018-03-25 00:07:36,808 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:07:36,808 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521936036.48 - hugepages-mm-2048Kb vmpage_number-used 1521936056.48 ... 2018-03-25 00:07:36,808 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:07:36,808 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521936036.48 - hugepages-node0-2048Kb vmpage_number-free 1521936056.48 ... 2018-03-25 00:07:36,808 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:07:36,808 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521936036.48 - hugepages-node0-2048Kb vmpage_number-used 1521936056.48 ... 2018-03-25 00:07:36,808 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:07:36,809 - baro_tests.barometer - INFO - OK 2018-03-25 00:07:36,809 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-25 00:08:28,719 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-25 00:09:06,700 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 5169bf44-2c89-4292-8457-de142443d52d... 2018-03-25 00:09:06,700 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-25 00:09:06,700 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-25 00:09:06,700 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-25 00:09:19,930 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 00:09:19,930 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521936156.48 ... 2018-03-25 00:09:19,930 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:09:19,931 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521936156.48 ... 2018-03-25 00:09:19,931 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:09:19,931 - baro_tests.barometer - INFO - OK 2018-03-25 00:09:19,931 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-25 00:09:31,942 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-25 00:09:44,269 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 00:09:44,269 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521936156.48, 1521936166.48), ('intel_rdt-0-2', 'ipc', 1521936156.48, 1521936166.48)] 2018-03-25 00:09:44,270 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521936186.48, 1521936196.48), ('intel_rdt-0-2', 'ipc', 1521936186.48, 1521936196.48)] 2018-03-25 00:09:44,270 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521936156.48 - intel_rdt-0-2 bytes-llc 1521936186.48 ... 2018-03-25 00:09:44,270 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:09:44,270 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521936156.48 - intel_rdt-0-2 ipc 1521936186.48 ... 2018-03-25 00:09:44,270 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:09:44,270 - baro_tests.barometer - INFO - OK 2018-03-25 00:09:44,270 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-25 00:10:33,452 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-25 00:10:58,065 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 5169bf44-2c89-4292-8457-de142443d52d... 2018-03-25 00:10:58,065 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-25 00:10:58,065 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-25 00:10:58,065 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-25 00:11:09,650 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-25 00:11:09,762 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-25 00:11:09,868 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-25 00:11:09,974 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-25 00:11:09,974 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-25 00:11:09,977 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-25 00:11:59,432 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-25 00:12:24,679 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 5169bf44-2c89-4292-8457-de142443d52d... 2018-03-25 00:12:24,680 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-25 00:12:24,682 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-25 00:12:24,684 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-25 00:12:37,702 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 00:12:37,702 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521936371.48 ... 2018-03-25 00:12:37,703 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:12:37,704 - baro_tests.barometer - INFO - OK 2018-03-25 00:12:37,705 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-25 00:12:40,706 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-25 00:12:52,873 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 00:12:52,873 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521936371.481, 1521936372.481)] 2018-03-25 00:12:52,873 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521936386.481, 1521936387.481)] 2018-03-25 00:12:52,873 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521936371.48 - ovs_events-br0 gauge-link_status 1521936386.48 ... 2018-03-25 00:12:52,874 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:12:52,874 - baro_tests.barometer - INFO - OK 2018-03-25 00:12:52,875 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-25 00:13:43,682 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-25 00:14:07,493 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 5169bf44-2c89-4292-8457-de142443d52d... 2018-03-25 00:14:07,493 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-25 00:14:07,494 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-25 00:14:07,494 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-25 00:14:19,865 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 00:14:19,865 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521936473.48 ... 2018-03-25 00:14:19,865 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:14:19,865 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521936473.48 ... 2018-03-25 00:14:19,865 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:14:19,866 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521936473.48 ... 2018-03-25 00:14:19,866 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:14:19,866 - baro_tests.barometer - INFO - OK 2018-03-25 00:14:19,867 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-25 00:14:22,869 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-25 00:14:35,712 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 00:14:35,713 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521936473.481, 1521936474.481), ('ovs_stats-br0.br0', 'if_errors', 1521936473.481, 1521936474.481), ('ovs_stats-br0.br0', 'if_packets', 1521936473.481, 1521936474.481)] 2018-03-25 00:14:35,713 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521936489.481, 1521936490.481), ('ovs_stats-br0.br0', 'if_errors', 1521936489.481, 1521936490.481), ('ovs_stats-br0.br0', 'if_packets', 1521936489.481, 1521936490.481)] 2018-03-25 00:14:35,714 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521936473.48 - ovs_stats-br0.br0 if_dropped 1521936489.48 ... 2018-03-25 00:14:35,714 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:14:35,714 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521936473.48 - ovs_stats-br0.br0 if_errors 1521936489.48 ... 2018-03-25 00:14:35,714 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:14:35,714 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521936473.48 - ovs_stats-br0.br0 if_packets 1521936489.48 ... 2018-03-25 00:14:35,714 - baro_tests.barometer - DEBUG - OK 2018-03-25 00:14:35,714 - baro_tests.barometer - INFO - OK 2018-03-25 00:14:35,715 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-25 00:14:48,996 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-25 00:14:48,999 - baro_tests.barometer - INFO - ====================================================================== 2018-03-25 00:14:48,999 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-25 00:14:49,001 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-25 00:14:49,002 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 00:14:49,004 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-25 00:14:49,006 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 00:14:49,006 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-25 00:14:49,006 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | FAIL | 2018-03-25 00:14:49,007 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | FAIL | 2018-03-25 00:14:49,007 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-25 00:14:49,008 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-03-25 00:14:49,008 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | FAIL | 2018-03-25 00:14:49,008 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 00:14:49,008 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-25 00:14:49,008 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-25 00:14:49,008 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | PASS | 2018-03-25 00:14:49,008 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 00:14:49,009 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-25 00:14:49,009 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-25 00:14:49,009 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-25 00:14:49,009 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-25 00:14:49,009 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-25 00:14:49,009 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-25 00:14:49,009 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 00:14:49,009 - baro_tests.barometer - INFO - ====================================================================== 2018-03-25 00:14:49,009 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-25 00:14:49,010 - baro_tests.barometer - ERROR - Overall Result is Fail