2018-03-23 01:21:25,499 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-23 01:21:25,499 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-23 01:21:29,635 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-23 01:21:29,636 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.7'] 2018-03-23 01:21:29,636 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.10', 'overcloud-novacompute-1: 192.30.9.5'] 2018-03-23 01:21:29,636 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-23 01:21:42,324 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-23 01:21:42,367 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-23 01:21:54,121 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-23 01:21:54,401 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-23 01:21:54,402 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-23 01:22:34,933 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-23 01:22:47,372 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-23 01:22:47,372 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-23 01:24:02,397 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-23 01:24:26,923 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:25:04,094 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-23 01:25:04,095 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-23 01:25:04,095 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Hugepages. 2018-03-23 01:26:02,949 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-23 01:26:26,025 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:26:28,538 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-23 01:27:27,616 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-23 01:27:51,650 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:27:54,233 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-23 01:28:51,559 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-23 01:29:14,733 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:29:51,799 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-23 01:29:51,799 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-23 01:29:51,799 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-03-23 01:30:51,409 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-23 01:31:14,831 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:31:51,792 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-23 01:31:51,792 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-23 01:31:51,793 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-03-23 01:32:50,706 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-23 01:33:12,472 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-23 01:33:31,079 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-23 01:33:31,079 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-23 01:34:29,291 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-23 01:34:52,410 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-23 01:34:54,568 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-23 01:34:54,569 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-23 01:34:54,569 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-23 01:35:05,637 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-23 01:35:16,577 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-23 01:35:26,887 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-23 01:36:35,304 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-23 01:36:57,733 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node d34022cb-1fb9-4d60-9e5d-6fc5a6bc309a... 2018-03-23 01:36:57,733 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-23 01:36:57,733 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-23 01:36:57,734 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-23 01:37:10,013 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-23 01:37:10,014 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521768648.75 ... 2018-03-23 01:37:10,014 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:37:10,014 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521768648.75 ... 2018-03-23 01:37:10,014 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:37:10,014 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521768648.75 ... 2018-03-23 01:37:10,014 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:37:10,014 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521768648.75 ... 2018-03-23 01:37:10,014 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:37:10,015 - baro_tests.barometer - INFO - OK 2018-03-23 01:37:10,015 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-23 01:37:22,027 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-23 01:37:34,436 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-23 01:37:34,436 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521768648.752, 1521768658.752), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521768648.752, 1521768658.752), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521768648.752, 1521768658.752), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521768648.752, 1521768658.752)] 2018-03-23 01:37:34,437 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521768668.752, 1521768678.752), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521768668.752, 1521768678.752), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521768668.752, 1521768678.752), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521768668.752, 1521768678.752)] 2018-03-23 01:37:34,437 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521768648.75 - hugepages-mm-2048Kb vmpage_number-free 1521768668.75 ... 2018-03-23 01:37:34,437 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:37:34,438 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521768648.75 - hugepages-mm-2048Kb vmpage_number-used 1521768668.75 ... 2018-03-23 01:37:34,438 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:37:34,439 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521768648.75 - hugepages-node0-2048Kb vmpage_number-free 1521768668.75 ... 2018-03-23 01:37:34,439 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:37:34,439 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521768648.75 - hugepages-node0-2048Kb vmpage_number-used 1521768668.75 ... 2018-03-23 01:37:34,439 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:37:34,439 - baro_tests.barometer - INFO - OK 2018-03-23 01:37:34,439 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-23 01:38:32,215 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-23 01:38:54,281 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node d34022cb-1fb9-4d60-9e5d-6fc5a6bc309a... 2018-03-23 01:38:54,282 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-23 01:38:54,282 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-23 01:38:54,282 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-23 01:39:06,120 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-23 01:39:06,121 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521768758.75 ... 2018-03-23 01:39:06,121 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:39:06,121 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521768758.75 ... 2018-03-23 01:39:06,122 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:39:06,122 - baro_tests.barometer - INFO - OK 2018-03-23 01:39:06,123 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-23 01:39:18,132 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-23 01:39:29,460 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-23 01:39:29,460 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521768758.752, 1521768768.752), ('intel_rdt-0-2', 'ipc', 1521768758.752, 1521768768.752)] 2018-03-23 01:39:29,461 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521768788.752, 1521768798.752), ('intel_rdt-0-2', 'ipc', 1521768788.752, 1521768798.752)] 2018-03-23 01:39:29,461 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521768758.75 - intel_rdt-0-2 bytes-llc 1521768788.75 ... 2018-03-23 01:39:29,461 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:39:29,462 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521768758.75 - intel_rdt-0-2 ipc 1521768788.75 ... 2018-03-23 01:39:29,462 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:39:29,463 - baro_tests.barometer - INFO - OK 2018-03-23 01:39:29,464 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-23 01:40:23,157 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-23 01:40:44,893 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node d34022cb-1fb9-4d60-9e5d-6fc5a6bc309a... 2018-03-23 01:40:44,894 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-23 01:40:44,894 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-23 01:40:44,894 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-23 01:40:57,058 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-23 01:40:57,160 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-23 01:40:57,288 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-23 01:40:57,413 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-23 01:40:57,414 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-23 01:40:57,415 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-23 01:41:53,823 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-23 01:42:16,689 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node d34022cb-1fb9-4d60-9e5d-6fc5a6bc309a... 2018-03-23 01:42:16,690 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-23 01:42:16,690 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-23 01:42:16,690 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-23 01:42:28,634 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-23 01:42:28,635 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521768978.75 ... 2018-03-23 01:42:28,635 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:42:28,635 - baro_tests.barometer - INFO - OK 2018-03-23 01:42:28,636 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-23 01:42:31,636 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-23 01:42:42,912 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-23 01:42:42,912 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521768978.753, 1521768979.754)] 2018-03-23 01:42:42,913 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521768992.754, 1521768994.753)] 2018-03-23 01:42:42,913 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521768978.75 - ovs_events-br0 gauge-link_status 1521768992.75 ... 2018-03-23 01:42:42,913 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:42:42,914 - baro_tests.barometer - INFO - OK 2018-03-23 01:42:42,914 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-23 01:43:37,727 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-23 01:44:00,519 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node d34022cb-1fb9-4d60-9e5d-6fc5a6bc309a... 2018-03-23 01:44:00,519 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-23 01:44:00,520 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-23 01:44:00,520 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-23 01:44:11,457 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-23 01:44:11,457 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521769081.75 ... 2018-03-23 01:44:11,458 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:44:11,458 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521769081.75 ... 2018-03-23 01:44:11,458 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:44:11,458 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521769081.75 ... 2018-03-23 01:44:11,459 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:44:11,459 - baro_tests.barometer - INFO - OK 2018-03-23 01:44:11,459 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-23 01:44:14,460 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-23 01:44:26,535 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-23 01:44:26,536 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521769081.753, 1521769082.753), ('ovs_stats-br0.br0', 'if_errors', 1521769081.753, 1521769082.753), ('ovs_stats-br0.br0', 'if_packets', 1521769081.753, 1521769082.753)] 2018-03-23 01:44:26,536 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521769096.753, 1521769097.753), ('ovs_stats-br0.br0', 'if_errors', 1521769096.753, 1521769097.753), ('ovs_stats-br0.br0', 'if_packets', 1521769096.753, 1521769097.753)] 2018-03-23 01:44:26,537 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521769081.75 - ovs_stats-br0.br0 if_dropped 1521769096.75 ... 2018-03-23 01:44:26,537 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:44:26,537 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521769081.75 - ovs_stats-br0.br0 if_errors 1521769096.75 ... 2018-03-23 01:44:26,537 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:44:26,537 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521769081.75 - ovs_stats-br0.br0 if_packets 1521769096.75 ... 2018-03-23 01:44:26,537 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:44:26,537 - baro_tests.barometer - INFO - OK 2018-03-23 01:44:26,537 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-23 01:44:38,748 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-23 01:44:49,421 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-23 01:44:49,422 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-23 01:45:59,756 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-23 01:46:21,730 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:46:58,818 - baro_tests.barometer - INFO - PASS 2018-03-23 01:46:58,818 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Hugepages. 2018-03-23 01:47:57,610 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-23 01:48:20,372 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:48:57,454 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-23 01:48:57,454 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-23 01:48:57,454 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Intel RDT. 2018-03-23 01:49:54,560 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-23 01:50:15,768 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:50:18,453 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-23 01:51:15,285 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-23 01:51:36,345 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:52:13,476 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-23 01:52:13,476 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-23 01:52:13,477 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-03-23 01:53:11,375 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-23 01:53:32,900 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-23 01:54:10,228 - baro_tests.barometer - INFO - PASS 2018-03-23 01:54:10,229 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on OVS stats. 2018-03-23 01:55:05,909 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-23 01:55:28,200 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-23 01:55:46,729 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-23 01:55:46,730 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-23 01:56:43,742 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-23 01:57:04,626 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-23 01:57:06,788 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-23 01:57:06,789 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-23 01:57:06,789 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-23 01:57:17,159 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-23 01:57:28,250 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-23 01:57:38,564 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-23 01:58:42,464 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-23 01:59:04,596 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 63bb7e21-7f3e-4dab-8f75-0421f69a86f2... 2018-03-23 01:59:04,596 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-23 01:59:04,596 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-23 01:59:04,597 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-23 01:59:16,622 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-23 01:59:16,622 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521769955.83 ... 2018-03-23 01:59:16,622 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:59:16,622 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521769955.83 ... 2018-03-23 01:59:16,622 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:59:16,622 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521769955.83 ... 2018-03-23 01:59:16,622 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:59:16,623 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521769955.83 ... 2018-03-23 01:59:16,623 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:59:16,623 - baro_tests.barometer - INFO - OK 2018-03-23 01:59:16,623 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-23 01:59:28,635 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-23 01:59:40,669 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-23 01:59:40,670 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521769955.834, 1521769965.834), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521769955.834, 1521769965.834), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521769955.834, 1521769965.834), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521769955.834, 1521769965.834)] 2018-03-23 01:59:40,673 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521769985.834, 1521769995.834), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521769985.834, 1521769995.834), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521769985.834, 1521769995.834), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521769985.834, 1521769995.834)] 2018-03-23 01:59:40,674 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521769955.83 - hugepages-mm-2048Kb vmpage_number-free 1521769985.83 ... 2018-03-23 01:59:40,675 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:59:40,675 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521769955.83 - hugepages-mm-2048Kb vmpage_number-used 1521769985.83 ... 2018-03-23 01:59:40,676 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:59:40,677 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521769955.83 - hugepages-node0-2048Kb vmpage_number-free 1521769985.83 ... 2018-03-23 01:59:40,678 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:59:40,678 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521769955.83 - hugepages-node0-2048Kb vmpage_number-used 1521769985.83 ... 2018-03-23 01:59:40,679 - baro_tests.barometer - DEBUG - OK 2018-03-23 01:59:40,679 - baro_tests.barometer - INFO - OK 2018-03-23 01:59:40,679 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-23 02:00:37,169 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-23 02:00:59,453 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 63bb7e21-7f3e-4dab-8f75-0421f69a86f2... 2018-03-23 02:00:59,456 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-23 02:00:59,457 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-23 02:00:59,457 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-23 02:01:10,970 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-23 02:01:10,970 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521770075.83 ... 2018-03-23 02:01:10,970 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:01:10,970 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521770075.83 ... 2018-03-23 02:01:10,970 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:01:10,970 - baro_tests.barometer - INFO - OK 2018-03-23 02:01:10,971 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-23 02:01:22,981 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-23 02:01:33,435 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-23 02:01:33,436 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521770075.834, 1521770085.834), ('intel_rdt-0-2', 'ipc', 1521770075.834, 1521770085.834)] 2018-03-23 02:01:33,436 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521770095.834, 1521770105.834), ('intel_rdt-0-2', 'ipc', 1521770095.834, 1521770105.834)] 2018-03-23 02:01:33,436 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521770075.83 - intel_rdt-0-2 bytes-llc 1521770095.83 ... 2018-03-23 02:01:33,437 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:01:33,437 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521770075.83 - intel_rdt-0-2 ipc 1521770095.83 ... 2018-03-23 02:01:33,437 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:01:33,437 - baro_tests.barometer - INFO - OK 2018-03-23 02:01:33,438 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-23 02:02:30,203 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-23 02:02:52,784 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 63bb7e21-7f3e-4dab-8f75-0421f69a86f2... 2018-03-23 02:02:52,784 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-23 02:02:52,785 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-23 02:02:52,785 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-23 02:03:03,997 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-23 02:03:04,126 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-23 02:03:04,243 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-23 02:03:04,348 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-23 02:03:04,349 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-23 02:03:04,351 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-23 02:03:58,164 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-23 02:04:18,987 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 63bb7e21-7f3e-4dab-8f75-0421f69a86f2... 2018-03-23 02:04:18,988 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-23 02:04:18,989 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-23 02:04:18,990 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-23 02:04:30,054 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-23 02:04:30,054 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521770285.84 ... 2018-03-23 02:04:30,054 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:04:30,054 - baro_tests.barometer - INFO - OK 2018-03-23 02:04:30,054 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-23 02:04:33,057 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-23 02:04:43,544 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-23 02:04:43,544 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521770285.836, 1521770286.835)] 2018-03-23 02:04:43,545 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521770298.835, 1521770299.835)] 2018-03-23 02:04:43,545 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521770285.84 - ovs_events-br0 gauge-link_status 1521770298.84 ... 2018-03-23 02:04:43,545 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:04:43,546 - baro_tests.barometer - INFO - OK 2018-03-23 02:04:43,546 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-23 02:05:36,579 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-23 02:05:57,070 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 63bb7e21-7f3e-4dab-8f75-0421f69a86f2... 2018-03-23 02:05:57,071 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-23 02:05:57,071 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-23 02:05:57,072 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-23 02:06:08,791 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-23 02:06:08,792 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521770383.84 ... 2018-03-23 02:06:08,792 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:06:08,792 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521770383.84 ... 2018-03-23 02:06:08,793 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:06:08,793 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521770383.84 ... 2018-03-23 02:06:08,793 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:06:08,793 - baro_tests.barometer - INFO - OK 2018-03-23 02:06:08,793 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-23 02:06:11,796 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-23 02:06:23,367 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-23 02:06:23,368 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521770383.835, 1521770384.835), ('ovs_stats-br0.br0', 'if_errors', 1521770383.835, 1521770384.835), ('ovs_stats-br0.br0', 'if_packets', 1521770383.835, 1521770384.835)] 2018-03-23 02:06:23,368 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521770398.835, 1521770399.835), ('ovs_stats-br0.br0', 'if_errors', 1521770398.835, 1521770399.835), ('ovs_stats-br0.br0', 'if_packets', 1521770398.835, 1521770399.835)] 2018-03-23 02:06:23,369 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521770383.84 - ovs_stats-br0.br0 if_dropped 1521770398.84 ... 2018-03-23 02:06:23,369 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:06:23,369 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521770383.84 - ovs_stats-br0.br0 if_errors 1521770398.84 ... 2018-03-23 02:06:23,370 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:06:23,370 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521770383.84 - ovs_stats-br0.br0 if_packets 1521770398.84 ... 2018-03-23 02:06:23,370 - baro_tests.barometer - DEBUG - OK 2018-03-23 02:06:23,370 - baro_tests.barometer - INFO - OK 2018-03-23 02:06:23,371 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-23 02:06:34,651 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-23 02:06:34,651 - baro_tests.barometer - INFO - ====================================================================== 2018-03-23 02:06:34,652 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-23 02:06:34,652 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-23 02:06:34,653 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-23 02:06:34,653 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-23 02:06:34,653 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-23 02:06:34,653 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-23 02:06:34,653 - baro_tests.barometer - INFO - | IN:Hugepages | FAIL | PASS | 2018-03-23 02:06:34,653 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | FAIL | 2018-03-23 02:06:34,653 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-23 02:06:34,654 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-03-23 02:06:34,654 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | PASS | 2018-03-23 02:06:34,654 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-23 02:06:34,654 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-23 02:06:34,654 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-23 02:06:34,654 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-03-23 02:06:34,655 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-23 02:06:34,655 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-23 02:06:34,655 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-23 02:06:34,655 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-23 02:06:34,655 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-23 02:06:34,655 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-23 02:06:34,655 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-23 02:06:34,655 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-23 02:06:34,656 - baro_tests.barometer - INFO - ====================================================================== 2018-03-23 02:06:34,656 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-23 02:06:34,657 - baro_tests.barometer - ERROR - Overall Result is Fail