2018-03-22 18:58:20,578 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-22 18:58:20,578 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-22 18:58:24,602 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-22 18:58:24,602 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.7', 'overcloud-controller-1: 192.30.9.6', 'overcloud-controller-2: 192.30.9.9'] 2018-03-22 18:58:24,602 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.8', 'overcloud-novacompute-1: 192.30.9.3'] 2018-03-22 18:58:24,602 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-22 18:58:37,570 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-22 18:58:37,612 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-22 18:58:51,473 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-22 18:58:51,725 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-22 18:58:51,725 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-22 19:00:40,821 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-22 19:00:53,679 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-22 19:00:53,679 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-22 19:02:14,836 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-22 19:02:40,968 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:03:18,250 - baro_tests.barometer - INFO - PASS 2018-03-22 19:03:18,250 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Hugepages. 2018-03-22 19:04:26,224 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-22 19:04:53,132 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:05:30,786 - baro_tests.barometer - INFO - PASS 2018-03-22 19:05:30,786 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Intel RDT. 2018-03-22 19:06:36,946 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-22 19:07:02,291 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:07:05,262 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-22 19:07:08,414 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-22 19:07:11,696 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-22 19:08:17,346 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-22 19:08:43,085 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:08:45,703 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-22 19:08:48,306 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-22 19:08:51,036 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-03-22 19:09:56,110 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-22 19:10:23,481 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:11:00,782 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-22 19:11:00,783 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-22 19:11:00,783 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-03-22 19:12:06,096 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-22 19:12:32,269 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-22 19:12:51,684 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-22 19:12:51,685 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-22 19:13:56,603 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-22 19:14:24,487 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-22 19:14:26,717 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-22 19:14:29,586 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-22 19:14:32,796 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-22 19:14:32,797 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-22 19:14:32,797 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-22 19:14:46,044 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-22 19:14:57,487 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-22 19:15:08,707 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-22 19:16:25,686 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-22 19:16:50,938 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 1f1b8349-ffb8-44de-9662-c20618e832a5... 2018-03-22 19:16:50,938 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-22 19:16:50,938 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-22 19:16:50,939 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-22 19:17:03,806 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-22 19:17:03,806 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521745841.24 ... 2018-03-22 19:17:03,806 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:17:03,807 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521745841.24 ... 2018-03-22 19:17:03,807 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:17:03,807 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521745841.24 ... 2018-03-22 19:17:03,807 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:17:03,808 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521745841.24 ... 2018-03-22 19:17:03,808 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:17:03,808 - baro_tests.barometer - INFO - OK 2018-03-22 19:17:03,808 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-22 19:17:15,811 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-22 19:17:29,506 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-22 19:17:29,506 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521745841.243, 1521745851.243), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521745841.243, 1521745851.243), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521745841.243, 1521745851.243), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521745841.243, 1521745851.243)] 2018-03-22 19:17:29,507 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521745871.243, 1521745881.243), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521745871.243, 1521745881.243), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521745871.243, 1521745881.243), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521745871.243, 1521745881.243)] 2018-03-22 19:17:29,508 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521745841.24 - hugepages-mm-2048Kb vmpage_number-free 1521745871.24 ... 2018-03-22 19:17:29,508 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:17:29,508 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521745841.24 - hugepages-mm-2048Kb vmpage_number-used 1521745871.24 ... 2018-03-22 19:17:29,508 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:17:29,508 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521745841.24 - hugepages-node0-2048Kb vmpage_number-free 1521745871.24 ... 2018-03-22 19:17:29,508 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:17:29,508 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521745841.24 - hugepages-node0-2048Kb vmpage_number-used 1521745871.24 ... 2018-03-22 19:17:29,508 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:17:29,508 - baro_tests.barometer - INFO - OK 2018-03-22 19:17:29,508 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-22 19:18:34,571 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-22 19:19:01,360 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 1f1b8349-ffb8-44de-9662-c20618e832a5... 2018-03-22 19:19:01,361 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-22 19:19:01,361 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-22 19:19:01,361 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-22 19:19:14,687 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-22 19:19:14,688 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521745971.24 ... 2018-03-22 19:19:14,688 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:19:14,688 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521745971.24 ... 2018-03-22 19:19:14,688 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:19:14,689 - baro_tests.barometer - INFO - OK 2018-03-22 19:19:14,689 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-22 19:19:26,698 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-22 19:19:39,883 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-22 19:19:39,883 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521745971.243, 1521745981.243), ('intel_rdt-0-2', 'ipc', 1521745971.243, 1521745981.243)] 2018-03-22 19:19:39,884 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521746001.243, 1521746011.243), ('intel_rdt-0-2', 'ipc', 1521746001.243, 1521746011.243)] 2018-03-22 19:19:39,884 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521745971.24 - intel_rdt-0-2 bytes-llc 1521746001.24 ... 2018-03-22 19:19:39,884 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:19:39,885 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521745971.24 - intel_rdt-0-2 ipc 1521746001.24 ... 2018-03-22 19:19:39,885 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:19:39,885 - baro_tests.barometer - INFO - OK 2018-03-22 19:19:39,885 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-22 19:20:44,895 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-22 19:21:10,447 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 1f1b8349-ffb8-44de-9662-c20618e832a5... 2018-03-22 19:21:10,447 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-22 19:21:10,447 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-22 19:21:10,448 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-22 19:21:23,330 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-22 19:21:23,460 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-22 19:21:23,598 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-22 19:21:23,726 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-22 19:21:23,727 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-22 19:21:23,728 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-22 19:22:28,652 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-22 19:22:54,517 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 1f1b8349-ffb8-44de-9662-c20618e832a5... 2018-03-22 19:22:54,517 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-22 19:22:54,518 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-22 19:22:54,519 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-22 19:23:07,564 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-22 19:23:07,565 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521746218.24 ... 2018-03-22 19:23:07,565 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:23:07,565 - baro_tests.barometer - INFO - OK 2018-03-22 19:23:07,565 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-22 19:23:10,567 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-22 19:23:23,509 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-22 19:23:23,509 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521746218.244, 1521746219.244)] 2018-03-22 19:23:23,510 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521746234.244, 1521746235.244)] 2018-03-22 19:23:23,510 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521746218.24 - ovs_events-br0 gauge-link_status 1521746234.24 ... 2018-03-22 19:23:23,510 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:23:23,510 - baro_tests.barometer - INFO - OK 2018-03-22 19:23:23,510 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-22 19:24:27,552 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-22 19:24:54,132 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 1f1b8349-ffb8-44de-9662-c20618e832a5... 2018-03-22 19:24:54,132 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-22 19:24:54,132 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-22 19:24:54,133 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-22 19:25:07,796 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-22 19:25:07,797 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521746338.24 ... 2018-03-22 19:25:07,797 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:25:07,798 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521746338.24 ... 2018-03-22 19:25:07,799 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:25:07,800 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521746339.24 ... 2018-03-22 19:25:07,801 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:25:07,801 - baro_tests.barometer - INFO - OK 2018-03-22 19:25:07,801 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-22 19:25:10,805 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-22 19:25:24,039 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-22 19:25:24,040 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521746338.244, 1521746339.244), ('ovs_stats-br0.br0', 'if_errors', 1521746338.244, 1521746340.244), ('ovs_stats-br0.br0', 'if_packets', 1521746339.244, 1521746340.244)] 2018-03-22 19:25:24,040 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521746355.244, 1521746356.244), ('ovs_stats-br0.br0', 'if_errors', 1521746355.244, 1521746356.244), ('ovs_stats-br0.br0', 'if_packets', 1521746355.244, 1521746356.244)] 2018-03-22 19:25:24,040 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521746338.24 - ovs_stats-br0.br0 if_dropped 1521746355.24 ... 2018-03-22 19:25:24,040 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:25:24,040 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521746338.24 - ovs_stats-br0.br0 if_errors 1521746355.24 ... 2018-03-22 19:25:24,041 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:25:24,041 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521746339.24 - ovs_stats-br0.br0 if_packets 1521746355.24 ... 2018-03-22 19:25:24,041 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:25:24,041 - baro_tests.barometer - INFO - OK 2018-03-22 19:25:24,041 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-22 19:25:37,088 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-22 19:25:49,638 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-22 19:25:49,638 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-22 19:27:07,237 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-22 19:27:32,204 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:28:09,523 - baro_tests.barometer - INFO - PASS 2018-03-22 19:28:09,524 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Hugepages. 2018-03-22 19:29:12,589 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-22 19:29:39,839 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:30:17,118 - baro_tests.barometer - INFO - PASS 2018-03-22 19:30:17,119 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Intel RDT. 2018-03-22 19:31:21,281 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-22 19:31:47,192 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:31:49,905 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-22 19:31:52,634 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-1 2018-03-22 19:31:55,235 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-22 19:32:58,667 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-22 19:33:24,198 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:34:01,407 - baro_tests.barometer - INFO - PASS 2018-03-22 19:34:01,408 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on OVS events. 2018-03-22 19:35:04,110 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-22 19:35:30,843 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-2 2018-03-22 19:36:08,391 - baro_tests.barometer - INFO - PASS 2018-03-22 19:36:08,391 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on OVS stats. 2018-03-22 19:37:13,685 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-22 19:37:40,084 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-22 19:37:58,664 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-22 19:37:58,665 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-22 19:39:02,773 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-22 19:39:27,010 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-2 2018-03-22 19:39:29,221 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-22 19:39:31,449 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-1 2018-03-22 19:39:33,637 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-22 19:39:33,639 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-22 19:39:33,639 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-22 19:39:46,559 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-22 19:39:57,909 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-22 19:40:08,219 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-22 19:41:21,146 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-22 19:41:47,240 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node c7f3e941-b533-4b97-9763-75cfe8647e20... 2018-03-22 19:41:47,242 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-22 19:41:47,244 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-22 19:41:47,244 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-22 19:42:00,264 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-22 19:42:00,264 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521747325.86 ... 2018-03-22 19:42:00,265 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:42:00,268 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521747325.86 ... 2018-03-22 19:42:00,268 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:42:00,269 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521747325.86 ... 2018-03-22 19:42:00,270 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:42:00,270 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521747325.86 ... 2018-03-22 19:42:00,271 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:42:00,272 - baro_tests.barometer - INFO - OK 2018-03-22 19:42:00,274 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-22 19:42:12,286 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-22 19:42:25,727 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-22 19:42:25,727 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521747325.864, 1521747335.864), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521747325.864, 1521747335.864), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521747325.864, 1521747335.864), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521747325.864, 1521747335.864)] 2018-03-22 19:42:25,727 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521747345.864, 1521747355.864), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521747345.864, 1521747355.864), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521747345.865, 1521747355.864), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521747345.865, 1521747355.864)] 2018-03-22 19:42:25,728 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521747325.86 - hugepages-mm-2048Kb vmpage_number-free 1521747345.86 ... 2018-03-22 19:42:25,728 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:42:25,728 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521747325.86 - hugepages-mm-2048Kb vmpage_number-used 1521747345.86 ... 2018-03-22 19:42:25,729 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:42:25,729 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521747325.86 - hugepages-node0-2048Kb vmpage_number-free 1521747345.87 ... 2018-03-22 19:42:25,729 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:42:25,729 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521747325.86 - hugepages-node0-2048Kb vmpage_number-used 1521747345.87 ... 2018-03-22 19:42:25,729 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:42:25,729 - baro_tests.barometer - INFO - OK 2018-03-22 19:42:25,730 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-22 19:43:27,542 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-22 19:43:52,102 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node c7f3e941-b533-4b97-9763-75cfe8647e20... 2018-03-22 19:43:52,102 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-22 19:43:52,103 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-22 19:43:52,104 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-22 19:44:05,249 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-22 19:44:05,250 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521747445.86 ... 2018-03-22 19:44:05,250 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:44:05,251 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521747445.86 ... 2018-03-22 19:44:05,252 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:44:05,253 - baro_tests.barometer - INFO - OK 2018-03-22 19:44:05,258 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-22 19:44:17,268 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-22 19:44:29,821 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-22 19:44:29,821 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521747445.864, 1521747455.864), ('intel_rdt-0-2', 'ipc', 1521747445.864, 1521747455.864)] 2018-03-22 19:44:29,821 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521747475.864, 1521747485.864), ('intel_rdt-0-2', 'ipc', 1521747475.864, 1521747485.864)] 2018-03-22 19:44:29,822 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521747445.86 - intel_rdt-0-2 bytes-llc 1521747475.86 ... 2018-03-22 19:44:29,822 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:44:29,822 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521747445.86 - intel_rdt-0-2 ipc 1521747475.86 ... 2018-03-22 19:44:29,823 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:44:29,823 - baro_tests.barometer - INFO - OK 2018-03-22 19:44:29,824 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-22 19:45:32,692 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-22 19:45:57,992 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node c7f3e941-b533-4b97-9763-75cfe8647e20... 2018-03-22 19:45:57,994 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-22 19:45:57,995 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-22 19:45:57,998 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-22 19:46:10,897 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-22 19:46:11,020 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-22 19:46:11,164 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-22 19:46:11,297 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-22 19:46:11,299 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-22 19:46:11,300 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-22 19:47:14,146 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-22 19:47:39,001 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node c7f3e941-b533-4b97-9763-75cfe8647e20... 2018-03-22 19:47:39,002 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-22 19:47:39,002 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-22 19:47:39,002 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-22 19:47:51,518 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-22 19:47:51,519 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521747687.87 ... 2018-03-22 19:47:51,519 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:47:51,519 - baro_tests.barometer - INFO - OK 2018-03-22 19:47:51,519 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-22 19:47:54,523 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-22 19:48:06,998 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-22 19:48:06,999 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521747687.865, 1521747688.865)] 2018-03-22 19:48:06,999 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521747702.865, 1521747703.865)] 2018-03-22 19:48:06,999 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521747687.87 - ovs_events-br0 gauge-link_status 1521747702.87 ... 2018-03-22 19:48:07,000 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:48:07,000 - baro_tests.barometer - INFO - OK 2018-03-22 19:48:07,000 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-22 19:49:08,043 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-22 19:49:31,639 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node c7f3e941-b533-4b97-9763-75cfe8647e20... 2018-03-22 19:49:31,640 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-22 19:49:31,641 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-22 19:49:31,641 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-22 19:49:44,057 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-22 19:49:44,058 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521747799.87 ... 2018-03-22 19:49:44,059 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:49:44,059 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521747799.87 ... 2018-03-22 19:49:44,061 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:49:44,061 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521747799.87 ... 2018-03-22 19:49:44,063 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:49:44,063 - baro_tests.barometer - INFO - OK 2018-03-22 19:49:44,063 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-22 19:49:47,066 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-22 19:50:00,033 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-22 19:50:00,033 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521747799.865, 1521747800.865), ('ovs_stats-br0.br0', 'if_errors', 1521747799.865, 1521747800.865), ('ovs_stats-br0.br0', 'if_packets', 1521747799.865, 1521747800.865)] 2018-03-22 19:50:00,033 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521747815.865, 1521747816.865), ('ovs_stats-br0.br0', 'if_errors', 1521747815.865, 1521747816.865), ('ovs_stats-br0.br0', 'if_packets', 1521747815.865, 1521747816.865)] 2018-03-22 19:50:00,034 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521747799.87 - ovs_stats-br0.br0 if_dropped 1521747815.87 ... 2018-03-22 19:50:00,034 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:50:00,034 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521747799.87 - ovs_stats-br0.br0 if_errors 1521747815.87 ... 2018-03-22 19:50:00,034 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:50:00,035 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521747799.87 - ovs_stats-br0.br0 if_packets 1521747815.87 ... 2018-03-22 19:50:00,035 - baro_tests.barometer - DEBUG - OK 2018-03-22 19:50:00,035 - baro_tests.barometer - INFO - OK 2018-03-22 19:50:00,035 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-22 19:50:12,665 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-22 19:50:12,665 - baro_tests.barometer - INFO - ====================================================================== 2018-03-22 19:50:12,666 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-22 19:50:12,666 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-22 19:50:12,667 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-22 19:50:12,667 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-22 19:50:12,667 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-22 19:50:12,667 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-22 19:50:12,667 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-22 19:50:12,667 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-22 19:50:12,667 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | PASS | 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | PASS | 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-22 19:50:12,668 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-22 19:50:12,669 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-22 19:50:12,669 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-22 19:50:12,669 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-22 19:50:12,669 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-22 19:50:12,669 - baro_tests.barometer - INFO - ====================================================================== 2018-03-22 19:50:12,669 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-22 19:50:12,669 - baro_tests.barometer - ERROR - Overall Result is Fail