2018-03-14 15:47:29,537 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-14 15:47:29,537 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-14 15:47:34,470 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-14 15:47:34,470 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.3'] 2018-03-14 15:47:34,470 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.9', 'overcloud-novacompute-1: 192.30.9.6'] 2018-03-14 15:47:34,470 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-14 15:47:47,273 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-14 15:47:47,317 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-14 15:47:59,754 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-14 15:48:00,011 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-14 15:48:00,011 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-14 15:48:39,452 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-14 15:48:51,741 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-14 15:48:51,742 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-14 15:49:58,883 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-14 15:50:23,120 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 15:50:42,051 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-14 15:50:42,052 - baro_tests.barometer - INFO - Data not updated after 12 seconds 2018-03-14 15:50:42,052 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Hugepages. 2018-03-14 15:51:30,703 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-14 15:51:54,016 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 15:52:12,913 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-14 15:52:12,914 - baro_tests.barometer - INFO - Data not updated after 12 seconds 2018-03-14 15:52:12,914 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-14 15:52:58,762 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-14 15:53:22,545 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 15:53:25,183 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-14 15:54:11,978 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-14 15:54:36,161 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 15:54:46,067 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-14 15:54:46,068 - baro_tests.barometer - INFO - Data not updated after 3 seconds 2018-03-14 15:54:46,068 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-03-14 15:55:32,128 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-14 15:55:54,609 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 15:56:04,453 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-14 15:56:04,454 - baro_tests.barometer - INFO - Data not updated after 3 seconds 2018-03-14 15:56:04,454 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-03-14 15:56:49,891 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-14 15:57:13,414 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-14 15:57:32,035 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-14 15:57:32,035 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-14 15:58:20,871 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-14 15:58:44,416 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-14 15:58:46,597 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-14 15:58:46,598 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-14 15:58:46,599 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-14 15:58:58,413 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-14 15:59:09,307 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-14 15:59:19,595 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-14 16:00:16,341 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-14 16:00:39,210 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 62937604-0057-4a59-b341-02d3406138d9... 2018-03-14 16:00:39,210 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-14 16:00:39,211 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb', 'hugepages-node1-2048Kb'] 2018-03-14 16:00:39,211 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-14 16:00:51,063 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-14 16:00:51,063 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521042880.02 ... 2018-03-14 16:00:51,063 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:00:51,064 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521042880.02 ... 2018-03-14 16:00:51,064 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:00:51,064 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521042880.02 ... 2018-03-14 16:00:51,064 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:00:51,064 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521042880.02 ... 2018-03-14 16:00:51,064 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:00:51,064 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-free 1521042880.02 ... 2018-03-14 16:00:51,065 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:00:51,065 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-used 1521042880.02 ... 2018-03-14 16:00:51,065 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:00:51,065 - baro_tests.barometer - INFO - OK 2018-03-14 16:00:51,065 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-14 16:01:03,073 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-14 16:01:16,391 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-14 16:01:16,391 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521042880.019, 1521042890.019), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521042880.019, 1521042890.019), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521042880.019, 1521042890.019), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521042880.019, 1521042890.019), ('hugepages-node1-2048Kb', 'vmpage_number-free', 1521042880.02, 1521042890.019), ('hugepages-node1-2048Kb', 'vmpage_number-used', 1521042880.02, 1521042890.019)] 2018-03-14 16:01:16,392 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521042900.019, 1521042910.019), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521042900.019, 1521042910.019), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521042900.019, 1521042910.019), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521042900.019, 1521042910.019), ('hugepages-node1-2048Kb', 'vmpage_number-free', 1521042900.02, 1521042910.02), ('hugepages-node1-2048Kb', 'vmpage_number-used', 1521042900.02, 1521042910.02)] 2018-03-14 16:01:16,392 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521042880.02 - hugepages-mm-2048Kb vmpage_number-free 1521042900.02 ... 2018-03-14 16:01:16,393 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:01:16,393 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521042880.02 - hugepages-mm-2048Kb vmpage_number-used 1521042900.02 ... 2018-03-14 16:01:16,393 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:01:16,393 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521042880.02 - hugepages-node0-2048Kb vmpage_number-free 1521042900.02 ... 2018-03-14 16:01:16,394 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:01:16,394 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521042880.02 - hugepages-node0-2048Kb vmpage_number-used 1521042900.02 ... 2018-03-14 16:01:16,394 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:01:16,395 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-free 1521042880.02 - hugepages-node1-2048Kb vmpage_number-free 1521042900.02 ... 2018-03-14 16:01:16,396 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:01:16,396 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-used 1521042880.02 - hugepages-node1-2048Kb vmpage_number-used 1521042900.02 ... 2018-03-14 16:01:16,396 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:01:16,396 - baro_tests.barometer - INFO - OK 2018-03-14 16:01:16,396 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-14 16:02:03,126 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-14 16:02:27,201 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 62937604-0057-4a59-b341-02d3406138d9... 2018-03-14 16:02:27,201 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-14 16:02:27,201 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-14 16:02:27,201 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-14 16:02:38,661 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-14 16:02:38,661 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521042980.02 ... 2018-03-14 16:02:38,661 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:02:38,662 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521042980.02 ... 2018-03-14 16:02:38,662 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:02:38,662 - baro_tests.barometer - INFO - OK 2018-03-14 16:02:38,663 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-14 16:02:50,669 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-14 16:03:01,844 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-14 16:03:01,844 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521042980.019, 1521042990.019), ('intel_rdt-0-2', 'ipc', 1521042980.019, 1521042990.019)] 2018-03-14 16:03:01,845 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521043010.019, 1521043020.019), ('intel_rdt-0-2', 'ipc', 1521043010.019, 1521043020.019)] 2018-03-14 16:03:01,845 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521042980.02 - intel_rdt-0-2 bytes-llc 1521043010.02 ... 2018-03-14 16:03:01,845 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:03:01,846 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521042980.02 - intel_rdt-0-2 ipc 1521043010.02 ... 2018-03-14 16:03:01,846 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:03:01,846 - baro_tests.barometer - INFO - OK 2018-03-14 16:03:01,846 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-14 16:03:46,896 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-14 16:04:08,490 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 62937604-0057-4a59-b341-02d3406138d9... 2018-03-14 16:04:08,490 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-14 16:04:08,491 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-14 16:04:08,491 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-14 16:04:19,803 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-14 16:04:19,937 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-14 16:04:20,064 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-14 16:04:20,166 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-14 16:04:20,167 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-14 16:04:20,167 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-14 16:05:06,852 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-14 16:05:30,298 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 62937604-0057-4a59-b341-02d3406138d9... 2018-03-14 16:05:30,298 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-14 16:05:30,299 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-14 16:05:30,299 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-14 16:05:41,546 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-14 16:05:41,546 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521043180.02 ... 2018-03-14 16:05:41,547 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:05:41,547 - baro_tests.barometer - INFO - OK 2018-03-14 16:05:41,548 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-14 16:05:44,551 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-14 16:05:56,376 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-14 16:05:56,376 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521043180.02, 1521043181.02)] 2018-03-14 16:05:56,377 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521043195.02, 1521043196.021)] 2018-03-14 16:05:56,377 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521043180.02 - ovs_events-br0 gauge-link_status 1521043195.02 ... 2018-03-14 16:05:56,377 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:05:56,377 - baro_tests.barometer - INFO - OK 2018-03-14 16:05:56,378 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-14 16:06:43,597 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-14 16:07:04,818 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 62937604-0057-4a59-b341-02d3406138d9... 2018-03-14 16:07:04,818 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-14 16:07:04,819 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-14 16:07:04,819 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-14 16:07:15,785 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-14 16:07:15,786 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521043274.02 ... 2018-03-14 16:07:15,786 - baro_tests.barometer - ERROR - Time of last two entries differ by 2.0, but interval is 1 2018-03-14 16:07:15,786 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on OVS stats. 2018-03-14 16:07:26,461 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-14 16:07:38,652 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-14 16:07:38,653 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-14 16:08:45,866 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-14 16:09:07,665 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 16:09:26,633 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-14 16:09:26,634 - baro_tests.barometer - INFO - Data not updated after 12 seconds 2018-03-14 16:09:26,634 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-03-14 16:10:14,284 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-14 16:10:37,553 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 16:10:56,456 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-14 16:10:56,457 - baro_tests.barometer - INFO - Data not updated after 12 seconds 2018-03-14 16:10:56,457 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Intel RDT. 2018-03-14 16:11:42,769 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-14 16:12:05,407 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 16:12:07,982 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-14 16:12:51,382 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-14 16:13:13,299 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 16:13:23,062 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-14 16:13:23,063 - baro_tests.barometer - INFO - Data not updated after 3 seconds 2018-03-14 16:13:23,063 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-03-14 16:14:10,413 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-14 16:14:32,627 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-14 16:14:42,590 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-14 16:14:42,590 - baro_tests.barometer - INFO - Data not updated after 3 seconds 2018-03-14 16:14:42,590 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS stats. 2018-03-14 16:15:27,034 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-14 16:15:49,402 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-14 16:16:07,868 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-14 16:16:07,868 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-14 16:16:50,419 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-14 16:17:12,749 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-14 16:17:14,946 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-14 16:17:14,947 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-14 16:17:14,947 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-14 16:17:26,639 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-14 16:17:37,850 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-14 16:17:48,135 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-14 16:18:44,594 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-14 16:19:07,449 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 451d003d-53c8-4776-bd32-30d35205b595... 2018-03-14 16:19:07,450 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-14 16:19:07,452 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb', 'hugepages-node1-2048Kb'] 2018-03-14 16:19:07,456 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-14 16:19:18,689 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-14 16:19:18,690 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521043964.77 ... 2018-03-14 16:19:18,690 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:19:18,690 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521043964.77 ... 2018-03-14 16:19:18,690 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:19:18,690 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521043964.77 ... 2018-03-14 16:19:18,691 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:19:18,691 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521043964.77 ... 2018-03-14 16:19:18,691 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:19:18,691 - baro_tests.barometer - DEBUG - hugepages-node1-2048Kb vmpage_number-free 1521043964.77 ... 2018-03-14 16:19:18,691 - baro_tests.barometer - ERROR - Time of last two entries differ by 20.0, but interval is 10 2018-03-14 16:19:18,691 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Hugepages. 2018-03-14 16:20:02,219 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-14 16:20:24,224 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 451d003d-53c8-4776-bd32-30d35205b595... 2018-03-14 16:20:24,224 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-14 16:20:24,224 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-14 16:20:24,225 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-14 16:20:35,068 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-14 16:20:35,068 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521044044.77 ... 2018-03-14 16:20:35,068 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:20:35,068 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521044044.77 ... 2018-03-14 16:20:35,068 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:20:35,068 - baro_tests.barometer - INFO - OK 2018-03-14 16:20:35,069 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-14 16:20:47,069 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-14 16:20:58,162 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-14 16:20:58,162 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521044044.773, 1521044054.773), ('intel_rdt-0-2', 'ipc', 1521044044.773, 1521044054.773)] 2018-03-14 16:20:58,162 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521044064.773, 1521044074.773), ('intel_rdt-0-2', 'ipc', 1521044064.773, 1521044074.773)] 2018-03-14 16:20:58,162 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521044044.77 - intel_rdt-0-2 bytes-llc 1521044064.77 ... 2018-03-14 16:20:58,163 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:20:58,163 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521044044.77 - intel_rdt-0-2 ipc 1521044064.77 ... 2018-03-14 16:20:58,163 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:20:58,163 - baro_tests.barometer - INFO - OK 2018-03-14 16:20:58,163 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-14 16:21:42,447 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-14 16:22:04,273 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 451d003d-53c8-4776-bd32-30d35205b595... 2018-03-14 16:22:04,273 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-14 16:22:04,274 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-14 16:22:04,274 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-14 16:22:15,472 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-14 16:22:15,584 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-14 16:22:15,693 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-14 16:22:15,820 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-14 16:22:15,820 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-14 16:22:15,821 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-14 16:23:01,315 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-14 16:23:23,422 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 451d003d-53c8-4776-bd32-30d35205b595... 2018-03-14 16:23:23,422 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-14 16:23:23,422 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-14 16:23:23,423 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-14 16:23:35,091 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-14 16:23:35,091 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521044239.77 ... 2018-03-14 16:23:35,091 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:23:35,091 - baro_tests.barometer - INFO - OK 2018-03-14 16:23:35,092 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-14 16:23:38,095 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-14 16:23:48,732 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-14 16:23:48,733 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521044239.774, 1521044240.774)] 2018-03-14 16:23:48,733 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521044253.774, 1521044254.774)] 2018-03-14 16:23:48,733 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521044239.77 - ovs_events-br0 gauge-link_status 1521044253.77 ... 2018-03-14 16:23:48,734 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:23:48,734 - baro_tests.barometer - INFO - OK 2018-03-14 16:23:48,734 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-14 16:24:33,683 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-14 16:24:56,538 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 451d003d-53c8-4776-bd32-30d35205b595... 2018-03-14 16:24:56,538 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-14 16:24:56,539 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-14 16:24:56,539 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-14 16:25:08,102 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-14 16:25:08,102 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521044332.77 ... 2018-03-14 16:25:08,102 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:25:08,102 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521044332.77 ... 2018-03-14 16:25:08,102 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:25:08,103 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521044332.77 ... 2018-03-14 16:25:08,103 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:25:08,103 - baro_tests.barometer - INFO - OK 2018-03-14 16:25:08,103 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-14 16:25:11,105 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-14 16:25:23,231 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-14 16:25:23,231 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521044332.774, 1521044333.774), ('ovs_stats-br0.br0', 'if_errors', 1521044332.774, 1521044333.774), ('ovs_stats-br0.br0', 'if_packets', 1521044332.774, 1521044333.774)] 2018-03-14 16:25:23,232 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521044347.774, 1521044348.774), ('ovs_stats-br0.br0', 'if_errors', 1521044347.774, 1521044349.774), ('ovs_stats-br0.br0', 'if_packets', 1521044348.774, 1521044349.774)] 2018-03-14 16:25:23,232 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521044332.77 - ovs_stats-br0.br0 if_dropped 1521044347.77 ... 2018-03-14 16:25:23,232 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:25:23,232 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521044332.77 - ovs_stats-br0.br0 if_errors 1521044347.77 ... 2018-03-14 16:25:23,232 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:25:23,232 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521044332.77 - ovs_stats-br0.br0 if_packets 1521044348.77 ... 2018-03-14 16:25:23,232 - baro_tests.barometer - DEBUG - OK 2018-03-14 16:25:23,233 - baro_tests.barometer - INFO - OK 2018-03-14 16:25:23,233 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-14 16:25:33,735 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-14 16:25:33,737 - baro_tests.barometer - INFO - ====================================================================== 2018-03-14 16:25:33,739 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-14 16:25:33,741 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-14 16:25:33,743 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-14 16:25:33,746 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-14 16:25:33,747 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-14 16:25:33,748 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-14 16:25:33,749 - baro_tests.barometer - INFO - | IN:Hugepages | FAIL | FAIL | 2018-03-14 16:25:33,749 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | FAIL | 2018-03-14 16:25:33,750 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-14 16:25:33,750 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-03-14 16:25:33,750 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | FAIL | 2018-03-14 16:25:33,750 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-14 16:25:33,750 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-14 16:25:33,750 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-14 16:25:33,750 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-03-14 16:25:33,750 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-14 16:25:33,751 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-14 16:25:33,751 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | FAIL | 2018-03-14 16:25:33,751 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-14 16:25:33,751 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-14 16:25:33,751 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-14 16:25:33,752 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | PASS | 2018-03-14 16:25:33,752 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-14 16:25:33,754 - baro_tests.barometer - INFO - ====================================================================== 2018-03-14 16:25:33,755 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-14 16:25:33,756 - baro_tests.barometer - ERROR - Overall Result is Fail