2018-03-25 11:48:02,656 - baro_tests.barometer - INFO - =================== Starting barometer tests suite =================== 2018-03-25 11:48:02,656 - baro_tests.barometer - INFO - RSA key file /root/.ssh/id_rsa exists. 2018-03-25 11:48:06,660 - baro_tests.barometer - INFO - ==== Display of Control and Compute nodes available in the set up ==== 2018-03-25 11:48:06,660 - baro_tests.barometer - INFO - controllers: ['overcloud-controller-0: 192.30.9.3'] 2018-03-25 11:48:06,661 - baro_tests.barometer - INFO - computes: ['overcloud-novacompute-0: 192.30.9.9', 'overcloud-novacompute-1: 192.30.9.5'] 2018-03-25 11:48:06,661 - baro_tests.barometer - INFO - ========== Enabling mcelog and OVS bridges on compute nodes ========== 2018-03-25 11:48:18,132 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-0). 2018-03-25 11:48:18,173 - baro_tests.barometer - INFO - Mcelog will NOT be enabled on node-{}. Unsupported CentOS release found (overcloud-novacompute-1). 2018-03-25 11:48:29,472 - baro_tests.barometer - INFO - Creating OVS bridges on computes nodes 2018-03-25 11:48:29,725 - baro_tests.barometer - INFO - OVS Bridges created on compute nodes 2018-03-25 11:48:29,725 - baro_tests.barometer - INFO - ============= Testing Gnocchi and AODH plugins on nodes ============== 2018-03-25 11:49:09,334 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-0 2018-03-25 11:49:21,178 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-0 2018-03-25 11:49:21,178 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-0: Enabling Test Plug-in and Test case execution === 2018-03-25 11:50:31,074 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-25 11:50:53,923 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 11:51:31,144 - baro_tests.barometer - INFO - PASS 2018-03-25 11:51:31,144 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with Gnocchi PASSED on Hugepages. 2018-03-25 11:52:17,797 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-25 11:52:50,903 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 11:52:53,656 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Intel RDT. 2018-03-25 11:53:38,873 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-25 11:54:02,367 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 11:54:05,033 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on Mcelog. 2018-03-25 11:54:51,426 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-25 11:55:13,761 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 11:55:16,346 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS events. 2018-03-25 11:56:00,413 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-25 11:56:21,384 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 11:56:58,460 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-25 11:56:58,460 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-25 11:56:58,461 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with Gnocchi FAILED on OVS stats. 2018-03-25 11:57:45,324 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-25 11:58:06,815 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-25 11:58:26,459 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-25 11:58:26,460 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on Mcelog. 2018-03-25 11:59:11,499 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-25 11:59:33,076 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-25 11:59:35,322 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with AODH FAILED on OVS events. 2018-03-25 11:59:35,323 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-0: Executing CSV Testcases ======== 2018-03-25 11:59:35,323 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-25 11:59:46,855 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-25 11:59:57,900 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-25 12:00:08,189 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-25 12:01:00,816 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Hugepages Test case execution === 2018-03-25 12:01:21,284 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 9fc8f7e3-af67-4354-9613-a07b3bb52e19... 2018-03-25 12:01:21,284 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-25 12:01:21,284 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-25 12:01:21,285 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-25 12:01:32,754 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 12:01:32,755 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521978906.14 ... 2018-03-25 12:01:32,755 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:01:32,755 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521978906.14 ... 2018-03-25 12:01:32,755 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:01:32,755 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521978906.14 ... 2018-03-25 12:01:32,756 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:01:32,756 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521978906.14 ... 2018-03-25 12:01:32,756 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:01:32,756 - baro_tests.barometer - INFO - OK 2018-03-25 12:01:32,756 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-25 12:01:44,761 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-25 12:01:55,712 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 12:01:55,713 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521978906.136, 1521978916.136), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521978906.136, 1521978916.136), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521978906.136, 1521978916.136), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521978906.136, 1521978916.136)] 2018-03-25 12:01:55,713 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521978926.136, 1521978936.136), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521978926.136, 1521978936.136), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521978926.136, 1521978936.136), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521978926.136, 1521978936.136)] 2018-03-25 12:01:55,714 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521978906.14 - hugepages-mm-2048Kb vmpage_number-free 1521978926.14 ... 2018-03-25 12:01:55,714 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:01:55,715 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521978906.14 - hugepages-mm-2048Kb vmpage_number-used 1521978926.14 ... 2018-03-25 12:01:55,715 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:01:55,715 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521978906.14 - hugepages-node0-2048Kb vmpage_number-free 1521978926.14 ... 2018-03-25 12:01:55,715 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:01:55,715 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521978906.14 - hugepages-node0-2048Kb vmpage_number-used 1521978926.14 ... 2018-03-25 12:01:55,715 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:01:55,715 - baro_tests.barometer - INFO - OK 2018-03-25 12:01:55,716 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Hugepages. 2018-03-25 12:02:37,457 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Intel RDT Test case execution === 2018-03-25 12:03:08,545 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 9fc8f7e3-af67-4354-9613-a07b3bb52e19... 2018-03-25 12:03:08,546 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-25 12:03:08,546 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-25 12:03:08,546 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-25 12:03:19,410 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 12:03:19,411 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521979016.14 ... 2018-03-25 12:03:19,411 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:03:19,411 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521979016.14 ... 2018-03-25 12:03:19,411 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:03:19,412 - baro_tests.barometer - INFO - OK 2018-03-25 12:03:19,412 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-25 12:03:31,424 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-25 12:03:41,725 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 12:03:41,726 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521979016.136, 1521979026.136), ('intel_rdt-0-2', 'ipc', 1521979016.136, 1521979026.136)] 2018-03-25 12:03:41,727 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521979036.136, 1521979046.136), ('intel_rdt-0-2', 'ipc', 1521979036.136, 1521979046.136)] 2018-03-25 12:03:41,727 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521979016.14 - intel_rdt-0-2 bytes-llc 1521979036.14 ... 2018-03-25 12:03:41,727 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:03:41,728 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521979016.14 - intel_rdt-0-2 ipc 1521979036.14 ... 2018-03-25 12:03:41,728 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:03:41,729 - baro_tests.barometer - INFO - OK 2018-03-25 12:03:41,730 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on Intel RDT. 2018-03-25 12:04:23,098 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in Mcelog Test case execution === 2018-03-25 12:04:45,384 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 9fc8f7e3-af67-4354-9613-a07b3bb52e19... 2018-03-25 12:04:45,385 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-25 12:04:45,385 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-25 12:04:45,385 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-25 12:04:56,812 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-25 12:04:56,913 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-25 12:04:57,014 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-25 12:04:57,116 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-25 12:04:57,117 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-25 12:04:57,118 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-0 with CSV FAILED on Mcelog. 2018-03-25 12:05:38,817 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS events Test case execution === 2018-03-25 12:06:00,036 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 9fc8f7e3-af67-4354-9613-a07b3bb52e19... 2018-03-25 12:06:00,036 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-25 12:06:00,037 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-25 12:06:00,037 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-25 12:06:10,713 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 12:06:10,714 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521979197.14 ... 2018-03-25 12:06:10,714 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:06:10,714 - baro_tests.barometer - INFO - OK 2018-03-25 12:06:10,715 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-25 12:06:13,718 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-25 12:06:24,108 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 12:06:24,108 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521979197.137, 1521979198.137)] 2018-03-25 12:06:24,109 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521979210.137, 1521979211.137)] 2018-03-25 12:06:24,109 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521979197.14 - ovs_events-br0 gauge-link_status 1521979210.14 ... 2018-03-25 12:06:24,109 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:06:24,110 - baro_tests.barometer - INFO - OK 2018-03-25 12:06:24,111 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS events. 2018-03-25 12:07:07,295 - baro_tests.barometer - INFO - === Node overcloud-novacompute-0: Plug-in OVS stats Test case execution === 2018-03-25 12:07:28,129 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 9fc8f7e3-af67-4354-9613-a07b3bb52e19... 2018-03-25 12:07:28,129 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-25 12:07:28,129 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-25 12:07:28,129 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-25 12:07:38,912 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 12:07:38,912 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521979285.14 ... 2018-03-25 12:07:38,913 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:07:38,913 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521979285.14 ... 2018-03-25 12:07:38,913 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:07:38,913 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521979285.14 ... 2018-03-25 12:07:38,913 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:07:38,914 - baro_tests.barometer - INFO - OK 2018-03-25 12:07:38,914 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-25 12:07:41,916 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-0... 2018-03-25 12:07:52,466 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 12:07:52,466 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521979285.137, 1521979286.137), ('ovs_stats-br0.br0', 'if_errors', 1521979285.137, 1521979286.137), ('ovs_stats-br0.br0', 'if_packets', 1521979285.137, 1521979286.137)] 2018-03-25 12:07:52,466 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521979299.137, 1521979300.137), ('ovs_stats-br0.br0', 'if_errors', 1521979299.137, 1521979300.137), ('ovs_stats-br0.br0', 'if_packets', 1521979299.137, 1521979300.137)] 2018-03-25 12:07:52,467 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521979285.14 - ovs_stats-br0.br0 if_dropped 1521979299.14 ... 2018-03-25 12:07:52,467 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:07:52,467 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521979285.14 - ovs_stats-br0.br0 if_errors 1521979299.14 ... 2018-03-25 12:07:52,467 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:07:52,467 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521979285.14 - ovs_stats-br0.br0 if_packets 1521979299.14 ... 2018-03-25 12:07:52,467 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:07:52,467 - baro_tests.barometer - INFO - OK 2018-03-25 12:07:52,468 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-0 with CSV PASSED on OVS stats. 2018-03-25 12:08:03,104 - baro_tests.barometer - INFO - Gnocchi plugin available in compute node overcloud-novacompute-1 2018-03-25 12:08:13,413 - baro_tests.barometer - INFO - AODH plugin present in compute node overcloud-novacompute-1 2018-03-25 12:08:13,413 - baro_tests.barometer - INFO - === NODE overcloud-novacompute-1: Enabling Test Plug-in and Test case execution === 2018-03-25 12:09:13,015 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-25 12:09:33,456 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 12:10:10,425 - baro_tests.barometer - INFO - Plugin Interval is 10 2018-03-25 12:10:10,426 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-25 12:10:10,427 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Hugepages. 2018-03-25 12:10:54,231 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-25 12:11:26,496 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 12:12:03,623 - baro_tests.barometer - INFO - PASS 2018-03-25 12:12:03,624 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with Gnocchi PASSED on Intel RDT. 2018-03-25 12:12:45,443 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-25 12:13:06,674 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 12:13:09,430 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on Mcelog. 2018-03-25 12:13:50,742 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-25 12:14:11,514 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 12:14:13,972 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS events. 2018-03-25 12:14:55,226 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-25 12:15:15,802 - baro_tests.barometer - INFO - Getting gnocchi metric list on overcloud-controller-0 2018-03-25 12:15:52,960 - baro_tests.barometer - INFO - Plugin Interval is 1 2018-03-25 12:15:52,961 - baro_tests.barometer - INFO - Data not updated after 30 seconds 2018-03-25 12:15:52,962 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with Gnocchi FAILED on OVS stats. 2018-03-25 12:16:33,352 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-25 12:16:53,800 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-25 12:17:12,510 - baro_tests.barometer - INFO - Data not updated after interval of 12 seconds 2018-03-25 12:17:12,511 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on Mcelog. 2018-03-25 12:17:55,466 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-25 12:18:16,245 - baro_tests.barometer - INFO - Getting AODH Alarm list on overcloud-controller-0 2018-03-25 12:18:18,499 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with AODH FAILED on OVS events. 2018-03-25 12:18:18,499 - baro_tests.barometer - INFO - ======= Node overcloud-novacompute-1: Executing CSV Testcases ======== 2018-03-25 12:18:18,499 - baro_tests.barometer - INFO - Restarting collectd for CSV tests 2018-03-25 12:18:28,867 - baro_tests.barometer - INFO - Stopping collectd service... 2018-03-25 12:18:39,203 - baro_tests.barometer - INFO - Starting collectd service... 2018-03-25 12:18:49,502 - baro_tests.barometer - INFO - Sleeping for 10 seconds after collectd restart... 2018-03-25 12:19:41,536 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Hugepages Test case execution === 2018-03-25 12:20:00,805 - baro_tests.barometer - INFO - Getting CSV metrics of plugin hugepages on compute node 6795c57b-3258-4917-97b6-0bb6b0dcfba7... 2018-03-25 12:20:00,805 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-25 12:20:00,806 - baro_tests.barometer - DEBUG - Plugin subdirs: ['hugepages-mm-2048Kb', 'hugepages-node0-2048Kb'] 2018-03-25 12:20:00,806 - baro_tests.barometer - DEBUG - Plugin meter categories: ['vmpage_number-free', 'vmpage_number-used'] 2018-03-25 12:20:11,645 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 12:20:11,645 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521980014.54 ... 2018-03-25 12:20:11,646 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:20:11,646 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521980014.54 ... 2018-03-25 12:20:11,646 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:20:11,646 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521980014.54 ... 2018-03-25 12:20:11,646 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:20:11,646 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521980014.54 ... 2018-03-25 12:20:11,646 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:20:11,647 - baro_tests.barometer - INFO - OK 2018-03-25 12:20:11,647 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-25 12:20:23,659 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-25 12:20:34,740 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 12:20:34,740 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521980014.544, 1521980024.544), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521980014.544, 1521980024.544), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521980014.544, 1521980024.544), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521980014.544, 1521980024.544)] 2018-03-25 12:20:34,740 - baro_tests.barometer - DEBUG - [('hugepages-mm-2048Kb', 'vmpage_number-free', 1521980034.544, 1521980044.544), ('hugepages-mm-2048Kb', 'vmpage_number-used', 1521980034.544, 1521980044.544), ('hugepages-node0-2048Kb', 'vmpage_number-free', 1521980034.544, 1521980044.545), ('hugepages-node0-2048Kb', 'vmpage_number-used', 1521980034.544, 1521980044.545)] 2018-03-25 12:20:34,741 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-free 1521980014.54 - hugepages-mm-2048Kb vmpage_number-free 1521980034.54 ... 2018-03-25 12:20:34,741 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:20:34,741 - baro_tests.barometer - DEBUG - hugepages-mm-2048Kb vmpage_number-used 1521980014.54 - hugepages-mm-2048Kb vmpage_number-used 1521980034.54 ... 2018-03-25 12:20:34,741 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:20:34,741 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-free 1521980014.54 - hugepages-node0-2048Kb vmpage_number-free 1521980034.54 ... 2018-03-25 12:20:34,741 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:20:34,741 - baro_tests.barometer - DEBUG - hugepages-node0-2048Kb vmpage_number-used 1521980014.54 - hugepages-node0-2048Kb vmpage_number-used 1521980034.54 ... 2018-03-25 12:20:34,741 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:20:34,742 - baro_tests.barometer - INFO - OK 2018-03-25 12:20:34,742 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Hugepages. 2018-03-25 12:21:15,730 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Intel RDT Test case execution === 2018-03-25 12:21:46,719 - baro_tests.barometer - INFO - Getting CSV metrics of plugin intel_rdt on compute node 6795c57b-3258-4917-97b6-0bb6b0dcfba7... 2018-03-25 12:21:46,719 - baro_tests.barometer - DEBUG - Interval: 10 2018-03-25 12:21:46,719 - baro_tests.barometer - DEBUG - Plugin subdirs: ['intel_rdt-0-2'] 2018-03-25 12:21:46,720 - baro_tests.barometer - DEBUG - Plugin meter categories: ['bytes-llc', 'ipc'] 2018-03-25 12:21:57,215 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 12:21:57,215 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521980114.54 ... 2018-03-25 12:21:57,215 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:21:57,215 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521980114.54 ... 2018-03-25 12:21:57,215 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:21:57,215 - baro_tests.barometer - INFO - OK 2018-03-25 12:21:57,215 - baro_tests.barometer - INFO - Sleeping for 12 seconds to get updated entries (interval is 10 sec)... 2018-03-25 12:22:09,228 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-25 12:22:20,180 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 12:22:20,182 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521980114.544, 1521980124.544), ('intel_rdt-0-2', 'ipc', 1521980114.544, 1521980124.544)] 2018-03-25 12:22:20,184 - baro_tests.barometer - DEBUG - [('intel_rdt-0-2', 'bytes-llc', 1521980144.544, 1521980154.544), ('intel_rdt-0-2', 'ipc', 1521980144.544, 1521980154.544)] 2018-03-25 12:22:20,189 - baro_tests.barometer - DEBUG - intel_rdt-0-2 bytes-llc 1521980114.54 - intel_rdt-0-2 bytes-llc 1521980144.54 ... 2018-03-25 12:22:20,189 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:22:20,191 - baro_tests.barometer - DEBUG - intel_rdt-0-2 ipc 1521980114.54 - intel_rdt-0-2 ipc 1521980144.54 ... 2018-03-25 12:22:20,192 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:22:20,193 - baro_tests.barometer - INFO - OK 2018-03-25 12:22:20,194 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on Intel RDT. 2018-03-25 12:23:02,873 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in Mcelog Test case execution === 2018-03-25 12:23:22,564 - baro_tests.barometer - INFO - Getting CSV metrics of plugin mcelog on compute node 6795c57b-3258-4917-97b6-0bb6b0dcfba7... 2018-03-25 12:23:22,564 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-25 12:23:22,564 - baro_tests.barometer - DEBUG - Plugin subdirs: ['mcelog-SOCKET_0_CHANNEL_0_DIMM_any', 'mcelog-SOCKET_0_CHANNEL_any_DIMM_any'] 2018-03-25 12:23:22,564 - baro_tests.barometer - DEBUG - Plugin meter categories: ['errors-corrected_memory_errors', 'errors-uncorrected_memory_errors'] 2018-03-25 12:23:33,483 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-25 12:23:33,616 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_0_DIMM_any subdir failed 2018-03-25 12:23:33,719 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-corrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-25 12:23:33,824 - baro_tests.barometer - ERROR - Getting last two CSV entries of meter category errors-uncorrected_memory_errors in mcelog-SOCKET_0_CHANNEL_any_DIMM_any subdir failed 2018-03-25 12:23:33,827 - baro_tests.barometer - ERROR - Some plugin metrics not found 2018-03-25 12:23:33,833 - baro_tests.barometer - ERROR - Test case for overcloud-novacompute-1 with CSV FAILED on Mcelog. 2018-03-25 12:24:16,661 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS events Test case execution === 2018-03-25 12:24:37,354 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_events on compute node 6795c57b-3258-4917-97b6-0bb6b0dcfba7... 2018-03-25 12:24:37,354 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-25 12:24:37,355 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_events-br0'] 2018-03-25 12:24:37,355 - baro_tests.barometer - DEBUG - Plugin meter categories: ['gauge-link_status'] 2018-03-25 12:24:48,020 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 12:24:48,020 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521980301.55 ... 2018-03-25 12:24:48,020 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:24:48,021 - baro_tests.barometer - INFO - OK 2018-03-25 12:24:48,021 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-25 12:24:51,024 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-25 12:25:01,581 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 12:25:01,581 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521980301.545, 1521980302.545)] 2018-03-25 12:25:01,581 - baro_tests.barometer - DEBUG - [('ovs_events-br0', 'gauge-link_status', 1521980315.545, 1521980316.545)] 2018-03-25 12:25:01,582 - baro_tests.barometer - DEBUG - ovs_events-br0 gauge-link_status 1521980301.55 - ovs_events-br0 gauge-link_status 1521980315.55 ... 2018-03-25 12:25:01,583 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:25:01,583 - baro_tests.barometer - INFO - OK 2018-03-25 12:25:01,584 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS events. 2018-03-25 12:25:44,316 - baro_tests.barometer - INFO - === Node overcloud-novacompute-1: Plug-in OVS stats Test case execution === 2018-03-25 12:26:04,000 - baro_tests.barometer - INFO - Getting CSV metrics of plugin ovs_stats on compute node 6795c57b-3258-4917-97b6-0bb6b0dcfba7... 2018-03-25 12:26:04,001 - baro_tests.barometer - DEBUG - Interval: 1 2018-03-25 12:26:04,001 - baro_tests.barometer - DEBUG - Plugin subdirs: ['ovs_stats-br0.br0'] 2018-03-25 12:26:04,001 - baro_tests.barometer - DEBUG - Plugin meter categories: ['if_dropped', 'if_errors', 'if_packets'] 2018-03-25 12:26:15,434 - baro_tests.barometer - INFO - Checking that last two entries in metrics are correspondingto interval... 2018-03-25 12:26:15,434 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521980388.55 ... 2018-03-25 12:26:15,435 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:26:15,435 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521980388.55 ... 2018-03-25 12:26:15,435 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:26:15,435 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521980388.55 ... 2018-03-25 12:26:15,435 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:26:15,435 - baro_tests.barometer - INFO - OK 2018-03-25 12:26:15,435 - baro_tests.barometer - INFO - Sleeping for 3 seconds to get updated entries (interval is 1 sec)... 2018-03-25 12:26:18,439 - baro_tests.barometer - INFO - Getting new metrics of compute node overcloud-novacompute-1... 2018-03-25 12:26:30,089 - baro_tests.barometer - INFO - Comparing old and new metrics... 2018-03-25 12:26:30,089 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521980388.545, 1521980389.545), ('ovs_stats-br0.br0', 'if_errors', 1521980388.545, 1521980389.545), ('ovs_stats-br0.br0', 'if_packets', 1521980388.545, 1521980389.545)] 2018-03-25 12:26:30,090 - baro_tests.barometer - DEBUG - [('ovs_stats-br0.br0', 'if_dropped', 1521980402.545, 1521980403.545), ('ovs_stats-br0.br0', 'if_errors', 1521980403.545, 1521980404.545), ('ovs_stats-br0.br0', 'if_packets', 1521980403.545, 1521980404.545)] 2018-03-25 12:26:30,090 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_dropped 1521980388.55 - ovs_stats-br0.br0 if_dropped 1521980402.55 ... 2018-03-25 12:26:30,090 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:26:30,090 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_errors 1521980388.55 - ovs_stats-br0.br0 if_errors 1521980403.55 ... 2018-03-25 12:26:30,090 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:26:30,091 - baro_tests.barometer - DEBUG - ovs_stats-br0.br0 if_packets 1521980388.55 - ovs_stats-br0.br0 if_packets 1521980403.55 ... 2018-03-25 12:26:30,091 - baro_tests.barometer - DEBUG - OK 2018-03-25 12:26:30,091 - baro_tests.barometer - INFO - OK 2018-03-25 12:26:30,091 - baro_tests.barometer - INFO - Test case for overcloud-novacompute-1 with CSV PASSED on OVS stats. 2018-03-25 12:26:41,250 - baro_tests.barometer - INFO - Mcelog is deleted from all compute nodes 2018-03-25 12:26:41,251 - baro_tests.barometer - INFO - ====================================================================== 2018-03-25 12:26:41,252 - baro_tests.barometer - INFO - +----------------------------------+ 2018-03-25 12:26:41,252 - baro_tests.barometer - INFO - | OVERALL SUMMARY | 2018-03-25 12:26:41,252 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 12:26:41,253 - baro_tests.barometer - INFO - | Test | Node-0 | Node-1 | 2018-03-25 12:26:41,254 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 12:26:41,254 - baro_tests.barometer - INFO - | OUT:Gnocchi | PASS | PASS | 2018-03-25 12:26:41,255 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | FAIL | 2018-03-25 12:26:41,255 - baro_tests.barometer - INFO - | IN:Intel RDT | FAIL | PASS | 2018-03-25 12:26:41,256 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-25 12:26:41,256 - baro_tests.barometer - INFO - | IN:OVS events | FAIL | FAIL | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | IN:OVS stats | FAIL | FAIL | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | OUT:AODH | PASS | PASS | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | IN:OVS events | SKIP | SKIP | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | OUT:CSV | PASS | PASS | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | IN:Hugepages | PASS | PASS | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | IN:Intel RDT | PASS | PASS | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | IN:Mcelog | FAIL | FAIL | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | IN:OVS events | PASS | PASS | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - | IN:OVS stats | PASS | PASS | 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - +----------------+--------+--------+ 2018-03-25 12:26:41,257 - baro_tests.barometer - INFO - ====================================================================== 2018-03-25 12:26:41,257 - baro_tests.barometer - ERROR - Some tests have failed or have not been executed 2018-03-25 12:26:41,258 - baro_tests.barometer - ERROR - Overall Result is Fail