teuthology-2017-02-27_17:05:17-rgw-master---basic-smithi

HTTPConnectionPool(host='smithi185.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7faa61d23350>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 00:53:32.597232 osd.4 172.21.15.157:6800/26939 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4118.0:149 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e29) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 00:59:58.012999 osd.1 172.21.15.62:6804/17247 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4299.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

HTTPConnectionPool(host='smithi064.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fd2cdc9abd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 00:50:53.642488 osd.0 172.21.15.172:6800/14679 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4144.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e30) v8" in cluster log

HTTPConnectionPool(host='smithi196.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fda9ec5ac50>: Failed to establish a new connection: [Errno 111] Connection refused',))

HTTPConnectionPool(host='smithi046.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f9afd5df150>: Failed to establish a new connection: [Errno 111] Connection refused',))

HTTPConnectionPool(host='smithi100.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa60f213310>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 00:57:45.707466 osd.2 172.21.15.165:6808/32208 79 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4453.0:149 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e91) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 01:35:24.571945 osd.2 172.21.15.104:6804/23688 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e52) v8" in cluster log

HTTPConnectionPool(host='smithi161.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fbc9efa9c10>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:05:00.246117 osd.1 172.21.15.55:6804/31088 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4289.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

HTTPConnectionPool(host='smithi085.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fdf5f167bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 00:58:32.341084 osd.0 172.21.15.177:6800/25306 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4143.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e28) v8" in cluster log

"2017-02-28 00:59:34.505849 osd.4 172.21.15.44:6804/3392 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4135.0:98 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e29) v8" in cluster log

{'smithi166.front.sepia.ceph.com': {'_ansible_parsed': True, 'failed': True, '_ansible_no_log': False, 'invocation': {'module_name': 'apt', 'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'package': None, 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': None, 'update_cache': True, 'deb': None, 'only_upgrade': False, 'cache_valid_time': 0, 'default_release': None, 'install_recommends': None}}, 'changed': False, 'msg': 'Failed to update apt cache.'}, 'smithi052.front.sepia.ceph.com': {'_ansible_parsed': True, 'failed': True, '_ansible_no_log': False, 'invocation': {'module_name': 'apt', 'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'package': None, 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': None, 'update_cache': True, 'deb': None, 'only_upgrade': False, 'cache_valid_time': 0, 'default_release': None, 'install_recommends': None}}, 'changed': False, 'msg': 'Failed to update apt cache.'}}

{'smithi156.front.sepia.ceph.com': {'_ansible_parsed': True, 'failed': True, '_ansible_no_log': False, 'invocation': {'module_name': 'apt', 'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'package': None, 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': None, 'update_cache': True, 'deb': None, 'only_upgrade': False, 'cache_valid_time': 0, 'default_release': None, 'install_recommends': None}}, 'changed': False, 'msg': 'Failed to update apt cache.'}}

saw valgrind issues

"2017-02-28 01:01:47.962814 osd.2 172.21.15.117:6808/27134 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4109.0:149 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e27) v8" in cluster log

HTTPConnectionPool(host='smithi174.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fc29e9fcbd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 01:39:35.417462 osd.0 172.21.15.180:6800/11972 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

HTTPConnectionPool(host='smithi051.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f88ed487c10>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 01:09:41.240880 osd.4 172.21.15.155:6808/5115 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4326.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e46) v8" in cluster log

HTTPConnectionPool(host='smithi166.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f9a68a42350>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:13:52.096444 osd.4 172.21.15.87:6804/12979 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4287.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi193.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb09c847350>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:05:49.268290 osd.2 172.21.15.95:6804/18056 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4144.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e30) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi063.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fc21b23abd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

saw valgrind issues

HTTPConnectionPool(host='smithi185.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fbcb7b07c10>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:47:35.694502 osd.2 172.21.15.176:6804/31660 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4200.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e48) v8" in cluster log

"2017-02-28 01:10:03.335618 osd.2 172.21.15.64:6808/16583 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4122.0:98 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e27) v8" in cluster log

HTTPConnectionPool(host='smithi196.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa879499310>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:49:21.524411 osd.2 172.21.15.191:6804/1504058 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

"2017-02-28 01:11:07.953884 osd.4 172.21.15.98:6804/18135 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4131.0:149 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e29) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 01:20:18.357571 osd.1 172.21.15.162:6804/22678 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4300.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e51) v8" in cluster log

HTTPConnectionPool(host='smithi082.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f0a989abbd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:11:06.385824 osd.2 172.21.15.187:6804/2194882 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4144.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e31) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi083.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fc2a0018c10>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi056.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff91153a310>: Failed to establish a new connection: [Errno 111] Connection refused',))

HTTPConnectionPool(host='smithi114.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f85cde032d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 01:56:57.578580 osd.0 172.21.15.165:6800/16643 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

HTTPConnectionPool(host='smithi054.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f4e6635bbd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:26:00.534284 osd.4 172.21.15.115:6804/12344 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4288.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi172.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7388d7dbd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:18:27.588747 osd.2 172.21.15.61:6804/15632 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4143.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e28) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 01:22:51.445414 osd.4 172.21.15.81:6805/24088 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4137.0:98 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e30) v8" in cluster log

HTTPConnectionPool(host='smithi179.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7abd452310>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:20:27.419004 osd.0 172.21.15.51:6800/16628 1 : cluster [WRN] bad locator @10 on object @10 op osd_op(client.4146.0:79 10.5 10:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e32) v8" in cluster log

"2017-02-28 01:22:57.777494 osd.2 172.21.15.200:6804/9892 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4124.0:149 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e27) v8" in cluster log

HTTPConnectionPool(host='smithi073.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff2c373abd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 02:01:17.031028 osd.0 172.21.15.59:6800/16595 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e52) v8" in cluster log

HTTPConnectionPool(host='smithi055.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fbd61877c10>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:33:12.736691 osd.1 172.21.15.64:6804/16741 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.24272.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e46) v8" in cluster log

HTTPConnectionPool(host='smithi204.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa73b240310>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:27:50.398444 osd.4 172.21.15.108:6804/4344 71 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4397.0:98 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e87) v8" in cluster log

"2017-02-28 01:24:53.716977 mon.0 172.21.15.45:6789/0 5 : cluster [WRN] message from mon.1 was stamped 0.730862s in the future, clocks not synchronized" in cluster log

"2017-02-28 01:34:44.117860 osd.4 172.21.15.185:6804/12446 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4320.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi082.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff5a0346310>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:26:07.446349 osd.2 172.21.15.177:6804/29397 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4144.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e32) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi067.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fde1be99bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

HTTPConnectionPool(host='smithi056.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f3aa725cbd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:10:33.345932 osd.0 172.21.15.196:6800/13216 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4200.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e46) v8" in cluster log

"2017-02-28 01:33:20.856561 osd.4 172.21.15.87:6804/13403 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4140.0:98 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e27) v8" in cluster log

"2017-02-28 02:13:34.042871 osd.0 172.21.15.156:6800/17471 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

HTTPConnectionPool(host='smithi090.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f41e3c6c390>: Failed to establish a new connection: [Errno 111] Connection refused',))

saw valgrind issues

"2017-02-28 01:42:44.945296 osd.1 172.21.15.54:6804/17271 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4301.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

HTTPConnectionPool(host='smithi168.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fe2df0b0bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:34:49.383135 osd.0 172.21.15.92:6800/13697 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4144.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e30) v8" in cluster log

HTTPConnectionPool(host='smithi183.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f64a3b9ec50>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:41:35.580767 osd.4 172.21.15.93:6804/6390 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4134.0:98 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e29) v8" in cluster log

HTTPConnectionPool(host='smithi162.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f4308a652d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:41:42.066891 osd.5 172.21.15.63:6804/29455 1 : cluster [WRN] bad locator @14 on object @14 op osd_op(client.4136.0:149 14.5 14:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e31) v8" in cluster log

"2017-02-28 01:44:03.595818 osd.4 172.21.15.184:6804/12440 84 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4418.0:149 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e96) v8" in cluster log

"2017-02-28 02:20:34.403590 osd.2 172.21.15.57:6804/16578 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

HTTPConnectionPool(host='smithi172.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f52576b2c50>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:50:31.006181 osd.1 172.21.15.197:6804/20596 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4294.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi082.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f721411cc10>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:43:04.875714 osd.0 172.21.15.70:6800/16346 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4143.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e28) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi204.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f9b65965390>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:45:39.651232 osd.2 172.21.15.98:6807/24026 1 : cluster [WRN] bad locator @10 on object @10 op osd_op(client.4146.0:79 10.5 10:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e32) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi056.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f4f58f492d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi067.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6899edfc10>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:28:58.386506 osd.0 172.21.15.179:6800/28812 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi051.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff4f840abd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 01:59:48.794203 osd.4 172.21.15.110:6804/21572 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4282.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e48) v8" in cluster log

"2017-02-28 01:54:22.664178 osd.0 172.21.15.185:6804/12608 1 : cluster [WRN] bad locator @14 on object @14 op osd_op(client.4145.0:98 14.5 14:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e31) v8" in cluster log

HTTPConnectionPool(host='smithi115.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f7db6578190>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:02:29.709747 osd.4 172.21.15.105:6804/23727 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4311.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 01:56:32.253990 osd.2 172.21.15.161:6800/16713 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4119.0:149 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e30) v8" in cluster log

Command failed on smithi039 with status 39: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data'

"2017-02-28 01:53:15.412527 osd.2 172.21.15.157:6808/16584 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4144.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e30) v8" in cluster log

HTTPConnectionPool(host='smithi050.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f3eae113bd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi162.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f3745f03c90>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 02:36:43.089106 osd.2 172.21.15.77:6808/16754 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4200.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e46) v8" in cluster log

HTTPConnectionPool(host='smithi070.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fe1628d6310>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:40:44.560632 osd.2 172.21.15.83:6804/13642 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

HTTPConnectionPool(host='smithi082.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f9406edb3d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 02:10:01.033487 osd.4 172.21.15.174:6804/379 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4288.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

HTTPConnectionPool(host='smithi056.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f910a45bbd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:01:25.614901 osd.2 172.21.15.52:6808/17512 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4144.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e30) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi090.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fb6b5a65c90>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 02:09:02.587813 osd.2 172.21.15.108:6805/1934 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4134.0:98 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e29) v8" in cluster log

HTTPConnectionPool(host='smithi074.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f460a72b350>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:10:30.202655 osd.0 172.21.15.114:6804/58647 1 : cluster [WRN] bad locator @14 on object @14 op osd_op(client.4128.0:157 14.5 14:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e31) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 02:47:38.785819 osd.0 172.21.15.64:6800/30627 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e52) v8" in cluster log

HTTPConnectionPool(host='smithi062.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f0af3db4c90>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:17:29.391005 osd.4 172.21.15.63:6804/30841 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4287.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e50) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi093.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f21e2c0dc50>: Failed to establish a new connection: [Errno 111] Connection refused',))

No summary info found for user: foo

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi195.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f0dd6bb03d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:11:28.685750 osd.0 172.21.15.50:6800/16593 1 : cluster [WRN] bad locator @10 on object @10 op osd_op(client.4146.0:79 10.5 10:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e32) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi161.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f641f0d7310>: Failed to establish a new connection: [Errno 111] Connection refused',))

reached maximum tries (50) after waiting for 300 seconds

HTTPConnectionPool(host='smithi082.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f03d9cd9c90>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:57:32.847781 osd.2 172.21.15.88:6804/12755 1 : cluster [WRN] bad locator @24 on object @24 op osd_op(client.4202.0:62 24.5 24:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e52) v8" in cluster log

SELinux denials found on ubuntu@smithi004.front.sepia.ceph.com: ['type=AVC msg=audit(1488253381.293:4606): avc: denied { create } for pid=28322 comm="logrotate" name="logrotate.status.tmp" scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1488253381.293:4606): avc: denied { write } for pid=28322 comm="logrotate" path="/var/lib/logrotate/logrotate.status.tmp" dev="sda1" ino=27265362 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1488253381.293:4607): avc: denied { setattr } for pid=28322 comm="logrotate" name="logrotate.status.tmp" dev="sda1" ino=27265362 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1488253381.283:4605): avc: denied { open } for pid=28322 comm="logrotate" path="/var/lib/logrotate/logrotate.status" dev="sda1" ino=27265361 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1488253381.283:4605): avc: denied { read } for pid=28322 comm="logrotate" name="logrotate.status" dev="sda1" ino=27265361 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1488253381.333:4608): avc: denied { rename } for pid=28322 comm="logrotate" name="logrotate.status.tmp" dev="sda1" ino=27265362 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1488253381.333:4608): avc: denied { unlink } for pid=28322 comm="logrotate" name="logrotate.status" dev="sda1" ino=27265361 scontext=system_u:system_r:logrotate_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file permissive=1']

HTTPConnectionPool(host='smithi045.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fae594c1c10>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:27:27.020608 osd.1 172.21.15.56:6804/16778 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4302.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e46) v8" in cluster log

"2017-02-28 02:21:28.224344 osd.0 172.21.15.157:6800/28341 1 : cluster [WRN] bad locator @14 on object @14 op osd_op(client.4118.0:98 14.5 14:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e31) v8" in cluster log

"2017-02-28 02:20:41.342241 osd.2 172.21.15.158:6808/16753 72 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4402.0:98 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e87) v8" in cluster log

HTTPConnectionPool(host='smithi078.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff513e3a3d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

"2017-02-28 02:27:40.780336 osd.4 172.21.15.69:6804/54749 1 : cluster [WRN] bad locator @23 on object @23 op osd_op(client.4292.0:62 23.5 23:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e53) v8" in cluster log

reached maximum tries (50) after waiting for 300 seconds

"2017-02-28 02:23:09.949618 osd.2 172.21.15.90:6808/12818 1 : cluster [WRN] bad locator @13 on object @13 op osd_op(client.4125.0:149 13.5 13:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e29) v8" in cluster log

"2017-02-28 02:21:04.008938 osd.0 172.21.15.183:6800/12299 1 : cluster [WRN] bad locator @9 on object @9 op osd_op(client.4144.0:79 9.5 9:b08b92bd::::head [delete] snapc 0=[] ondisk+write+known_if_redirected e30) v8" in cluster log

HTTPConnectionPool(host='smithi165.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f5078903c10>: Failed to establish a new connection: [Errno 111] Connection refused',))

HTTPConnectionPool(host='smithi074.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f4650b9ecd0>: Failed to establish a new connection: [Errno 111] Connection refused',))

HTTPConnectionPool(host='smithi108.front.sepia.ceph.com', port=7280): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f45736bb350>: Failed to establish a new connection: [Errno 111] Connection refused',))