Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 4233068 2019-08-20 05:03:42 2019-08-20 06:43:36 2019-08-20 11:03:39 4:20:03 4:11:24 0:08:39 smithi master rhel 7.6 rbd/mirror-thrash/{base/install.yaml cluster/{2-node.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml policy/none.yaml rbd-mirror/four-per-cluster.yaml supported-random-distro$/{rhel_7.yaml} users/mirror.yaml workloads/rbd-mirror-fsx-workunit.yaml} 2
Failure Reason:

"2019-08-20T07:04:32.938215+0000 mon.a (mon.0) 107 : cluster [WRN] Health check failed: 1 pools have too many placement groups (POOL_TOO_MANY_PGS)" in cluster log

fail 4233069 2019-08-20 05:03:43 2019-08-20 06:43:42 2019-08-20 11:05:45 4:22:03 3:51:24 0:30:39 smithi master rhel 7.6 rbd/mirror-thrash/{base/install.yaml cluster/{2-node.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-stupid.yaml policy/simple.yaml rbd-mirror/four-per-cluster.yaml supported-random-distro$/{rhel_7.yaml} users/mirror.yaml workloads/rbd-mirror-fsx-workunit.yaml} 2
Failure Reason:

"2019-08-20T07:26:34.264296+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: 2 pools have too many placement groups (POOL_TOO_MANY_PGS)" in cluster log

fail 4233070 2019-08-20 05:03:44 2019-08-20 06:43:49 2019-08-20 07:01:47 0:17:58 0:03:18 0:14:40 smithi master centos 7.6 rbd/mirror-thrash/{base/install.yaml cluster/{2-node.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-comp.yaml policy/none.yaml rbd-mirror/four-per-cluster.yaml supported-random-distro$/{centos_7.yaml} users/mirror.yaml workloads/rbd-mirror-fsx-workunit.yaml} 2
Failure Reason:

5451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nfailure: repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.\nhttp://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nhttp://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nhttp://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nhttp://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found", u'cmd': [u'yum-complete-transaction', u'--cleanup-only'], u'rc': 1, u'stderr': u'', u'delta': u'0:00:08.997830', u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'strip_empty_ends': True, u'_raw_params': u'yum-complete-transaction --cleanup-only', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}, 'stdout_lines': [u'Loaded plugins: fastestmirror, langpacks, priorities', u'Determining fastest mirrors', u' * base: mirror.vtti.vt.edu', u' * epel: ftp.linux.ncsu.edu', u' * extras: mirror.fileplanet.com', u' * updates: mirror.cc.columbia.edu', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'To address this issue please refer to the below wiki article ', u'', u'https://wiki.centos.org/yum-errors', u'', u"If above article doesn't help to resolve this issue please use https://bugs.centos.org/.", u'', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'failure: repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found'], u'start': u'2019-08-20 07:00:20.887417', u'msg': u'non-zero return code'}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 219, in represent_list return self.represent_sequence(u'tag:yaml.org,2002:seq', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 102, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'yum-complete-transaction')Failure object was: {'smithi084.front.sepia.ceph.com': {'stderr_lines': [], u'changed': False, u'end': u'2019-08-20 07:00:35.628628', '_ansible_no_log': False, u'stdout': u"Loaded plugins: fastestmirror, langpacks, priorities\nDetermining fastest mirrors\n * base: mirror.vtti.vt.edu\n * epel: ftp.linux.ncsu.edu\n * extras: mirror.fileplanet.com\n * updates: mirror.cc.columbia.edu\nhttp://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nTo address this issue please refer to the below wiki article \n\nhttps://wiki.centos.org/yum-errors\n\nIf above article doesn't help to resolve this issue please use https://bugs.centos.org/.\n\nhttp://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.pnl.gov/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.oss.ou.edu/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nfailure: repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.\nhttp://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nhttp://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nhttp://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nhttp://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found", u'cmd': [u'yum-complete-transaction', u'--cleanup-only'], u'rc': 1, u'stderr': u'', u'delta': u'0:00:08.890946', u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': False, u'strip_empty_ends': True, u'_raw_params': u'yum-complete-transaction --cleanup-only', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}, 'stdout_lines': [u'Loaded plugins: fastestmirror, langpacks, priorities', u'Determining fastest mirrors', u' * base: mirror.vtti.vt.edu', u' * epel: ftp.linux.ncsu.edu', u' * extras: mirror.fileplanet.com', u' * updates: mirror.cc.columbia.edu', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'To address this issue please refer to the below wiki article ', u'', u'https://wiki.centos.org/yum-errors', u'', u"If above article doesn't help to resolve this issue please use https://bugs.centos.org/.", u'', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/02ad53e4a8943c664c67aa7ea262847b706b870148fd16535451a2e543fa17af-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'failure: repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.', u'http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'http://mirror.pnl.gov/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'http://mirror.oss.ou.edu/epel/7/x86_64/repodata/53fc8cfd63713020d57c1607cf3cb099c7708a2809bcd3f242f90305af8cb924-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found'], u'start': u'2019-08-20 07:00:26.737682', u'msg': u'non-zero return code'}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 219, in represent_list return self.represent_sequence(u'tag:yaml.org,2002:seq', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 102, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'yum-complete-transaction')

fail 4233071 2019-08-20 05:03:45 2019-08-20 06:46:05 2019-08-20 10:22:06 3:36:01 3:22:04 0:13:57 smithi master rhel 7.6 rbd/mirror-thrash/{base/install.yaml cluster/{2-node.yaml openstack.yaml} msgr-failures/few.yaml objectstore/filestore-xfs.yaml policy/simple.yaml rbd-mirror/four-per-cluster.yaml supported-random-distro$/{rhel_7.yaml} users/mirror.yaml workloads/rbd-mirror-fsx-workunit.yaml} 2
Failure Reason:

"2019-08-20T07:10:15.636488+0000 mon.a (mon.0) 114 : cluster [WRN] Health check failed: 2 pools have too many placement groups (POOL_TOO_MANY_PGS)" in cluster log

fail 4233072 2019-08-20 05:03:45 2019-08-20 06:47:48 2019-08-20 10:43:51 3:56:03 3:37:50 0:18:13 smithi master rhel 7.6 rbd/mirror-thrash/{base/install.yaml cluster/{2-node.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-low-osd-mem-target.yaml policy/none.yaml rbd-mirror/four-per-cluster.yaml supported-random-distro$/{rhel_7.yaml} users/mirror.yaml workloads/rbd-mirror-fsx-workunit.yaml} 2
Failure Reason:

"2019-08-20T07:18:13.837219+0000 mon.a (mon.0) 108 : cluster [WRN] Health check failed: 1 pools have too many placement groups (POOL_TOO_MANY_PGS)" in cluster log