Nodes: mira066 mira083

Description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/redirect.yaml}

Log: http://qa-proxy.ceph.com/teuthology/kchai-2019-11-17_01:29:22-rados-wip-kefu-testing-2019-11-17-0135-distro-basic-mira/4513993/teuthology.log

Failure Reason:

d': True, u'stderr': u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#18 - "transfer closed with 1 bytes remaining to read"\nTrying other mirror.\n\n\n One of the configured repositories failed (Extra Packages for Enterprise Linux),\n and yum doesn\'t have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work "fix" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Run the command with the repository temporarily disabled\n yum --disablerepo=epel ...\n\n 4. Disable the repository permanently, so yum won\'t use it by default. Yum\n will then just ignore the repository until you permanently enable it\n again or use --enablerepo for temporary usage:\n\n yum-config-manager --disable epel\n or\n subscription-manager repos --disable=epel\n\n 5. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=epel.skip_if_unavailable=true\n\nfailure: repodata/repomd.xml from epel: [Errno 256] No more mirrors to try.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#18 - "transfer closed with 1 bytes remaining to read"', u'stdout': u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nCleaning repos: epel lab-extras rhel-7-fcgi-ceph rhel-7-server-extras-rpms\n : rhel-7-server-optional-rpms rhel-7-server-rpms\n : rhel-ha-for-rhel-7-server-rpms\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nDetermining fastest mirrors', u'msg': u'non-zero return code', u'delta': u'0:00:02.690009', 'stdout_lines': [u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-', u' : disabled-repos, subscription-manager', u'Cleaning repos: epel lab-extras rhel-7-fcgi-ceph rhel-7-server-extras-rpms', u' : rhel-7-server-optional-rpms rhel-7-server-rpms', u' : rhel-ha-for-rhel-7-server-rpms', u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-', u' : disabled-repos, subscription-manager', u'Determining fastest mirrors'], u'end': u'2019-11-18 03:58:12.159671', '_ansible_no_log': False, u'cmd': u'rm -rf /var/cache/yum/*; yum clean all; yum makecache', u'start': u'2019-11-18 03:58:09.469662', u'warnings': [u"Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."], u'rc': 1, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'strip_empty_ends': True, u'_raw_params': u'rm -rf /var/cache/yum/*; yum clean all; yum makecache', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'rm -rf /var/cache/yum/*; yum clean all; yum makecache')Failure object was: {'mira066.front.sepia.ceph.com': {'stderr_lines': [u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] curl#18 - "transfer closed with 1758586 bytes remaining to read"', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/78364192654f81f335a53ad2e150be332036a37b75ee154c010f36692077f4bc-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'To address this issue please refer to the below knowledge base article ', u'', u'https://access.redhat.com/articles/1320623', u'', u"If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.", u'', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/7ef61dc594674fe902020c298a6e8a943b6e15ced7768d0f8bd82047964fb22e-updateinfo.xml.zck: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/78364192654f81f335a53ad2e150be332036a37b75ee154c010f36692077f4bc-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'', u'', u' One of the configured repositories failed (Extra Packages for Enterprise Linux),', u" and yum doesn't have enough cached data to continue. At this point the only", u' safe thing yum can do is fail. There are a few ways to work "fix" this:', u'', u' 1. Contact the upstream for the repository and get them to fix the problem.', u'', u' 2. Reconfigure the baseurl/etc. for the repository, to point to a working', u' upstream. This is most often useful if you are using a newer', u' distribution release than is supported by the repository (and the', u' packages for the previous distribution release still work).', u'', u' 3. Run the command with the repository temporarily disabled', u' yum --disablerepo=epel ...', u'', u" 4. Disable the repository permanently, so yum won't use it by default. Yum", u' will then just ignore the repository until you permanently enable it', u' again or use --enablerepo for temporary usage:', u'', u' yum-config-manager --disable epel', u' or', u' subscription-manager repos --disable=epel', u'', u' 5. Configure the failing repository to be skipped, if it is unavailable.', u' Note that yum will try to contact the repo. when it runs most commands,', u' so will have to try and fail each time (and thus. yum will be be much', u' slower). If it is a very temporary problem though, this is often a nice', u' compromise:', u'', u' yum-config-manager --save --setopt=epel.skip_if_unavailable=true', u'', u'failure: repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found'], u'changed': True, u'stderr': u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] curl#18 - "transfer closed with 1758586 bytes remaining to read"\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/78364192654f81f335a53ad2e150be332036a37b75ee154c010f36692077f4bc-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nTo address this issue please refer to the below knowledge base article \n\nhttps://access.redhat.com/articles/1320623\n\nIf above article doesn\'t help to resolve this issue please open a ticket with Red Hat Support.\n\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/7ef61dc594674fe902020c298a6e8a943b6e15ced7768d0f8bd82047964fb22e-updateinfo.xml.zck: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/78364192654f81f335a53ad2e150be332036a37b75ee154c010f36692077f4bc-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\n\n\n One of the configured repositories failed (Extra Packages for Enterprise Linux),\n and yum doesn\'t have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work "fix" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Run the command with the repository temporarily disabled\n yum --disablerepo=epel ...\n\n 4. Disable the repository permanently, so yum won\'t use it by default. Yum\n will then just ignore the repository until you permanently enable it\n again or use --enablerepo for temporary usage:\n\n yum-config-manager --disable epel\n or\n subscription-manager repos --disable=epel\n\n 5. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=epel.skip_if_unavailable=true\n\nfailure: repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'stdout': u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nCleaning repos: epel lab-extras rhel-7-fcgi-ceph rhel-7-server-extras-rpms\n : rhel-7-server-optional-rpms rhel-7-server-rpms\n : rhel-ha-for-rhel-7-server-rpms\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nDetermining fastest mirrors', u'msg': u'non-zero return code', u'delta': u'0:00:19.995013', 'stdout_lines': [u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-', u' : disabled-repos, subscription-manager', u'Cleaning repos: epel lab-extras rhel-7-fcgi-ceph rhel-7-server-extras-rpms', u' : rhel-7-server-optional-rpms rhel-7-server-rpms', u' : rhel-ha-for-rhel-7-server-rpms', u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-', u' : disabled-repos, subscription-manager', u'Determining fastest mirrors'], u'end': u'2019-11-18 03:58:24.378963', '_ansible_no_log': False, u'cmd': u'rm -rf /var/cache/yum/*; yum clean all; yum makecache', u'start': u'2019-11-18 03:58:04.383950', u'warnings': [u"Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."], u'rc': 1, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'strip_empty_ends': True, u'_raw_params': u'rm -rf /var/cache/yum/*; yum clean all; yum makecache', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'rm -rf /var/cache/yum/*; yum clean all; yum makecache')

  • log_href: http://qa-proxy.ceph.com/teuthology/kchai-2019-11-17_01:29:22-rados-wip-kefu-testing-2019-11-17-0135-distro-basic-mira/4513993/teuthology.log
  • archive_path: /home/teuthworker/archive/kchai-2019-11-17_01:29:22-rados-wip-kefu-testing-2019-11-17-0135-distro-basic-mira/4513993
  • description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/redirect.yaml}
  • duration: 0:03:04
  • email: tchaikov@gmail.com
  • failure_reason: d': True, u'stderr': u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#18 - "transfer closed with 1 bytes remaining to read"\nTrying other mirror.\n\n\n One of the configured repositories failed (Extra Packages for Enterprise Linux),\n and yum doesn\'t have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work "fix" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Run the command with the repository temporarily disabled\n yum --disablerepo=epel ...\n\n 4. Disable the repository permanently, so yum won\'t use it by default. Yum\n will then just ignore the repository until you permanently enable it\n again or use --enablerepo for temporary usage:\n\n yum-config-manager --disable epel\n or\n subscription-manager repos --disable=epel\n\n 5. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=epel.skip_if_unavailable=true\n\nfailure: repodata/repomd.xml from epel: [Errno 256] No more mirrors to try.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/repomd.xml: [Errno 14] curl#18 - "transfer closed with 1 bytes remaining to read"', u'stdout': u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nCleaning repos: epel lab-extras rhel-7-fcgi-ceph rhel-7-server-extras-rpms\n : rhel-7-server-optional-rpms rhel-7-server-rpms\n : rhel-ha-for-rhel-7-server-rpms\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nDetermining fastest mirrors', u'msg': u'non-zero return code', u'delta': u'0:00:02.690009', 'stdout_lines': [u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-', u' : disabled-repos, subscription-manager', u'Cleaning repos: epel lab-extras rhel-7-fcgi-ceph rhel-7-server-extras-rpms', u' : rhel-7-server-optional-rpms rhel-7-server-rpms', u' : rhel-ha-for-rhel-7-server-rpms', u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-', u' : disabled-repos, subscription-manager', u'Determining fastest mirrors'], u'end': u'2019-11-18 03:58:12.159671', '_ansible_no_log': False, u'cmd': u'rm -rf /var/cache/yum/*; yum clean all; yum makecache', u'start': u'2019-11-18 03:58:09.469662', u'warnings': [u"Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."], u'rc': 1, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'strip_empty_ends': True, u'_raw_params': u'rm -rf /var/cache/yum/*; yum clean all; yum makecache', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'rm -rf /var/cache/yum/*; yum clean all; yum makecache')Failure object was: {'mira066.front.sepia.ceph.com': {'stderr_lines': [u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] curl#18 - "transfer closed with 1758586 bytes remaining to read"', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/78364192654f81f335a53ad2e150be332036a37b75ee154c010f36692077f4bc-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'To address this issue please refer to the below knowledge base article ', u'', u'https://access.redhat.com/articles/1320623', u'', u"If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.", u'', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/7ef61dc594674fe902020c298a6e8a943b6e15ced7768d0f8bd82047964fb22e-updateinfo.xml.zck: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/78364192654f81f335a53ad2e150be332036a37b75ee154c010f36692077f4bc-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'Trying other mirror.', u'', u'', u' One of the configured repositories failed (Extra Packages for Enterprise Linux),', u" and yum doesn't have enough cached data to continue. At this point the only", u' safe thing yum can do is fail. There are a few ways to work "fix" this:', u'', u' 1. Contact the upstream for the repository and get them to fix the problem.', u'', u' 2. Reconfigure the baseurl/etc. for the repository, to point to a working', u' upstream. This is most often useful if you are using a newer', u' distribution release than is supported by the repository (and the', u' packages for the previous distribution release still work).', u'', u' 3. Run the command with the repository temporarily disabled', u' yum --disablerepo=epel ...', u'', u" 4. Disable the repository permanently, so yum won't use it by default. Yum", u' will then just ignore the repository until you permanently enable it', u' again or use --enablerepo for temporary usage:', u'', u' yum-config-manager --disable epel', u' or', u' subscription-manager repos --disable=epel', u'', u' 5. Configure the failing repository to be skipped, if it is unavailable.', u' Note that yum will try to contact the repo. when it runs most commands,', u' so will have to try and fail each time (and thus. yum will be be much', u' slower). If it is a very temporary problem though, this is often a nice', u' compromise:', u'', u' yum-config-manager --save --setopt=epel.skip_if_unavailable=true', u'', u'failure: repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.', u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found'], u'changed': True, u'stderr': u'http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] curl#18 - "transfer closed with 1758586 bytes remaining to read"\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/78364192654f81f335a53ad2e150be332036a37b75ee154c010f36692077f4bc-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nTo address this issue please refer to the below knowledge base article \n\nhttps://access.redhat.com/articles/1320623\n\nIf above article doesn\'t help to resolve this issue please open a ticket with Red Hat Support.\n\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/7ef61dc594674fe902020c298a6e8a943b6e15ced7768d0f8bd82047964fb22e-updateinfo.xml.zck: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/78364192654f81f335a53ad2e150be332036a37b75ee154c010f36692077f4bc-other.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found\nTrying other mirror.\n\n\n One of the configured repositories failed (Extra Packages for Enterprise Linux),\n and yum doesn\'t have enough cached data to continue. At this point the only\n safe thing yum can do is fail. There are a few ways to work "fix" this:\n\n 1. Contact the upstream for the repository and get them to fix the problem.\n\n 2. Reconfigure the baseurl/etc. for the repository, to point to a working\n upstream. This is most often useful if you are using a newer\n distribution release than is supported by the repository (and the\n packages for the previous distribution release still work).\n\n 3. Run the command with the repository temporarily disabled\n yum --disablerepo=epel ...\n\n 4. Disable the repository permanently, so yum won\'t use it by default. Yum\n will then just ignore the repository until you permanently enable it\n again or use --enablerepo for temporary usage:\n\n yum-config-manager --disable epel\n or\n subscription-manager repos --disable=epel\n\n 5. Configure the failing repository to be skipped, if it is unavailable.\n Note that yum will try to contact the repo. when it runs most commands,\n so will have to try and fail each time (and thus. yum will be be much\n slower). If it is a very temporary problem though, this is often a nice\n compromise:\n\n yum-config-manager --save --setopt=epel.skip_if_unavailable=true\n\nfailure: repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2 from epel: [Errno 256] No more mirrors to try.\nhttp://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/35487e8c13fe4d62fd897b5e8610b8de5436779d0800441eb38a7919f96ded32-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found', u'stdout': u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nCleaning repos: epel lab-extras rhel-7-fcgi-ceph rhel-7-server-extras-rpms\n : rhel-7-server-optional-rpms rhel-7-server-rpms\n : rhel-ha-for-rhel-7-server-rpms\nLoaded plugins: fastestmirror, langpacks, priorities, product-id, search-\n : disabled-repos, subscription-manager\nDetermining fastest mirrors', u'msg': u'non-zero return code', u'delta': u'0:00:19.995013', 'stdout_lines': [u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-', u' : disabled-repos, subscription-manager', u'Cleaning repos: epel lab-extras rhel-7-fcgi-ceph rhel-7-server-extras-rpms', u' : rhel-7-server-optional-rpms rhel-7-server-rpms', u' : rhel-ha-for-rhel-7-server-rpms', u'Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-', u' : disabled-repos, subscription-manager', u'Determining fastest mirrors'], u'end': u'2019-11-18 03:58:24.378963', '_ansible_no_log': False, u'cmd': u'rm -rf /var/cache/yum/*; yum clean all; yum makecache', u'start': u'2019-11-18 03:58:04.383950', u'warnings': [u"Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."], u'rc': 1, u'invocation': {u'module_args': {u'warn': True, u'executable': None, u'_uses_shell': True, u'strip_empty_ends': True, u'_raw_params': u'rm -rf /var/cache/yum/*; yum clean all; yum makecache', u'removes': None, u'argv': None, u'creates': None, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'rm -rf /var/cache/yum/*; yum clean all; yum makecache')
  • flavor:
  • job_id: 4513993
  • kernel:
    • sha1: distro
    • kdb: True
  • last_in_suite: False
  • machine_type: mira
  • name: kchai-2019-11-17_01:29:22-rados-wip-kefu-testing-2019-11-17-0135-distro-basic-mira
  • nuke_on_error: True
  • os_type: rhel
  • os_version: 7.7
  • overrides:
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(REQUEST_STUCK\)
        • \(OSD_SLOW_PING_TIME
        • overall HEALTH_
        • \(OSDMAP_FLAGS\)
        • \(OSD_
        • \(PG_
        • \(POOL_
        • \(CACHE_POOL_
        • \(SMALLER_PGP_NUM\)
        • \(OBJECT_
        • \(SLOW_OPS\)
        • \(REQUEST_SLOW\)
        • \(TOO_FEW_PGS\)
        • slow requests
      • mon_bind_msgr2: False
      • fs: xfs
      • conf:
        • global:
          • ms inject delay max: 1
          • ms inject delay type: osd
          • osd_pool_default_min_size: 2
          • ms bind msgr2: False
          • ms inject delay probability: 0.005
          • mon client directed command retry: 5
          • ms type: async
          • ms inject socket failures: 2500
          • osd_recovery_max_active: 10
          • osd_pool_default_size: 2
          • osd_recovery_max_single_start: 10
          • ms inject internal delays: 0.002
        • mon:
          • debug mon: 20
          • debug paxos: 20
          • debug ms: 1
        • osd:
          • osd op queue cut off: debug_random
          • debug ms: 1
          • debug journal: 20
          • debug osd: 25
          • osd objectstore: filestore
          • osd debug verify cached snaps: True
          • osd debug verify missing on start: True
          • osd op queue: debug_random
          • osd sloppy crc: True
          • debug filestore: 20
          • osd shutdown pgref assert: True
          • osd backoff on peering: True
      • sha1: 70968846f95926b5f8ae4f251788ce3818969018
    • ceph-deploy:
      • fs: xfs
      • filestore: True
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • osd sloppy crc: True
          • osd objectstore: filestore
    • workunit:
      • sha1: 70968846f95926b5f8ae4f251788ce3818969018
      • branch: wip-kefu-testing-2019-11-17-0135
    • install:
      • ceph:
        • sha1: 70968846f95926b5f8ae4f251788ce3818969018
    • admin_socket:
      • branch: wip-kefu-testing-2019-11-17-0135
  • owner: scheduled_kchai@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
    • ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
  • sentry_event:
  • status: fail
  • success: False
  • branch: wip-kefu-testing-2019-11-17-0135
  • seed:
  • sha1: 70968846f95926b5f8ae4f251788ce3818969018
  • subset:
  • suite:
  • suite_branch: wip-kefu-testing-2019-11-17-0135
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 70968846f95926b5f8ae4f251788ce3818969018
  • targets:
    • mira066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDm24gcGL5TKSHFs6Nb0nITV52TU2YZ4yos1oAwVOIEcJA36ieAV6FS5Q5OviFQXy3SOdltAwi89oYz95qmGe85CEyi1o9ohebnCNXi7Bgj1z3CBEVtchQ35aM6JBRLzm/KPK5O62HKCkpV2UBdsIsNfOst8Zt5ytNSIBFpDCO6QC5dL0XuxIJBrEWhXXC7gpIUkP2EbcmxJ5Gqy7Y+USyZ3OQWes+4fHbXF2gmKV3Yf+6zht2HyzqDQag1wrMYNLBdgOAT9coUEtHZNI9wKCR5wAclf1o4d1D+DXXHbvlm1zG63mi6FgToFSxj5CAAXVlcWONnLYbWrH/3gMkbY1Qj
    • mira083.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD8PrptxDr4++mf3aUjWr5S8uHhd8Kwso1/11UqRC2yRryJEinzxpPpOoTd4tV0l1mIdQXjv8KYtNy68gfF1D3RAEIj/p4RKgUHDRE4ZhfnhZGpwR24jkRk1g0lOLhlAwm7vYx2BZoIPHV+mdMWo7k84r08k8ivi3h/8VRFuiCWkJDsM5JYoePn/OLv1eOEw9AwJHRPpjsmnV72UcJ8ItHiwQOWBomRORkH3M/r7xlP6ugaOUC3DAquLYxP4qql1yuZEpjoleOAhlBNIonmSatvS71BMepPtfSnv9bfIrIUoqbk76wicQug5Mp172azxCikRAqkiMGLO26w26zfPRyf
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 2
      • mira
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • sha1: distro
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
    • ceph:
    • exec:
      • mon.a:
        • while ! ceph balancer status ; do sleep 1 ; done
        • ceph osd set-require-min-compat-client luminous
        • ceph balancer mode upmap
        • ceph balancer on
    • exec:
      • client.0:
        • sudo ceph osd pool create low_tier 4
    • rados:
      • op_weights:
        • read: 100
        • write: 100
        • delete: 50
        • copy_from: 50
      • ops: 4000
      • clients:
        • client.0
      • set_redirect: True
      • objects: 500
      • low_tier_pool: low_tier
  • teuthology_branch: master
  • verbose: False
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2019-11-17 01:31:26
  • started: 2019-11-18 03:49:37
  • updated: 2019-11-18 03:59:36
  • status_class: danger
  • runtime: 0:09:59
  • wait_time: 0:06:55