nojha-2020-04-29_18:44:55-rados:singleton-nomsgr-master-distro-basic-smithi

"2020-04-29T20:15:02.951141+0000 mon.a (mon.0) 135 : cluster [WRN] Health check failed: Degraded data redundancy: 254/13190 objects degraded (1.926%), 5 pgs degraded (PG_DEGRADED)" in cluster log

"2020-04-29T20:14:13.862182+0000 mon.a (mon.0) 135 : cluster [WRN] Health check failed: Degraded data redundancy: 561/22018 objects degraded (2.548%), 7 pgs degraded (PG_DEGRADED)" in cluster log

Command failed with status 2: 'ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml'

Command failed with status 2: 'ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml'

"2020-04-29T20:13:09.985546+0000 mon.a (mon.0) 119 : cluster [WRN] Health check failed: Degraded data redundancy: 42/4354 objects degraded (0.965%), 4 pgs degraded (PG_DEGRADED)" in cluster log

"2020-04-29T20:14:38.044668+0000 mon.a (mon.0) 119 : cluster [WRN] Health check failed: Degraded data redundancy: 68/2972 objects degraded (2.288%), 7 pgs degraded (PG_DEGRADED)" in cluster log

Command failed with status 2: 'ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml'

Command failed on smithi168 with status 1: 'sudo yum -y install ceph-test'

{'Failure object was': {'smithi060.front.sepia.ceph.com': {'msg': 'non-zero return code', 'cmd': 'dnf -y groupinstall base --nobest && dnf -y group upgrade base', 'stdout': "Updating Subscription Management repositories.\nCopr repo for ceph-el8 owned by ktdreyer 59 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 148 kB/s | 4.7 kB 00:00 \nlab-extras 1.3 MB/s | 2.9 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 59 kB/s | 2.4 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling Groups:\n Base \nSkipping packages with conflicts:\n(add '--best --allowerasing' to command line to force their upgrade):\n dnf-plugin-subscription-manager\n x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 260 k\n dnf-plugin-subscription-manager\n x86_64 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 278 k\nSkipping packages with broken dependencies:\n subscription-manager x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M\n subscription-manager-cockpit\n noarch 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M\n\nTransaction Summary\n================================================================================\nSkip 4 Packages\n\nComplete!\nUpdating Subscription Management repositories.\nRed Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 \n(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)", 'stderr': '\\n Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed\\n - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed\\n - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed\\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64\\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64\\n - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64\\n - cannot install the best candidate for the job\\nError: \\n Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed\\n - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed\\n - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed\\n - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64\\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64\\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64\\n - cannot install the best update candidate for package subscription-manager-plugin-container-1.25.17-1.el8.x86_64\\n - cannot install the best update candidate for package subscription-manager-cockpit-1.25.17-1.el8.noarch\\n - cannot install the best update candidate for package dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64', 'rc': 1, 'start': '2020-04-29 20:08:26.623971', 'end': '2020-04-29 20:08:34.904813', 'delta': '0:00:08.280842', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'dnf -y groupinstall base --nobest && dnf -y group upgrade base', '_uses_shell': True, 'warn': True, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': 'None', 'chdir': 'None', 'executable': 'None', 'creates': 'None', 'removes': 'None', 'stdin': 'None'}}, 'warnings': ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."], 'stdout_lines': ['Updating Subscription Management repositories.', 'Copr repo for ceph-el8 owned by ktdreyer 59 kB/s | 3.3 kB 00:00 ', 'Extra Packages for Enterprise Linux 148 kB/s | 4.7 kB 00:00 ', 'lab-extras 1.3 MB/s | 2.9 kB 00:00 ', 'Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 ', 'Red Hat Enterprise Linux 8 for x86_64 - BaseOS 59 kB/s | 2.4 kB 00:00 ', 'Dependencies resolved.', '================================================================================', ' Package Arch Version Repository Size', '================================================================================', 'Installing Groups:', ' Base ', 'Skipping packages with conflicts:', "(add '--best --allowerasing' to command line to force their upgrade):", ' dnf-plugin-subscription-manager', ' x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 260 k', ' dnf-plugin-subscription-manager', ' x86_64 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 278 k', 'Skipping packages with broken dependencies:', ' subscription-manager x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M', ' subscription-manager-cockpit', ' noarch 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M', '', 'Transaction Summary', '================================================================================', 'Skip 4 Packages', '', 'Complete!', 'Updating Subscription Management repositories.', 'Red Hat Enterprise Linux 8 for x86_64 - AppStre 73 kB/s | 2.8 kB 00:00 ', 'Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 kB/s | 2.4 kB 00:00 ', "(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)"], 'stderr_lines': ['', ' Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed', ' - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed', ' - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed', ' - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64', ' - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64', ' - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64', ' - cannot install the best candidate for the job', 'Error: ', ' Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed', ' - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed', ' - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed', ' - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64', ' - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64', ' - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64', ' - cannot install the best update candidate for package subscription-manager-plugin-container-1.25.17-1.el8.x86_64', ' - cannot install the best update candidate for package subscription-manager-cockpit-1.25.17-1.el8.noarch', ' - cannot install the best update candidate for package dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64'], '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'dnf -y groupinstall base --nobest && dnf -y group upgrade base')"}

Command failed on smithi189 with status 1: 'sudo yum -y install ceph-test'

"2020-04-29T20:14:12.959121+0000 mon.a (mon.0) 117 : cluster [WRN] Health check failed: Degraded data redundancy: 62/2812 objects degraded (2.205%), 8 pgs degraded (PG_DEGRADED)" in cluster log

Command failed with status 2: 'ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi109.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml'

Command failed with status 2: 'ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml'

Command failed with status 2: 'ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml'

Command failed with status 2: 'ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml'

Command failed on smithi145 with status 1: 'sudo yum -y install ceph'

"2020-04-29T20:14:09.243861+0000 mon.a (mon.0) 121 : cluster [WRN] Health check failed: Degraded data redundancy: 59/2666 objects degraded (2.213%), 7 pgs degraded (PG_DEGRADED)" in cluster log