yuriw-2020-04-28_15:14:23-rgw-wip-yuri7-testing-2020-04-27-1705-octopus-distro-basic-smithi

Command failed on smithi203 with status 1: 'cd /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-venv/bin/activate && tox -e venv --notest'

rgw multisite, rgw multisite pubsub, test failures

Command failed on smithi167 with status 1: 'cd /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-venv/bin/activate && tox -e venv --notest'

"2020-04-28T20:14:48.740529+0000 mon.a (mon.0) 39 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log

rgw multisite, rgw multisite pubsub, test failures

Command failed on smithi149 with status 1: 'cd /home/ubuntu/cephtest/keystone && source /home/ubuntu/cephtest/tox-venv/bin/activate && tox -e venv --notest'

-1.25.17-1.el8.x86_64', u' - cannot install the best update candidate for package subscription-manager-cockpit-1.25.17-1.el8.noarch', u' - cannot install the best update candidate for package dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64'], u'changed': True, u'stderr': u'\n Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed\n - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed\n - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64\n - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64\n - cannot install the best candidate for the job\nError: \n Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed\n - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed\n - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed\n - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64\n - cannot install the best update candidate for package subscription-manager-plugin-container-1.25.17-1.el8.x86_64\n - cannot install the best update candidate for package subscription-manager-cockpit-1.25.17-1.el8.noarch\n - cannot install the best update candidate for package dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64', u'stdout': u"Updating Subscription Management repositories.\nCopr repo for ceph-el8 owned by ktdreyer 64 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 148 kB/s | 4.7 kB 00:00 \nlab-extras 1.0 MB/s | 2.9 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling Groups:\n Base \nSkipping packages with conflicts:\n(add '--best --allowerasing' to command line to force their upgrade):\n dnf-plugin-subscription-manager\n x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 260 k\n dnf-plugin-subscription-manager\n x86_64 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 278 k\nSkipping packages with broken dependencies:\n subscription-manager x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M\n subscription-manager-cockpit\n noarch 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M\n\nTransaction Summary\n================================================================================\nSkip 4 Packages\n\nComplete!\nUpdating Subscription Management repositories.\nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 \n(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)", u'msg': u'non-zero return code', u'delta': u'0:00:07.285239', 'stdout_lines': [u'Updating Subscription Management repositories.', u'Copr repo for ceph-el8 owned by ktdreyer 64 kB/s | 3.3 kB 00:00 ', u'Extra Packages for Enterprise Linux 148 kB/s | 4.7 kB 00:00 ', u'lab-extras 1.0 MB/s | 2.9 kB 00:00 ', u'Red Hat Enterprise Linux 8 for x86_64 - BaseOS 80 kB/s | 2.4 kB 00:00 ', u'Red Hat Enterprise Linux 8 for x86_64 - AppStre 96 kB/s | 2.8 kB 00:00 ', u'Dependencies resolved.', u'================================================================================', u' Package Arch Version Repository Size', u'================================================================================', u'Installing Groups:', u' Base ', u'Skipping packages with conflicts:', u"(add '--best --allowerasing' to command line to force their upgrade):", u' dnf-plugin-subscription-manager', u' x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 260 k', u' dnf-plugin-subscription-manager', u' x86_64 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 278 k', u'Skipping packages with broken dependencies:', u' subscription-manager x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M', u' subscription-manager-cockpit', u' noarch 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M', u'', u'Transaction Summary', u'================================================================================', u'Skip 4 Packages', u'', u'Complete!', u'Updating Subscription Management repositories.', u'Red Hat Enterprise Linux 8 for x86_64 - BaseOS 81 kB/s | 2.4 kB 00:00 ', u'Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 ', u"(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)"], u'end': u'2020-04-28 19:59:23.636000', '_ansible_no_log': False, u'cmd': u'dnf -y groupinstall base --nobest && dnf -y group upgrade base', u'start': u'2020-04-28 19:59:16.350761', u'warnings': [u"Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."], u'rc': 1, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'strip_empty_ends': True, u'_raw_params': u'dnf -y groupinstall base --nobest && dnf -y group upgrade base', u'removes': None, u'argv': None, u'warn': True, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'dnf -y groupinstall base --nobest && dnf -y group upgrade base')Failure object was: {'smithi139.front.sepia.ceph.com': {'stderr_lines': [u'', u' Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed', u' - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed', u' - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed', u' - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64', u' - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64', u' - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64', u' - cannot install the best candidate for the job', u'Error: ', u' Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed', u' - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed', u' - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed', u' - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64', u' - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64', u' - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64', u' - cannot install the best update candidate for package subscription-manager-plugin-container-1.25.17-1.el8.x86_64', u' - cannot install the best update candidate for package subscription-manager-cockpit-1.25.17-1.el8.noarch', u' - cannot install the best update candidate for package dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64'], u'changed': True, u'stderr': u'\n Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed\n - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed\n - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64\n - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64\n - cannot install the best candidate for the job\nError: \n Problem: package subscription-manager-cockpit-1.26.16-1.el8.noarch requires subscription-manager, but none of the providers can be installed\n - package subscription-manager-1.25.17-1.el8.x86_64 requires dnf-plugin-subscription-manager = 1.25.17, but none of the providers can be installed\n - package subscription-manager-1.23.8-35.el8.x86_64 requires dnf-plugin-subscription-manager = 1.23.8, but none of the providers can be installed\n - package subscription-manager-1.26.16-1.el8.x86_64 obsoletes subscription-manager-plugin-container provided by subscription-manager-plugin-container-1.25.17-1.el8.x86_64\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64\n - cannot install both dnf-plugin-subscription-manager-1.26.16-1.el8.x86_64 and dnf-plugin-subscription-manager-1.23.8-35.el8.x86_64\n - cannot install the best update candidate for package subscription-manager-plugin-container-1.25.17-1.el8.x86_64\n - cannot install the best update candidate for package subscription-manager-cockpit-1.25.17-1.el8.noarch\n - cannot install the best update candidate for package dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64', u'stdout': u"Updating Subscription Management repositories.\nCopr repo for ceph-el8 owned by ktdreyer 63 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 122 kB/s | 4.7 kB 00:00 \nlab-extras 1.2 MB/s | 2.9 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling Groups:\n Base \nSkipping packages with conflicts:\n(add '--best --allowerasing' to command line to force their upgrade):\n dnf-plugin-subscription-manager\n x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 260 k\n dnf-plugin-subscription-manager\n x86_64 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 278 k\nSkipping packages with broken dependencies:\n subscription-manager x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M\n subscription-manager-cockpit\n noarch 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M\n\nTransaction Summary\n================================================================================\nSkip 4 Packages\n\nComplete!\nUpdating Subscription Management repositories.\nRed Hat Enterprise Linux 8 for x86_64 - AppStre 89 kB/s | 2.8 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 \n(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)", u'msg': u'non-zero return code', u'delta': u'0:00:07.538655', 'stdout_lines': [u'Updating Subscription Management repositories.', u'Copr repo for ceph-el8 owned by ktdreyer 63 kB/s | 3.3 kB 00:00 ', u'Extra Packages for Enterprise Linux 122 kB/s | 4.7 kB 00:00 ', u'lab-extras 1.2 MB/s | 2.9 kB 00:00 ', u'Red Hat Enterprise Linux 8 for x86_64 - AppStre 90 kB/s | 2.8 kB 00:00 ', u'Red Hat Enterprise Linux 8 for x86_64 - BaseOS 79 kB/s | 2.4 kB 00:00 ', u'Dependencies resolved.', u'================================================================================', u' Package Arch Version Repository Size', u'================================================================================', u'Installing Groups:', u' Base ', u'Skipping packages with conflicts:', u"(add '--best --allowerasing' to command line to force their upgrade):", u' dnf-plugin-subscription-manager', u' x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 260 k', u' dnf-plugin-subscription-manager', u' x86_64 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 278 k', u'Skipping packages with broken dependencies:', u' subscription-manager x86_64 1.23.8-35.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M', u' subscription-manager-cockpit', u' noarch 1.26.16-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M', u'', u'Transaction Summary', u'================================================================================', u'Skip 4 Packages', u'', u'Complete!', u'Updating Subscription Management repositories.', u'Red Hat Enterprise Linux 8 for x86_64 - AppStre 89 kB/s | 2.8 kB 00:00 ', u'Red Hat Enterprise Linux 8 for x86_64 - BaseOS 78 kB/s | 2.4 kB 00:00 ', u"(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)"], u'end': u'2020-04-28 19:59:23.722365', '_ansible_no_log': False, u'cmd': u'dnf -y groupinstall base --nobest && dnf -y group upgrade base', u'start': u'2020-04-28 19:59:16.183710', u'warnings': [u"Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."], u'rc': 1, u'invocation': {u'module_args': {u'creates': None, u'executable': None, u'_uses_shell': True, u'strip_empty_ends': True, u'_raw_params': u'dnf -y groupinstall base --nobest && dnf -y group upgrade base', u'removes': None, u'argv': None, u'warn': True, u'chdir': None, u'stdin_add_newline': True, u'stdin': None}}}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'dnf -y groupinstall base --nobest && dnf -y group upgrade base')