sage-2019-05-11_01:08:55-rados-wip-sage-testing-2019-05-10-1632-distro-basic-smithi

"2019-05-12 00:41:24.897792 mon.b (mon.1) 55 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

SSH connection to smithi201 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

"2019-05-12 09:00:00.108564 mon.a (mon.1) 172 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

"2019-05-12 01:31:51.178956 mon.a (mon.1) 471 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

"2019-05-12 07:18:06.054845 mon.a (mon.1) 162 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

"2019-05-12 08:06:18.762439 mon.c (mon.1) 28 : cluster [WRN] Health check failed: 1/3 mons down, quorum c,a (MON_DOWN)" in cluster log

"2019-05-12 10:21:56.113267 mon.a (mon.1) 277 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

"2019-05-12 10:19:11.310116 mon.b (mon.1) 62 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

SSH connection to smithi017 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

"2019-05-12 10:57:50.092435 mon.a (mon.1) 5645 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

"2019-05-12 10:55:14.738535 mon.c (mon.1) 208 : cluster [WRN] Health check failed: 1/3 mons down, quorum c,a (MON_DOWN)" in cluster log

Command failed on smithi046 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

"2019-05-12 10:50:06.281737 mon.a (mon.1) 18 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

"2019-05-12 10:31:37.799383 mon.b (mon.1) 36 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

SSH connection to smithi022 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

"2019-05-12 11:11:37.081716 mon.a (mon.1) 132 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

SSH connection to smithi002 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

SSH connection to smithi105 was lost: 'uname -r'

"2019-05-12 11:36:33.156682 mon.a (mon.1) 157 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

SSH connection to smithi083 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

SSH connection to smithi012 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

Command failed on smithi067 with status 124: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_health_to_clog=true"

SSH connection to smithi080 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

Command failed on smithi142 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1'

"2019-05-12 13:04:30.417222 mon.a (mon.1) 65 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

SSH connection to smithi158 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

Command failed on smithi158 with status 100: 'sudo apt-get clean'

SSH connection to smithi005 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

SSH connection to smithi095 was lost: 'sudo apt-get update'

Could not reconnect to ubuntu@smithi095.front.sepia.ceph.com

SSH connection to smithi003 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

SSH connection to smithi043 was lost: 'sudo rm /etc/logrotate.d/ceph-test.conf'

SSH connection to smithi196 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

Command failed on smithi080 with status 100: 'sudo apt-get clean'

reached maximum tries (60) after waiting for 900 seconds

"2019-05-12 15:51:27.415172 mon.a (mon.1) 493 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

reached maximum tries (60) after waiting for 900 seconds

'ssh_keyscan smithi023.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds

"2019-05-12 22:22:37.625907 mon.a (mon.1) 466 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

SSH connection to smithi008 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

SSH connection to smithi017 was lost: 'rpm -q kernel --last | head -n 1'

failed to install new kernel version within timeout

Could not reconnect to ubuntu@smithi105.front.sepia.ceph.com

Command failed on smithi017 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

SSH connection to smithi050 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

Command failed (workunit test misc/test-ceph-helpers.sh) on smithi130 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=dc46926df33c49ecb4236b8bc2f33597e45430bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh'

g transaction check\n---> Package qemu-img.x86_64 10:1.5.3-156.el7_5.3 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-156.el7_5.3 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-156.el7_5.3 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librbd1 x86_64 1:0.94.5-2.el7 @base 5.5 M\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-156.el7_5.3 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-156.el7_5.3 @updates 5.8 M\n qemu-kvm-common x86_64 10:1.5.3-156.el7_5.3 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package (+3 Dependent packages)\n\nInstalled size: 14 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n'], u'failed': True, u'rc': -11, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'librbd1'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, u'msg': u''}, {'_ansible_parsed': True, u'changed': True, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package librados2.x86_64 1:0.94.5-2.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-156.el7_5.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 1:librbd1-0.94.5-2.el7.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-156.el7_5.3.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-156.el7_5.3.x86_64\n--> Processing Dependency: librados2 = 1:0.94.5-2.el7 for package: 1:librbd1-0.94.5-2.el7.x86_64\n--> Running transaction check\n---> Package librbd1.x86_64 1:0.94.5-2.el7 will be erased\n---> Package qemu-img.x86_64 10:1.5.3-156.el7_5.3 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-156.el7_5.3 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-156.el7_5.3 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n librados2 x86_64 1:0.94.5-2.el7 @base 5.1 M\nRemoving for dependencies:\n librbd1 x86_64 1:0.94.5-2.el7 @base 5.5 M\n qemu-img x86_64 10:1.5.3-156.el7_5.3 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-156.el7_5.3 @updates 5.8 M\n qemu-kvm-common x86_64 10:1.5.3-156.el7_5.3 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package (+4 Dependent packages)\n\nInstalled size: 19 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-156.el7_5.3.x86_64 1/5 \n Erasing : 10:qemu-img-1.5.3-156.el7_5.3.x86_64 2/5 \n Erasing : 10:qemu-kvm-common-1.5.3-156.el7_5.3.x86_64 3/5 \n Erasing : 1:librbd1-0.94.5-2.el7.x86_64 4/5 \n Erasing : 1:librados2-0.94.5-2.el7.x86_64 5/5 \n Verifying : 1:librbd1-0.94.5-2.el7.x86_64 1/5 \n Verifying : 10:qemu-kvm-1.5.3-156.el7_5.3.x86_64 2/5 \n Verifying : 10:qemu-img-1.5.3-156.el7_5.3.x86_64 3/5 \n Verifying : 1:librados2-0.94.5-2.el7.x86_64 4/5 \n Verifying : 10:qemu-kvm-common-1.5.3-156.el7_5.3.x86_64 5/5 \n\nRemoved:\n librados2.x86_64 1:0.94.5-2.el7 \n\nDependency Removed:\n librbd1.x86_64 1:0.94.5-2.el7 \n qemu-img.x86_64 10:1.5.3-156.el7_5.3 \n qemu-kvm.x86_64 10:1.5.3-156.el7_5.3 \n qemu-kvm-common.x86_64 10:1.5.3-156.el7_5.3 \n\nComplete!\n'], 'item': u'librados2', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'librados2'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u'There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don\'t work you\'ll have to try removing/installing packages by hand (maybe package-cleanup can help).\n'}, {'_ansible_parsed': True, u'changed': True, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 213 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 1/1 \n\nRemoved:\n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nComplete!\n'], 'item': u'mod_fastcgi', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'mod_fastcgi'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u'There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don\'t work you\'ll have to try removing/installing packages by hand (maybe package-cleanup can help).\n'}, {'_ansible_parsed': True, u'changed': True, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 1.3 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 1/1 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n\nComplete!\n'], 'item': u'iozone', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'iozone'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u'Existing lock /var/run/yum.pid: another copy is running as pid 27071.\nAnother app is currently holding the yum lock; waiting for it to exit...\n The other application is: yum-complete-tr\n Memory : 91 M RSS (408 MB VSZ)\n Started: Sun May 12 22:29:03 2019 - 00:11 ago\n State : Running, pid: 27071\n'}]}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 218, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 190, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 28, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 225, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 225, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 217, in represent_list return self.represent_sequence(u'tag:yaml.org,2002:seq', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 101, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 225, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 67, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 249, in represent_undefined raise RepresenterError("cannot represent an object: %s" % data)RepresenterError: cannot represent an object: ceph

SSH connection to smithi017 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

ignore_errors': None, 'item': u'python-ceph'}, {'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'python-flask is not installed'], 'item': u'python-flask', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'python-flask'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u''}, {'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'python-requests is not installed'], 'item': u'python-requests', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'python-requests'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u''}, {'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'boost-random is not installed'], 'item': u'boost-random', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'boost-random'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u''}, {'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'python-urllib3 is not installed'], 'item': u'python-urllib3', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'python-urllib3'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u''}, {'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'python-babel is not installed'], 'item': u'python-babel', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'python-babel'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u''}, {'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'hdparm is not installed'], 'item': u'hdparm', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'hdparm'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u''}, {'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'python-markupsafe is not installed'], 'item': u'python-markupsafe', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'python-markupsafe'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u''}, {'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'python-werkzeug is not installed'], 'item': u'python-werkzeug', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'python-werkzeug'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u''}, {'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': False, '_ansible_item_result': True, u'results': [u'python-itsdangerous is not installed'], 'item': u'python-itsdangerous', u'rc': 0, u'invocation': {u'module_args': {u'allow_downgrade': False, u'name': [u'python-itsdangerous'], u'list': None, u'disable_gpg_check': False, u'conf_file': None, u'install_repoquery': True, u'state': u'absent', u'disablerepo': None, u'update_cache': False, u'enablerepo': None, u'exclude': None, u'security': False, u'validate_certs': True, u'installroot': u'/', u'skip_broken': False}}, '_ansible_ignore_errors': None, u'msg': u''}]}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 218, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 190, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 28, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 225, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 225, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 217, in represent_list return self.represent_sequence(u'tag:yaml.org,2002:seq', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 101, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 225, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 67, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 249, in represent_undefined raise RepresenterError("cannot represent an object: %s" % data)RepresenterError: cannot represent an object: boost-random

"2019-05-12 22:57:37.355059 mon.b (mon.1) 63 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

SSH connection to smithi035 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

Command failed on smithi035 with status 100: 'sudo apt-get clean'

SSH connection to smithi035 was lost: 'uname -r'

Could not reconnect to ubuntu@smithi035.front.sepia.ceph.com

"2019-05-12 23:03:31.820884 mon.c (mon.1) 66 : cluster [WRN] Health check failed: 1/3 mons down, quorum c,a (MON_DOWN)" in cluster log

'ssh_keyscan smithi105.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds

"2019-05-12 23:00:41.004226 mon.b (mon.1) 78 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

{'smithi035.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['E: Could not get lock /var/cache/apt/archives/lock - open (11: Resource temporarily unavailable)', 'E: Unable to lock directory /var/cache/apt/archives/'], 'cmd': ['apt-get', 'clean'], 'end': '2019-05-12 22:52:18.741946', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'apt-get clean', 'removes': None, 'warn': True, '_uses_shell': False, 'stdin': None}}, 'start': '2019-05-12 22:52:18.722676', 'delta': '0:00:00.019270', 'stderr': 'E: Could not get lock /var/cache/apt/archives/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock directory /var/cache/apt/archives/', 'rc': 100, 'msg': 'non-zero return code', 'stdout_lines': [], 'warnings': ['Consider using apt module rather than running apt-get']}}

SSH connection to smithi035 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

Command failed on smithi095 with status 124: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_health_to_clog=true"

SSH connection to smithi002 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

"2019-05-12 23:03:18.848987 mon.a (mon.1) 21 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

"2019-05-12 23:32:48.784921 mon.a (mon.1) 479 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

"2019-05-13 00:03:31.071108 mon.b (mon.1) 73 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

SSH connection to smithi151 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"