pdonnell-2018-05-04_23:11:19-multimds-wip-pdonnell-testing-20180504.211521-testing-basic-smithi

2477076

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi010 smithi116 smithi163

multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477078

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi056 smithi105 smithi145

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 04:20:47.156051 mon.a mon.0 172.21.15.56:6789/0 1108 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477084

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi094 smithi150 smithi161

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}

Command failed (workunit test suites/fsstress.sh) on smithi150 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=wip-pdonnell-testing-20180504.211521 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

2477086

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi100 smithi173 smithi194

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 04:51:35.230993 mon.a mon.0 172.21.15.173:6789/0 199 : cluster [WRN] MDS health message (mds.0): 2 slow requests are blocked > 30 sec" in cluster log

2477091

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi073 smithi157 smithi163

multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477093

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi050 smithi073 smithi093

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 04:27:49.750863 mon.b mon.0 172.21.15.50:6789/0 516 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477099

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi119 smithi158 smithi203

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}

reached maximum tries (50) after waiting for 300 seconds

2477103

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi071 smithi146 smithi169

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477110

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi033 smithi088 smithi155

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml}

"2018-05-05 04:55:39.843592 mon.a mon.0 172.21.15.88:6789/0 217 : cluster [WRN] MDS health message (mds.1): 9 slow requests are blocked > 30 sec" in cluster log

2477116

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi040 smithi070 smithi174

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 04:54:55.940944 mon.a mon.0 172.21.15.40:6789/0 327 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

2477117

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi049 smithi059 smithi113

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477121

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi031 smithi077 smithi198

multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477123

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi051 smithi055 smithi140

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 04:37:03.586118 mon.a mon.0 172.21.15.51:6789/0 1254 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477127

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi048 smithi065 smithi197

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477131

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi056 smithi105 smithi145

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 04:52:39.178624 mon.a mon.0 172.21.15.56:6789/0 305 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log

2477133

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi018 smithi111 smithi143

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477147

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi035 smithi157 smithi193

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477149

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi027 smithi062 smithi182

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml}

Command failed (workunit test suites/pjd.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=wip-pdonnell-testing-20180504.211521 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

2477152

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi112 smithi121 smithi205

multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477163

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi044 smithi069 smithi076

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477166

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi013 smithi135 smithi151

multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477174

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi031 smithi077 smithi198

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml}

reached maximum tries (50) after waiting for 300 seconds

2477176

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi092 smithi132 smithi136

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 05:07:41.062867 mon.b mon.0 172.21.15.92:6789/0 185 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

2477193

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi008 smithi010 smithi179

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477196

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi024 smithi062 smithi134

multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477197

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi114 smithi132 smithi196

multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477198

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi015 smithi081 smithi197

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 05:27:11.316675 mon.a mon.0 172.21.15.15:6789/0 949 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477204

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi027 smithi157 smithi182

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml}

reached maximum tries (50) after waiting for 300 seconds

2477223

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi112 smithi121 smithi163

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

u'attr', u'dbench', u'bonnie++', u'valgrind', u'python-nose', u'ant', u'libtool', u'automake', u'gettext', u'uuid-dev', u'libacl1-dev', u'bc', u'xfsdump', u'xfslibs-dev', u'libattr1-dev', u'quota', u'libcap2-bin', u'libncurses5-dev', u'lvm2', u'vim', u'pdsh', u'collectl', u'blktrace', u'python-numpy', u'python-matplotlib', u'genisoimage', u'libjson-xs-perl', u'xml-twig-tools', u'default-jdk', u'junit4', u'tgt', u'open-iscsi', u'cifs-utils', u'ipcalc', u'nfs-common', u'nfs-kernel-server', u'software-properties-common'], u'purge': False, u'allow_unauthenticated': False, u'state': u'present', u'autoremove': False, u'update_cache': None, u'default_release': None, u'only_upgrade': False, u'cache_valid_time': 0, u'deb': None, u'install_recommends': None}}, 'item': [u'libboost-thread1.58.0', u'mpich', u'qemu-system-x86', u'lttng-tools', u'libcrypto++9v5', u'libtool-bin', u'libfcgi0ldbl', u'python-virtualenv', u'python-dev', u'libevent-dev', u'perl', u'libwww-perl', u'lsb-release', u'build-essential', u'sysstat', u'gdb', u'python-configobj', u'python-gevent', u'libedit2', u'cryptsetup-bin', u'xfsprogs', u'gdisk', u'parted', u'libuuid1', u'btrfs-tools', u'libatomic-ops-dev', u'git-core', u'attr', u'dbench', u'bonnie++', u'valgrind', u'python-nose', u'ant', u'libtool', u'automake', u'gettext', u'uuid-dev', u'libacl1-dev', u'bc', u'xfsdump', u'xfslibs-dev', u'libattr1-dev', u'quota', u'libcap2-bin', u'libncurses5-dev', u'lvm2', u'vim', u'pdsh', u'collectl', u'blktrace', u'python-numpy', u'python-matplotlib', u'genisoimage', u'libjson-xs-perl', u'xml-twig-tools', u'default-jdk', u'junit4', u'tgt', u'open-iscsi', u'cifs-utils', u'ipcalc', u'nfs-common', u'nfs-kernel-server', u'software-properties-common'], u'stderr': u'E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?\n', u'rc': 100, u'msg': u'\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" --force-yes install \'qemu-system-x86\'\' failed: E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)\nE: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?\n', 'stdout_lines': [], u'cache_update_time': 1525500328}]}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 218, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 190, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 28, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 225, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 225, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 217, in represent_list return self.represent_sequence(u'tag:yaml.org,2002:seq', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 101, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 225, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 123, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 57, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 217, in represent_list return self.represent_sequence(u'tag:yaml.org,2002:seq', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 101, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 67, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 249, in represent_undefined raise RepresenterError("cannot represent an object: %s" % data)RepresenterError: cannot represent an object: libboost-thread1.58.0

2477228

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi050 smithi069 smithi105

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 07:38:26.865208 mon.a mon.0 172.21.15.50:6789/0 952 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477236

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi081 smithi130 smithi197

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 06:14:24.322943 mon.b mon.0 172.21.15.81:6789/0 182 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

2477249

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi100 smithi173 smithi194

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}

reached maximum tries (50) after waiting for 300 seconds

2477253

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi060 smithi116 smithi204

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477258

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi038 smithi050 smithi171

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 06:15:16.365850 mon.a mon.0 172.21.15.38:6789/0 975 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477266

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi022 smithi121 smithi163

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 06:25:27.403696 mon.a mon.0 172.21.15.121:6789/0 184 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

2477267

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi004 smithi134 smithi138

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477273

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi072 smithi092 smithi196

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 06:23:41.538397 mon.b mon.0 172.21.15.72:6789/0 1071 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477281

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi029 smithi087 smithi192

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 07:09:49.245451 mon.b mon.0 172.21.15.29:6789/0 155 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

2477283

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi056 smithi105 smithi145

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477288

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi101 smithi164 smithi202

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 07:45:40.090459 mon.a mon.0 172.21.15.101:6789/0 1372 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 31 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477294

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi023 smithi116 smithi133

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}

reached maximum tries (50) after waiting for 300 seconds

2477297

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi081 smithi130 smithi197

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477303

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi019 smithi085 smithi104

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 06:49:12.709656 mon.b mon.0 172.21.15.19:6789/0 689 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477362

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi022 smithi121 smithi163

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477424

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi019 smithi051 smithi204

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml}

Command failed on smithi019 with status 1: "sudo yum -y install '' ceph-radosgw"

2477427

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi024 smithi112 smithi116

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477437

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi033 smithi088 smithi155

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477439

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi050 smithi069 smithi105

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml}

Command failed on smithi105 with status 1: "sudo yum -y install '' ceph-radosgw"

2477443

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi059 smithi170 smithi191

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477456

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi085 smithi141 smithi182

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 07:17:09.544278 mon.a mon.0 172.21.15.85:6789/0 327 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

2477463

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi022 smithi076 smithi196

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 07:43:31.198033 mon.a mon.0 172.21.15.22:6789/0 619 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477473

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi036 smithi156 smithi165

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477491

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi072 smithi106 smithi109

multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477503

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi111 smithi143 smithi205

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477514

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi036 smithi072 smithi156

multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}

Command failed on smithi072 with status 1: "sudo yum -y install '' ceph-radosgw"

2477516

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi033 smithi088 smithi142

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 07:55:25.351651 mon.b mon.0 172.21.15.33:6789/0 140 : cluster [WRN] MDS health message (mds.1): 2 slow requests are blocked > 30 sec" in cluster log

2477523

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi015 smithi039 smithi148

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 08:01:57.748230 mon.a mon.0 172.21.15.15:6789/0 712 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477527

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi023 smithi133 smithi145

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477531

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi081 smithi150 smithi161

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 08:13:41.622070 mon.b mon.0 172.21.15.150:6789/0 202 : cluster [WRN] Health check failed: Degraded data redundancy: 447/13568 objects degraded (3.295%), 4 pgs degraded (PG_DEGRADED)" in cluster log

2477533

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi016 smithi118 smithi144

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477538

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi084 smithi085 smithi090

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 08:28:34.444175 mon.b mon.0 172.21.15.84:6789/0 877 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477544

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

centos 

smithi091 smithi109 smithi163

multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml}

Command failed on smithi091 with status 1: "sudo yum -y install '' ceph-radosgw"

2477546

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi001 smithi165 smithi166

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 08:45:20.380917 mon.b mon.0 172.21.15.1:6789/0 242 : cluster [WRN] MDS health message (mds.0): 3 slow requests are blocked > 30 sec" in cluster log

2477551

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi029 smithi040 smithi087

multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477552

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi017 smithi093 smithi117

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477553

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi026 smithi031 smithi198

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 08:14:37.866898 mon.b mon.0 172.21.15.26:6789/0 599 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477561

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi092 smithi187 smithi192

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 08:37:38.506948 mon.a mon.0 172.21.15.92:6789/0 338 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

2477563

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi013 smithi149 smithi151

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477566

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi146 smithi155 smithi177

multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477567

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi035 smithi121 smithi163

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477568

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi079 smithi097 smithi176

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 08:30:31.859687 mon.a mon.0 172.21.15.97:6789/0 745 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477576

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi132 smithi148 smithi201

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 08:29:59.417356 mon.a mon.0 172.21.15.148:6789/0 162 : cluster [WRN] MDS health message (mds.0): 3 slow requests are blocked > 30 sec" in cluster log

2477579

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi094 smithi134 smithi189

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml}

Command failed (workunit test suites/pjd.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=wip-pdonnell-testing-20180504.211521 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

2477582

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi150 smithi170 smithi194

multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477583

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi016 smithi142 smithi158

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml}

"2018-05-05 08:59:33.790323 mon.b mon.0 172.21.15.16:6789/0 602 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

2477593

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi060 smithi074 smithi162

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds

2477606

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi006 smithi044 smithi102

multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}

"2018-05-05 09:43:57.489110 mon.a mon.0 172.21.15.44:6789/0 268 : cluster [WRN] MDS health message (mds.1): 3 slow requests are blocked > 30 sec" in cluster log

2477623

FAIL

wip-pdonnell-testing-20180504.211521

wip-pdonnell-testing-20180504.211521

master

smithi

 

smithi095 smithi165 smithi166

multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml}

reached maximum tries (150) after waiting for 900 seconds