Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 5069268 2020-05-19 13:18:49 2020-05-19 13:20:08 2020-05-19 15:06:10 1:46:02 0:11:34 1:34:28 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 5069269 2020-05-19 13:18:50 2020-05-19 13:20:08 2020-05-19 15:50:11 2:30:03 0:15:38 2:14:25 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 5069270 2020-05-19 13:18:51 2020-05-19 13:20:08 2020-05-19 16:58:13 3:38:05 1:31:34 2:06:31 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2020-05-19 15:52:43.207663 mon.a mon.0 172.21.15.184:6789/0 278 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 5069271 2020-05-19 13:18:52 2020-05-19 13:20:08 2020-05-19 15:34:11 2:14:03 1:28:40 0:45:23 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2020-05-19 14:12:54.308046 mon.a mon.0 172.21.15.37:6789/0 203 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069272 2020-05-19 13:18:53 2020-05-19 13:20:36 2020-05-19 15:16:37 1:56:01 1:22:24 0:33:37 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
pass 5069273 2020-05-19 13:18:54 2020-05-19 13:23:58 2020-05-19 14:01:58 0:38:00 0:15:35 0:22:25 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
pass 5069274 2020-05-19 13:18:55 2020-05-19 13:24:03 2020-05-19 14:10:03 0:46:00 0:21:44 0:24:16 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
pass 5069275 2020-05-19 13:18:55 2020-05-19 13:24:04 2020-05-19 14:52:05 1:28:01 1:16:44 0:11:17 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
fail 5069276 2020-05-19 13:18:56 2020-05-19 13:24:09 2020-05-19 15:30:11 2:06:02 0:06:15 1:59:47 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

present_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'force-confdef,force-confold')

fail 5069277 2020-05-19 13:18:57 2020-05-19 13:24:09 2020-05-19 17:08:15 3:44:06 3:19:59 0:24:07 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 5069278 2020-05-19 13:18:58 2020-05-19 13:25:38 2020-05-19 14:43:39 1:18:01 1:04:13 0:13:48 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2020-05-19 13:56:24.855929 mon.b mon.0 172.21.15.94:6789/0 154 : cluster [WRN] Health check failed: 3131 slow requests are blocked > 32 sec. Implicated osds 0,4,5,6 (REQUEST_SLOW)" in cluster log

fail 5069279 2020-05-19 13:18:59 2020-05-19 13:25:46 2020-05-19 15:17:47 1:52:01 0:17:43 1:34:18 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2020-05-19 15:06:43.829961 mon.a mon.0 172.21.15.112:6789/0 302 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069280 2020-05-19 13:19:00 2020-05-19 13:26:09 2020-05-19 16:12:12 2:46:03 1:26:09 1:19:54 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 5069281 2020-05-19 13:19:01 2020-05-19 13:26:30 2020-05-19 13:54:30 0:28:00 0:11:09 0:16:51 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069282 2020-05-19 13:19:01 2020-05-19 13:28:13 2020-05-19 16:56:18 3:28:05 0:11:39 3:16:26 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 5069283 2020-05-19 13:19:02 2020-05-19 13:28:17 2020-05-19 14:34:17 1:06:00 0:14:41 0:51:19 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 5069284 2020-05-19 13:19:03 2020-05-19 13:29:45 2020-05-19 16:21:49 2:52:04 1:18:06 1:33:58 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
fail 5069285 2020-05-19 13:19:04 2020-05-19 13:29:45 2020-05-19 15:59:49 2:30:04 1:13:08 1:16:56 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2020-05-19 14:54:13.906213 mon.b mon.0 172.21.15.180:6789/0 291 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069286 2020-05-19 13:19:05 2020-05-19 13:29:58 2020-05-19 16:14:02 2:44:04 2:33:03 0:11:01 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
pass 5069287 2020-05-19 13:19:06 2020-05-19 13:30:24 2020-05-19 14:28:24 0:58:00 0:13:11 0:44:49 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
pass 5069288 2020-05-19 13:19:07 2020-05-19 13:32:01 2020-05-19 14:50:02 1:18:01 0:35:03 0:42:58 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
pass 5069289 2020-05-19 13:19:07 2020-05-19 13:32:17 2020-05-19 15:44:20 2:12:03 1:06:14 1:05:49 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
pass 5069290 2020-05-19 13:19:08 2020-05-19 13:33:53 2020-05-19 17:29:59 3:56:06 0:11:18 3:44:48 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 5069291 2020-05-19 13:19:09 2020-05-19 13:33:53 2020-05-19 22:04:07 8:30:14 3:21:22 5:08:52 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 5069292 2020-05-19 13:19:10 2020-05-19 13:33:53 2020-05-19 18:06:00 4:32:07 0:56:07 3:36:00 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2020-05-19 17:28:37.706725 mon.b mon.0 172.21.15.33:6789/0 157 : cluster [WRN] Health check failed: 77 slow requests are blocked > 32 sec. Implicated osds 2,6 (REQUEST_SLOW)" in cluster log

fail 5069293 2020-05-19 13:19:11 2020-05-19 13:33:53 2020-05-19 16:49:58 3:16:05 0:26:27 2:49:38 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2020-05-19 16:31:00.360452 mon.b mon.0 172.21.15.149:6789/0 186 : cluster [WRN] Health check failed: 2309 slow requests are blocked > 32 sec. Implicated osds 4 (REQUEST_SLOW)" in cluster log

pass 5069294 2020-05-19 13:19:12 2020-05-19 13:34:03 2020-05-19 14:52:04 1:18:01 0:44:53 0:33:08 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 5069295 2020-05-19 13:19:13 2020-05-19 13:38:00 2020-05-19 14:03:59 0:25:59 0:11:34 0:14:25 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069296 2020-05-19 13:19:13 2020-05-19 13:38:09 2020-05-19 14:14:08 0:35:59 0:11:12 0:24:47 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 5069297 2020-05-19 13:19:14 2020-05-19 13:38:13 2020-05-19 15:08:14 1:30:01 0:14:36 1:15:25 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 5069298 2020-05-19 13:19:15 2020-05-19 13:39:40 2020-05-19 15:59:43 2:20:03 0:55:21 1:24:42 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
fail 5069299 2020-05-19 13:19:16 2020-05-19 13:39:47 2020-05-19 14:39:48 1:00:01 0:35:40 0:24:21 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2020-05-19 14:12:48.233558 mon.b mon.0 172.21.15.16:6789/0 181 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069300 2020-05-19 13:19:17 2020-05-19 13:40:03 2020-05-19 15:26:04 1:46:01 1:05:45 0:40:16 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
pass 5069301 2020-05-19 13:19:18 2020-05-19 13:41:22 2020-05-19 14:17:22 0:36:00 0:16:26 0:19:34 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
pass 5069302 2020-05-19 13:19:18 2020-05-19 13:41:53 2020-05-19 17:11:58 3:30:05 0:26:10 3:03:55 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
fail 5069303 2020-05-19 13:19:19 2020-05-19 13:42:04 2020-05-19 18:56:11 5:14:07 1:29:30 3:44:37 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2020-05-19 17:46:44.143934 mon.a mon.0 172.21.15.94:6789/0 229 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069304 2020-05-19 13:19:20 2020-05-19 13:42:07 2020-05-19 15:28:08 1:46:01 0:12:23 1:33:38 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 5069305 2020-05-19 13:19:21 2020-05-19 13:43:17 2020-05-19 17:03:22 3:20:05 2:58:33 0:21:32 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

fail 5069306 2020-05-19 13:19:22 2020-05-19 13:44:10 2020-05-19 15:24:12 1:40:02 0:34:40 1:05:22 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2020-05-19 15:07:23.781194 mon.a mon.0 172.21.15.95:6789/0 180 : cluster [WRN] Health check failed: 410 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)" in cluster log

fail 5069307 2020-05-19 13:19:23 2020-05-19 13:46:24 2020-05-19 18:22:31 4:36:07 0:15:56 4:20:11 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2020-05-19 18:15:12.460643 mon.a mon.0 172.21.15.51:6789/0 296 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069308 2020-05-19 13:19:23 2020-05-19 13:48:21 2020-05-19 15:52:23 2:04:02 1:24:58 0:39:04 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 5069309 2020-05-19 13:19:24 2020-05-19 13:50:05 2020-05-19 15:46:07 1:56:02 0:11:18 1:44:44 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069310 2020-05-19 13:19:25 2020-05-19 13:52:27 2020-05-19 15:18:28 1:26:01 0:11:19 1:14:42 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 5069311 2020-05-19 13:19:26 2020-05-19 13:53:44 2020-05-19 17:09:49 3:16:05 0:17:58 2:58:07 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 5069312 2020-05-19 13:19:27 2020-05-19 13:53:44 2020-05-19 18:43:52 4:50:08 1:25:49 3:24:19 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
fail 5069313 2020-05-19 13:19:28 2020-05-19 13:54:31 2020-05-19 15:44:33 1:50:02 1:35:15 0:14:47 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2020-05-19 14:16:35.633716 mon.b mon.0 172.21.15.77:6789/0 180 : cluster [WRN] Health check failed: 2282 slow requests are blocked > 32 sec. Implicated osds 0,2,4 (REQUEST_SLOW)" in cluster log

fail 5069314 2020-05-19 13:19:28 2020-05-19 13:55:35 2020-05-19 16:11:38 2:16:03 0:10:01 2:06:02 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

Command failed on smithi159 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069315 2020-05-19 13:19:29 2020-05-19 13:56:24 2020-05-19 14:26:23 0:29:59 0:13:17 0:16:42 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
pass 5069316 2020-05-19 13:19:30 2020-05-19 13:58:18 2020-05-19 14:44:18 0:46:00 0:32:59 0:13:01 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
pass 5069317 2020-05-19 13:19:31 2020-05-19 13:58:18 2020-05-19 15:42:20 1:44:02 1:21:22 0:22:40 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
pass 5069318 2020-05-19 13:19:32 2020-05-19 13:58:18 2020-05-19 15:02:19 1:04:01 0:12:42 0:51:19 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 5069319 2020-05-19 13:19:33 2020-05-19 13:59:17 2020-05-19 17:13:22 3:14:05 3:00:38 0:13:27 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 5069320 2020-05-19 13:19:33 2020-05-19 14:01:52 2020-05-19 15:31:54 1:30:02 1:19:41 0:10:21 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2020-05-19 14:29:30.899188 mon.a mon.0 172.21.15.51:6789/0 179 : cluster [WRN] Health check failed: 353 slow requests are blocked > 32 sec. Implicated osds 0,1 (REQUEST_SLOW)" in cluster log

fail 5069321 2020-05-19 13:19:34 2020-05-19 14:01:59 2020-05-19 15:02:00 1:00:01 0:15:45 0:44:16 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2020-05-19 14:52:45.434376 mon.a mon.0 172.21.15.60:6789/0 186 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069322 2020-05-19 13:19:35 2020-05-19 14:01:59 2020-05-19 15:56:01 1:54:02 0:45:04 1:08:58 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 5069323 2020-05-19 13:19:36 2020-05-19 14:04:06 2020-05-19 17:16:10 3:12:04 0:11:21 3:00:43 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069324 2020-05-19 13:19:37 2020-05-19 14:04:06 2020-05-19 18:26:13 4:22:07 0:11:42 4:10:25 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 5069325 2020-05-19 13:19:38 2020-05-19 14:06:02 2020-05-19 14:54:03 0:48:01 0:17:32 0:30:29 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 5069326 2020-05-19 13:19:39 2020-05-19 14:06:05 2020-05-19 16:00:06 1:54:01 1:23:27 0:30:34 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
fail 5069327 2020-05-19 13:19:39 2020-05-19 14:10:22 2020-05-19 15:26:23 1:16:01 1:02:55 0:13:06 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2020-05-19 14:31:20.099423 mon.b mon.0 172.21.15.149:6789/0 176 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069328 2020-05-19 13:19:40 2020-05-19 14:10:37 2020-05-19 17:42:42 3:32:05 2:50:05 0:42:00 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
pass 5069329 2020-05-19 13:19:41 2020-05-19 14:11:53 2020-05-19 14:57:53 0:46:00 0:12:23 0:33:37 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
pass 5069330 2020-05-19 13:19:42 2020-05-19 14:12:09 2020-05-19 15:56:11 1:44:02 0:23:20 1:20:42 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
pass 5069331 2020-05-19 13:19:43 2020-05-19 14:13:44 2020-05-19 17:07:48 2:54:04 1:30:04 1:24:00 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
pass 5069332 2020-05-19 13:19:44 2020-05-19 14:14:10 2020-05-19 15:14:10 1:00:00 0:10:56 0:49:04 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 5069333 2020-05-19 13:19:44 2020-05-19 14:17:39 2020-05-19 17:29:44 3:12:05 2:36:26 0:35:39 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

fail 5069334 2020-05-19 13:19:45 2020-05-19 14:24:19 2020-05-19 15:42:20 1:18:01 1:02:03 0:15:58 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2020-05-19 14:56:49.913635 mon.a mon.0 172.21.15.121:6789/0 262 : cluster [WRN] Health check failed: 150 slow requests are blocked > 32 sec. Implicated osds 1 (REQUEST_SLOW)" in cluster log

fail 5069335 2020-05-19 13:19:46 2020-05-19 14:26:41 2020-05-19 16:18:43 1:52:02 0:23:39 1:28:23 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2020-05-19 16:02:23.819913 mon.a mon.0 172.21.15.67:6789/0 202 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069336 2020-05-19 13:19:47 2020-05-19 14:27:16 2020-05-19 15:35:16 1:08:00 0:43:15 0:24:45 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 5069337 2020-05-19 13:19:48 2020-05-19 14:28:41 2020-05-19 17:18:45 2:50:04 0:11:02 2:39:02 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069338 2020-05-19 13:19:49 2020-05-19 14:29:28 2020-05-19 15:05:28 0:36:00 0:11:33 0:24:27 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 5069339 2020-05-19 13:19:50 2020-05-19 14:34:24 2020-05-19 15:26:25 0:52:01 0:05:38 0:46:23 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

Command failed on smithi120 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069340 2020-05-19 13:19:51 2020-05-19 14:34:24 2020-05-19 15:14:25 0:40:01 0:16:18 0:23:43 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
pass 5069341 2020-05-19 13:19:51 2020-05-19 14:36:14 2020-05-19 18:58:20 4:22:06 3:41:44 0:40:22 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
pass 5069342 2020-05-19 13:19:52 2020-05-19 14:36:20 2020-05-19 16:10:22 1:34:02 0:57:20 0:36:42 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
pass 5069343 2020-05-19 13:19:53 2020-05-19 14:36:30 2020-05-19 18:26:36 3:50:06 0:16:59 3:33:07 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
pass 5069344 2020-05-19 13:19:54 2020-05-19 14:38:15 2020-05-19 15:36:15 0:58:00 0:20:57 0:37:03 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
pass 5069345 2020-05-19 13:19:55 2020-05-19 14:40:06 2020-05-19 16:32:08 1:52:02 1:26:37 0:25:25 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
pass 5069346 2020-05-19 13:19:56 2020-05-19 14:40:26 2020-05-19 15:04:26 0:24:00 0:11:36 0:12:24 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 5069347 2020-05-19 13:19:57 2020-05-19 14:40:32 2020-05-19 18:10:37 3:30:05 0:50:16 2:39:49 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2020-05-19 17:48:51.209709 mon.b mon.0 172.21.15.77:6789/0 456 : cluster [WRN] Health check failed: Degraded data redundancy: 27/2382 objects degraded (1.134%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 5069348 2020-05-19 13:19:57 2020-05-19 14:40:40 2020-05-19 18:52:47 4:12:07 2:09:11 2:02:56 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2020-05-19 16:51:03.549897 mon.a mon.0 172.21.15.44:6789/0 162 : cluster [WRN] Health check failed: 34147 slow requests are blocked > 32 sec. Implicated osds 0,1,3,4,5 (REQUEST_SLOW)" in cluster log

fail 5069349 2020-05-19 13:19:58 2020-05-19 14:43:57 2020-05-19 15:21:57 0:38:00 0:16:49 0:21:11 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2020-05-19 15:12:33.485453 mon.b mon.0 172.21.15.86:6789/0 282 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE)" in cluster log

pass 5069350 2020-05-19 13:19:59 2020-05-19 14:44:20 2020-05-19 17:58:24 3:14:04 1:13:49 2:00:15 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 5069351 2020-05-19 13:20:00 2020-05-19 14:48:20 2020-05-19 16:46:22 1:58:02 0:19:03 1:38:59 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 5069352 2020-05-19 13:20:01 2020-05-19 14:50:20 2020-05-19 16:08:21 1:18:01 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

Command failed on smithi112 with status 100: 'sudo apt-get update'

fail 5069353 2020-05-19 13:20:02 2020-05-19 14:50:50 2020-05-19 16:52:53 2:02:03 0:05:17 1:56:46 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

Command failed on smithi092 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev'

fail 5069354 2020-05-19 13:20:02 2020-05-19 14:52:22 2020-05-19 16:22:24 1:30:02 0:08:14 1:21:48 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

Command failed on smithi168 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069355 2020-05-19 13:20:03 2020-05-19 14:52:22 2020-05-19 19:36:30 4:44:08 2:30:21 2:13:47 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
pass 5069356 2020-05-19 13:20:04 2020-05-19 14:53:54 2020-05-19 17:25:57 2:32:03 1:39:20 0:52:43 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
fail 5069357 2020-05-19 13:20:05 2020-05-19 14:54:04 2020-05-19 16:36:06 1:42:02 0:09:47 1:32:15 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

Command failed on smithi121 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

fail 5069358 2020-05-19 13:20:06 2020-05-19 14:54:41 2020-05-19 16:20:42 1:26:01 0:07:18 1:18:43 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

Command failed on smithi160 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069359 2020-05-19 13:20:07 2020-05-19 14:54:41 2020-05-19 16:26:42 1:32:01 0:55:39 0:36:22 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
pass 5069360 2020-05-19 13:20:08 2020-05-19 14:56:42 2020-05-19 15:22:41 0:25:59 0:11:56 0:14:03 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 5069361 2020-05-19 13:20:08 2020-05-19 14:56:42 2020-05-19 16:50:44 1:54:02 0:58:09 0:55:53 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

fail 5069362 2020-05-19 13:20:09 2020-05-19 14:58:11 2020-05-19 16:10:12 1:12:01 0:34:51 0:37:10 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2020-05-19 15:42:42.612814 mon.b mon.0 172.21.15.110:6789/0 148 : cluster [WRN] Health check failed: 13842 slow requests are blocked > 32 sec. Implicated osds 0,2,4,5,6 (REQUEST_SLOW)" in cluster log

pass 5069363 2020-05-19 13:20:10 2020-05-19 15:02:18 2020-05-19 15:38:17 0:35:59 0:20:28 0:15:31 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 5069364 2020-05-19 13:20:11 2020-05-19 15:02:20 2020-05-19 16:20:21 1:18:01 0:37:31 0:40:30 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 5069365 2020-05-19 13:20:12 2020-05-19 15:02:36 2020-05-19 16:52:38 1:50:02 0:13:19 1:36:43 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069366 2020-05-19 13:20:13 2020-05-19 15:02:52 2020-05-19 17:06:54 2:04:02 0:10:58 1:53:04 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 5069367 2020-05-19 13:20:14 2020-05-19 15:04:45 2020-05-19 16:02:45 0:58:00 0:25:36 0:32:24 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 5069368 2020-05-19 13:20:14 2020-05-19 15:05:30 2020-05-19 15:57:30 0:52:00 0:15:04 0:36:56 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
pass 5069369 2020-05-19 13:20:15 2020-05-19 15:06:28 2020-05-19 17:36:31 2:30:03 1:54:28 0:35:35 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 5069370 2020-05-19 13:20:16 2020-05-19 15:08:32 2020-05-19 16:34:33 1:26:01 0:08:54 1:17:07 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

Command failed on smithi110 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069371 2020-05-19 13:20:17 2020-05-19 15:10:14 2020-05-19 15:58:14 0:48:00 0:18:44 0:29:16 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
pass 5069372 2020-05-19 13:20:18 2020-05-19 15:14:30 2020-05-19 15:58:30 0:44:00 0:23:00 0:21:00 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
pass 5069373 2020-05-19 13:20:18 2020-05-19 15:14:30 2020-05-19 17:02:32 1:48:02 1:28:26 0:19:36 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
pass 5069374 2020-05-19 13:20:19 2020-05-19 15:16:57 2020-05-19 16:04:58 0:48:01 0:13:22 0:34:39 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069375 2020-05-19 13:20:20 2020-05-19 15:17:49 2020-05-19 17:57:52 2:40:03 0:50:18 1:49:45 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
fail 5069376 2020-05-19 13:20:21 2020-05-19 15:18:46 2020-05-19 17:22:51 2:04:05 0:29:16 1:34:49 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2020-05-19 16:58:39.806766 mon.b mon.0 172.21.15.50:6789/0 137 : cluster [WRN] Health check failed: 208 slow requests are blocked > 32 sec. Implicated osds 3,4,5 (REQUEST_SLOW)" in cluster log

fail 5069377 2020-05-19 13:20:22 2020-05-19 15:18:57 2020-05-19 16:30:58 1:12:01 0:06:17 1:05:44 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

Command failed on smithi063 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev'

pass 5069378 2020-05-19 13:20:23 2020-05-19 15:20:42 2020-05-19 17:54:45 2:34:03 1:14:34 1:19:29 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 5069379 2020-05-19 13:20:23 2020-05-19 15:22:14 2020-05-19 17:42:16 2:20:02 0:11:32 2:08:30 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069380 2020-05-19 13:20:24 2020-05-19 15:22:42 2020-05-19 16:08:43 0:46:01 0:12:39 0:33:22 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 5069381 2020-05-19 13:20:25 2020-05-19 15:24:30 2020-05-19 16:06:30 0:42:00 0:29:18 0:12:42 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 5069382 2020-05-19 13:20:26 2020-05-19 15:26:21 2020-05-19 16:00:21 0:34:00 0:23:19 0:10:41 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
pass 5069383 2020-05-19 13:20:27 2020-05-19 15:26:25 2020-05-19 18:32:29 3:06:04 2:39:48 0:26:16 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 5069384 2020-05-19 13:20:28 2020-05-19 15:26:27 2020-05-19 16:38:28 1:12:01 0:12:21 0:59:40 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

Command failed on smithi142 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069385 2020-05-19 13:20:28 2020-05-19 15:28:26 2020-05-19 16:08:26 0:40:00 0:17:05 0:22:55 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
fail 5069386 2020-05-19 13:20:29 2020-05-19 15:30:30 2020-05-19 16:28:30 0:58:00 0:06:10 0:51:50 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

Command failed on smithi164 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev'

fail 5069387 2020-05-19 13:20:30 2020-05-19 15:32:13 2020-05-19 16:16:14 0:44:01 0:10:11 0:33:50 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

Command failed on smithi043 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069388 2020-05-19 13:20:31 2020-05-19 15:34:33 2020-05-19 15:56:33 0:22:00 0:10:14 0:11:46 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069389 2020-05-19 13:20:32 2020-05-19 15:35:18 2020-05-19 18:11:21 2:36:03 0:48:37 1:47:26 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
fail 5069390 2020-05-19 13:20:32 2020-05-19 15:36:37 2020-05-19 19:12:42 3:36:05 2:09:33 1:26:32 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2020-05-19 17:11:55.011058 mon.b mon.0 172.21.15.40:6789/0 230 : cluster [WRN] Health check failed: 49804 slow requests are blocked > 32 sec. Implicated osds 0,2,4,5,6 (REQUEST_SLOW)" in cluster log

fail 5069391 2020-05-19 13:20:33 2020-05-19 15:37:16 2020-05-19 16:05:15 0:27:59 0:05:33 0:22:26 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

Command failed on smithi143 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

fail 5069392 2020-05-19 13:20:34 2020-05-19 15:38:36 2020-05-19 16:18:36 0:40:00 0:11:06 0:28:54 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

Command failed on smithi045 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069393 2020-05-19 13:20:35 2020-05-19 15:42:07 2020-05-19 17:34:09 1:52:02 0:10:49 1:41:13 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 5069394 2020-05-19 13:20:36 2020-05-19 15:42:21 2020-05-19 16:36:22 0:54:01 0:08:01 0:46:00 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

Command failed on smithi092 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069395 2020-05-19 13:20:37 2020-05-19 15:42:21 2020-05-19 16:50:22 1:08:01 0:29:18 0:38:43 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 5069396 2020-05-19 13:20:38 2020-05-19 15:42:24 2020-05-19 16:14:24 0:32:00 smithi py2 multimds/verify/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

Command failed on smithi137 with status 100: 'sudo apt-get update'

pass 5069397 2020-05-19 13:20:39 2020-05-19 15:42:26 2020-05-19 18:26:29 2:44:03 2:33:07 0:10:56 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 5069398 2020-05-19 13:20:39 2020-05-19 15:44:37 2020-05-19 18:30:40 2:46:03 1:42:16 1:03:47 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2020-05-19 17:02:16.785266 mon.a mon.0 172.21.15.131:6789/0 288 : cluster [WRN] Health check failed: 303 slow requests are blocked > 32 sec. Implicated osds 0,4,5 (REQUEST_SLOW)" in cluster log

pass 5069399 2020-05-19 13:20:40 2020-05-19 15:44:37 2020-05-19 17:00:38 1:16:01 0:15:19 1:00:42 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
fail 5069400 2020-05-19 13:20:41 2020-05-19 15:44:37 2020-05-19 16:40:37 0:56:00 0:11:17 0:44:43 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

Command failed on smithi077 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev'

fail 5069401 2020-05-19 13:20:42 2020-05-19 15:46:24 2020-05-19 16:20:24 0:34:00 0:07:52 0:26:08 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

present_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u'force-confdef,force-confold')

pass 5069402 2020-05-19 13:20:43 2020-05-19 15:50:30 2020-05-19 16:30:30 0:40:00 0:13:17 0:26:43 smithi py2 multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 5069403 2020-05-19 13:20:44 2020-05-19 15:52:44 2020-05-19 16:56:46 1:04:02 0:47:51 0:16:11 smithi py2 centos multimds/verify/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
fail 5069404 2020-05-19 13:20:44 2020-05-19 15:55:12 2020-05-19 16:37:12 0:42:00 0:09:00 0:33:00 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

Command failed on smithi140 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

fail 5069405 2020-05-19 13:20:45 2020-05-19 15:56:04 2020-05-19 16:34:03 0:37:59 0:09:23 0:28:36 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

Command failed on smithi094 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

fail 5069406 2020-05-19 13:20:46 2020-05-19 15:56:30 2020-05-19 16:30:30 0:34:00 0:06:45 0:27:15 smithi py2 multimds/basic/{begin.yaml clusters/3-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

Command failed on smithi044 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=12.2.13-38-gb6acb29-1bionic ceph-mds=12.2.13-38-gb6acb29-1bionic ceph-mgr=12.2.13-38-gb6acb29-1bionic ceph-common=12.2.13-38-gb6acb29-1bionic ceph-fuse=12.2.13-38-gb6acb29-1bionic ceph-test=12.2.13-38-gb6acb29-1bionic radosgw=12.2.13-38-gb6acb29-1bionic python-ceph=12.2.13-38-gb6acb29-1bionic libcephfs2=12.2.13-38-gb6acb29-1bionic libcephfs-dev=12.2.13-38-gb6acb29-1bionic librados2=12.2.13-38-gb6acb29-1bionic librbd1=12.2.13-38-gb6acb29-1bionic rbd-fuse=12.2.13-38-gb6acb29-1bionic python3-cephfs=12.2.13-38-gb6acb29-1bionic'

pass 5069407 2020-05-19 13:20:47 2020-05-19 15:56:34 2020-05-19 17:02:35 1:06:01 0:10:51 0:55:10 smithi py2 multimds/basic/{begin.yaml clusters/9-mds.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml overrides/{basic/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3