Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 2558440 2018-05-20 20:11:53 2018-05-20 20:17:49 2018-05-20 22:55:51 2:38:02 0:19:52 2:18:10 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 22:45:48.957447 mon.a mon.0 172.21.15.103:6789/0 1250 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 30 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

fail 2558441 2018-05-20 20:11:54 2018-05-20 20:17:49 2018-05-20 23:29:52 3:12:03 1:54:25 1:17:38 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 21:42:50.652544 mon.a mon.0 172.21.15.182:6789/0 207 : cluster [WRN] MDS health message (mds.7): 1 slow requests are blocked > 30 sec" in cluster log

fail 2558442 2018-05-20 20:11:55 2018-05-20 20:17:51 2018-05-20 21:33:51 1:16:00 0:20:45 0:55:15 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 21:22:52.966170 mon.b mon.0 172.21.15.56:6789/0 502 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

dead 2558443 2018-05-20 20:11:55 2018-05-20 20:17:57 2018-05-21 08:20:24 12:02:27 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 2558444 2018-05-20 20:11:56 2018-05-20 20:18:03 2018-05-20 21:00:02 0:41:59 0:30:48 0:11:11 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 20:46:31.636360 mon.b mon.0 172.21.15.55:6789/0 188 : cluster [WRN] MDS health message (mds.5): 2 slow requests are blocked > 30 sec" in cluster log

fail 2558445 2018-05-20 20:11:57 2018-05-20 20:18:08 2018-05-20 21:26:09 1:08:01 0:20:03 0:47:58 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 21:21:30.115643 mon.a mon.0 172.21.15.59:6789/0 1441 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

pass 2558446 2018-05-20 20:11:58 2018-05-20 20:19:58 2018-05-20 22:44:00 2:24:02 0:52:58 1:31:04 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2558447 2018-05-20 20:11:59 2018-05-20 20:20:20 2018-05-20 23:08:22 2:48:02 0:22:30 2:25:32 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 23:06:24.931695 mon.a mon.0 172.21.15.15:6789/0 1367 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

pass 2558448 2018-05-20 20:12:00 2018-05-20 20:21:39 2018-05-20 22:45:41 2:24:02 0:23:16 2:00:46 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2558449 2018-05-20 20:12:01 2018-05-20 20:21:39 2018-05-20 21:31:39 1:10:00 0:20:04 0:49:56 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 21:20:38.506976 mon.b mon.0 172.21.15.66:6789/0 495 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

pass 2558450 2018-05-20 20:12:01 2018-05-20 20:21:50 2018-05-20 23:19:52 2:58:02 0:42:11 2:15:51 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 2558451 2018-05-20 20:12:02 2018-05-20 20:22:26 2018-05-20 21:28:27 1:06:01 0:52:40 0:13:21 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 20:56:28.873527 mon.b mon.0 172.21.15.77:6789/0 154 : cluster [WRN] MDS health message (mds.2): 1 slow requests are blocked > 30 sec" in cluster log

pass 2558452 2018-05-20 20:12:03 2018-05-20 20:22:34 2018-05-20 21:12:34 0:50:00 0:38:02 0:11:58 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
fail 2558453 2018-05-20 20:12:04 2018-05-20 20:23:50 2018-05-20 21:01:50 0:38:00 0:23:37 0:14:23 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 20:46:20.359848 mon.b mon.0 172.21.15.66:6789/0 183 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

dead 2558454 2018-05-20 20:12:05 2018-05-20 20:24:04 2018-05-21 08:26:31 12:02:27 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 2558455 2018-05-20 20:12:06 2018-05-20 20:24:07 2018-05-20 21:28:07 1:04:00 0:49:11 0:14:49 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
fail 2558456 2018-05-20 20:12:06 2018-05-20 20:24:19 2018-05-20 21:34:19 1:10:00 0:53:21 0:16:39 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 2558457 2018-05-20 20:12:07 2018-05-20 20:25:45 2018-05-20 22:43:46 2:18:01 2:05:42 0:12:19 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
dead 2558458 2018-05-20 20:12:08 2018-05-20 20:25:45 2018-05-21 08:28:12 12:02:27 11:50:13 0:12:14 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

SSH connection to smithi090 was lost: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1'

fail 2558459 2018-05-20 20:12:09 2018-05-20 20:25:46 2018-05-20 20:57:45 0:31:59 0:20:19 0:11:40 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 20:46:45.212479 mon.a mon.0 172.21.15.73:6789/0 608 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

pass 2558460 2018-05-20 20:12:10 2018-05-20 20:25:47 2018-05-20 21:35:47 1:10:00 0:59:22 0:10:38 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
pass 2558461 2018-05-20 20:12:11 2018-05-20 20:25:53 2018-05-20 22:05:54 1:40:01 0:46:12 0:53:49 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
pass 2558462 2018-05-20 20:12:12 2018-05-20 20:26:13 2018-05-20 21:14:13 0:48:00 0:31:38 0:16:22 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2558463 2018-05-20 20:12:13 2018-05-20 20:27:35 2018-05-20 22:55:37 2:28:02 2:12:57 0:15:05 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 20:51:14.449008 mon.b mon.0 172.21.15.9:6789/0 198 : cluster [WRN] MDS health message (mds.3): 2 slow requests are blocked > 30 sec" in cluster log

fail 2558464 2018-05-20 20:12:13 2018-05-20 20:27:53 2018-05-20 23:17:55 2:50:02 1:00:57 1:49:05 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-05-20 22:34:08.504321 mon.a mon.0 172.21.15.82:6789/0 218 : cluster [WRN] MDS health message (mds.1): 1 slow requests are blocked > 30 sec" in cluster log

fail 2558465 2018-05-20 20:12:14 2018-05-20 20:27:54 2018-05-20 22:39:55 2:12:01 0:31:31 1:40:30 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 22:21:10.848466 mon.a mon.0 172.21.15.28:6789/0 298 : cluster [WRN] MDS health message (mds.3): 1 slow requests are blocked > 30 sec" in cluster log

fail 2558466 2018-05-20 20:12:15 2018-05-20 20:27:54 2018-05-20 21:51:54 1:24:00 0:56:36 0:27:24 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 21:14:56.464399 mon.b mon.0 172.21.15.26:6789/0 161 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

pass 2558467 2018-05-20 20:12:16 2018-05-20 20:27:59 2018-05-20 21:03:59 0:36:00 0:21:06 0:14:54 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2558468 2018-05-20 20:12:17 2018-05-20 20:28:26 2018-05-20 22:22:27 1:54:01 0:20:34 1:33:27 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 22:17:03.539442 mon.b mon.0 172.21.15.45:6789/0 1067 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

fail 2558469 2018-05-20 20:12:18 2018-05-20 20:29:33 2018-05-20 21:59:34 1:30:01 0:44:44 0:45:17 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-05-20 21:23:25.480698 mon.b mon.0 172.21.15.84:6789/0 199 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

fail 2558470 2018-05-20 20:12:19 2018-05-20 20:29:43 2018-05-20 22:05:44 1:36:01 0:51:49 0:44:12 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

fail 2558471 2018-05-20 20:12:19 2018-05-20 20:29:43 2018-05-20 21:03:43 0:34:00 0:22:19 0:11:41 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 20:49:16.014505 mon.b mon.0 172.21.15.134:6789/0 277 : cluster [WRN] MDS health message (mds.1): 1 slow requests are blocked > 30 sec" in cluster log

dead 2558472 2018-05-20 20:12:20 2018-05-20 20:29:46 2018-05-21 08:32:12 12:02:26 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
fail 2558473 2018-05-20 20:12:21 2018-05-20 20:30:05 2018-05-20 21:08:05 0:38:00 0:20:00 0:18:00 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 21:03:12.658478 mon.a mon.0 172.21.15.44:6789/0 1226 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

pass 2558474 2018-05-20 20:12:22 2018-05-20 20:30:05 2018-05-20 21:36:05 1:06:00 0:50:35 0:15:25 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
pass 2558475 2018-05-20 20:12:23 2018-05-20 20:32:03 2018-05-20 22:28:04 1:56:01 0:44:02 1:11:59 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
fail 2558476 2018-05-20 20:12:23 2018-05-20 20:32:07 2018-05-20 22:48:09 2:16:02 1:56:10 0:19:52 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 21:08:36.682767 mon.a mon.0 172.21.15.150:6789/0 217 : cluster [WRN] MDS health message (mds.1): 2 slow requests are blocked > 30 sec" in cluster log

dead 2558477 2018-05-20 20:12:24 2018-05-20 20:33:33 2018-05-21 08:36:01 12:02:28 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
fail 2558478 2018-05-20 20:12:25 2018-05-20 20:33:45 2018-05-20 21:21:45 0:48:00 0:31:14 0:16:46 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 21:08:09.643201 mon.a mon.0 172.21.15.170:6789/0 198 : cluster [WRN] MDS health message (mds.1): 1 slow requests are blocked > 30 sec" in cluster log

pass 2558479 2018-05-20 20:12:26 2018-05-20 20:33:49 2018-05-20 21:51:50 1:18:01 0:48:57 0:29:04 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 2558480 2018-05-20 20:12:27 2018-05-20 20:33:51 2018-05-20 22:07:51 1:34:00 0:21:21 1:12:39 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 22:02:57.590793 mon.a mon.0 172.21.15.37:6789/0 1816 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

pass 2558481 2018-05-20 20:12:27 2018-05-20 20:33:59 2018-05-20 21:13:59 0:40:00 0:20:42 0:19:18 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
pass 2558482 2018-05-20 20:12:28 2018-05-20 20:36:07 2018-05-20 21:58:07 1:22:00 0:54:41 0:27:19 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 2558483 2018-05-20 20:12:29 2018-05-20 20:42:04 2018-05-20 22:14:04 1:32:00 0:50:26 0:41:34 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 21:38:39.966842 mon.b mon.0 172.21.15.33:6789/0 296 : cluster [WRN] MDS health message (mds.8): 1 slow requests are blocked > 30 sec" in cluster log

pass 2558484 2018-05-20 20:12:30 2018-05-20 20:42:04 2018-05-20 21:28:03 0:45:59 0:09:22 0:36:37 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 2558485 2018-05-20 20:12:31 2018-05-20 20:43:56 2018-05-20 21:33:56 0:50:00 0:19:42 0:30:18 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
pass 2558486 2018-05-20 20:12:31 2018-05-20 20:43:59 2018-05-20 21:52:00 1:08:01 0:21:03 0:46:58 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2558487 2018-05-20 20:12:32 2018-05-20 20:45:55 2018-05-20 21:37:55 0:52:00 0:22:30 0:29:30 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 21:22:02.515611 mon.b mon.0 172.21.15.63:6789/0 234 : cluster [WRN] MDS health message (mds.2): 1 slow requests are blocked > 30 sec" in cluster log

dead 2558488 2018-05-20 20:12:33 2018-05-20 20:46:04 2018-05-21 08:48:30 12:02:26 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 2558489 2018-05-20 20:12:34 2018-05-20 20:49:42 2018-05-20 22:19:43 1:30:01 0:21:02 1:08:59 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
pass 2558490 2018-05-20 20:12:35 2018-05-20 20:50:00 2018-05-20 21:58:01 1:08:01 0:43:02 0:24:59 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
dead 2558491 2018-05-20 20:12:35 2018-05-20 20:51:36 2018-05-21 08:54:02 12:02:26 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 2558492 2018-05-20 20:12:36 2018-05-20 20:57:53 2018-05-20 21:21:52 0:23:59 0:11:22 0:12:37 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
fail 2558493 2018-05-20 20:12:37 2018-05-20 21:00:04 2018-05-21 00:10:06 3:10:02 1:38:15 1:31:47 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-05-20 22:40:04.271537 mon.b mon.0 172.21.15.55:6789/0 196 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

pass 2558494 2018-05-20 20:12:38 2018-05-20 21:01:51 2018-05-20 22:29:52 1:28:01 0:46:23 0:41:38 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
pass 2558495 2018-05-20 20:12:39 2018-05-20 21:02:01 2018-05-21 00:08:04 3:06:03 1:51:43 1:14:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
pass 2558496 2018-05-20 20:12:39 2018-05-20 21:03:52 2018-05-20 21:39:52 0:36:00 0:21:15 0:14:45 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
pass 2558497 2018-05-20 20:12:40 2018-05-20 21:03:57 2018-05-20 22:21:58 1:18:01 0:52:03 0:25:58 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
fail 2558498 2018-05-20 20:12:41 2018-05-20 21:04:00 2018-05-20 22:20:01 1:16:01 0:57:06 0:18:55 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-05-20 21:40:52.223399 mon.a mon.0 172.21.15.60:6789/0 172 : cluster [WRN] MDS health message (mds.2): 1 slow requests are blocked > 30 sec" in cluster log

pass 2558499 2018-05-20 20:12:42 2018-05-20 21:05:48 2018-05-20 22:07:48 1:02:00 0:24:28 0:37:32 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
pass 2558500 2018-05-20 20:12:43 2018-05-20 21:05:48 2018-05-20 22:23:49 1:18:01 0:57:46 0:20:15 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 2558501 2018-05-20 20:12:44 2018-05-20 21:05:58 2018-05-20 22:49:59 1:44:01 0:33:41 1:10:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 22:33:30.498115 mon.a mon.0 172.21.15.59:6789/0 184 : cluster [WRN] MDS health message (mds.0): 1 slow requests are blocked > 30 sec" in cluster log

pass 2558502 2018-05-20 20:12:44 2018-05-20 21:07:53 2018-05-20 22:41:54 1:34:01 0:11:54 1:22:07 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 2558503 2018-05-20 20:12:45 2018-05-20 21:07:57 2018-05-20 22:31:58 1:24:01 0:46:19 0:37:42 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 2558504 2018-05-20 20:12:46 2018-05-20 21:08:07 2018-05-20 22:18:07 1:10:00 0:55:10 0:14:50 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 21:37:55.791385 mon.a mon.0 172.21.15.47:6789/0 258 : cluster [WRN] Health check failed: Degraded data redundancy: 1656/26232 objects degraded (6.313%), 3 pgs degraded (PG_DEGRADED)" in cluster log

fail 2558505 2018-05-20 20:12:47 2018-05-20 21:08:07 2018-05-20 21:44:07 0:36:00 0:19:19 0:16:41 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 21:40:42.583263 mon.a mon.0 172.21.15.73:6789/0 879 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

pass 2558506 2018-05-20 20:12:48 2018-05-20 21:09:54 2018-05-20 22:55:55 1:46:01 0:46:47 0:59:14 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
dead 2558507 2018-05-20 20:12:49 2018-05-20 21:09:54 2018-05-21 09:12:21 12:02:27 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 2558508 2018-05-20 20:12:50 2018-05-20 21:10:01 2018-05-20 22:28:01 1:18:00 0:47:15 0:30:45 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
pass 2558509 2018-05-20 20:12:50 2018-05-20 21:12:43 2018-05-20 22:56:44 1:44:01 0:28:04 1:15:57 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
pass 2558510 2018-05-20 20:12:51 2018-05-20 21:14:01 2018-05-20 21:46:00 0:31:59 0:09:45 0:22:14 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2558511 2018-05-20 20:12:52 2018-05-20 21:14:14 2018-05-20 22:26:14 1:12:00 0:19:21 0:52:39 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-05-20 22:15:22.012622 mon.a mon.0 172.21.15.154:6789/0 750 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec, mon.c has slow ops (SLOW_OPS)" in cluster log

fail 2558512 2018-05-20 20:12:53 2018-05-20 21:15:50 2018-05-20 23:25:51 2:10:01 1:49:21 0:20:40 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 21:43:46.841791 mon.a mon.0 172.21.15.153:6789/0 254 : cluster [WRN] MDS health message (mds.1): 2 slow requests are blocked > 30 sec" in cluster log

dead 2558513 2018-05-20 20:12:54 2018-05-20 21:21:53 2018-05-21 09:24:25 12:02:32 11:22:05 0:40:27 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

SSH connection to smithi163 was lost: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

fail 2558514 2018-05-20 20:12:54 2018-05-20 21:21:54 2018-05-20 22:13:54 0:52:00 0:31:06 0:20:54 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-05-20 21:58:14.911768 mon.b mon.0 172.21.15.56:6789/0 307 : cluster [WRN] MDS health message (mds.6): 1 slow requests are blocked > 30 sec" in cluster log