Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 2382499 2018-04-10 20:51:33 2018-04-12 05:02:29 2018-04-12 05:22:28 0:19:59 0:09:16 0:10:43 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382500 2018-04-10 20:51:34 2018-04-12 05:02:31 2018-04-12 05:26:31 0:24:00 0:08:59 0:15:01 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 05:22:20.480331 mon.a mon.0 172.21.15.129:6789/0 162 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 50 sec (SLOW_OPS)" in cluster log

fail 2382501 2018-04-10 20:51:35 2018-04-12 05:02:31 2018-04-12 06:00:32 0:58:01 0:43:23 0:14:38 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 05:34:39.524257 mon.b mon.0 172.21.15.66:6789/0 143 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382502 2018-04-10 20:51:35 2018-04-12 05:02:34 2018-04-12 05:36:34 0:34:00 0:21:38 0:12:22 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 05:24:35.071380 mon.b mon.0 172.21.15.52:6789/0 810 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382503 2018-04-10 20:51:36 2018-04-12 05:02:37 2018-04-12 06:18:37 1:16:00 1:04:04 0:11:56 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 2382504 2018-04-10 20:51:37 2018-04-12 05:04:15 2018-04-12 05:50:16 0:46:01 0:29:01 0:17:00 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 05:27:29.903628 mon.b mon.0 172.21.15.99:6789/0 312 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382505 2018-04-10 20:51:38 2018-04-12 05:04:21 2018-04-12 06:58:23 1:54:02 0:13:57 1:40:05 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
fail 2382506 2018-04-10 20:51:39 2018-04-12 05:04:23 2018-04-12 05:48:23 0:44:00 0:29:44 0:14:16 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 05:24:24.450411 mon.b mon.0 172.21.15.61:6789/0 304 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382507 2018-04-10 20:51:39 2018-04-12 05:04:24 2018-04-12 05:30:24 0:26:00 0:10:01 0:15:59 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 05:26:33.568496 mon.a mon.0 172.21.15.37:6789/0 214 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 73 sec (SLOW_OPS)" in cluster log

fail 2382508 2018-04-10 20:51:40 2018-04-12 05:04:25 2018-04-12 06:14:26 1:10:01 0:49:51 0:20:10 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

fail 2382509 2018-04-10 20:51:41 2018-04-12 05:06:22 2018-04-12 05:56:22 0:50:00 0:39:07 0:10:53 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 05:33:51.276084 mon.b mon.0 172.21.15.176:6789/0 139 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

dead 2382510 2018-04-10 20:51:42 2018-04-12 05:06:22 2018-04-12 05:30:21 0:23:59 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml}
Failure Reason:

reached maximum tries (60) after waiting for 900 seconds

pass 2382511 2018-04-10 20:51:42 2018-04-12 05:06:23 2018-04-12 05:30:22 0:23:59 0:08:52 0:15:07 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382512 2018-04-10 20:51:43 2018-04-12 05:06:23 2018-04-12 05:42:23 0:36:00 0:16:18 0:19:42 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 05:38:30.081440 mon.a mon.0 172.21.15.5:6789/0 331 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 438 sec (SLOW_OPS)" in cluster log

fail 2382513 2018-04-10 20:51:44 2018-04-12 05:06:23 2018-04-12 06:00:24 0:54:01 0:10:01 0:44:00 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 05:56:28.979010 mon.a mon.0 172.21.15.71:6789/0 170 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382514 2018-04-10 20:51:45 2018-04-12 05:06:24 2018-04-12 06:00:25 0:54:01 0:09:50 0:44:11 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382515 2018-04-10 20:51:46 2018-04-12 05:06:25 2018-04-12 06:00:25 0:54:00 0:09:02 0:44:58 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 05:57:52.667968 mon.b mon.0 172.21.15.6:6789/0 211 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 50 sec (SLOW_OPS)" in cluster log

fail 2382516 2018-04-10 20:51:46 2018-04-12 05:08:31 2018-04-12 06:08:31 1:00:00 0:42:33 0:17:27 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 05:30:43.377214 mon.b mon.0 172.21.15.169:6789/0 146 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382517 2018-04-10 20:51:47 2018-04-12 05:08:31 2018-04-12 06:38:32 1:30:01 0:21:56 1:08:05 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382518 2018-04-10 20:51:48 2018-04-12 05:08:31 2018-04-12 07:32:33 2:24:02 0:33:03 1:50:59 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 07:20:14.088381 mon.b mon.0 172.21.15.23:6789/0 183 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382519 2018-04-10 20:51:49 2018-04-12 05:08:31 2018-04-12 06:06:31 0:58:00 0:38:24 0:19:36 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 05:33:23.289518 mon.b mon.0 172.21.15.40:6789/0 140 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382520 2018-04-10 20:51:49 2018-04-12 05:08:31 2018-04-12 06:20:31 1:12:00 0:11:50 1:00:10 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 06:15:50.133636 mon.a mon.0 172.21.15.41:6789/0 334 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382521 2018-04-10 20:51:50 2018-04-12 05:08:31 2018-04-12 06:52:33 1:44:02 0:17:24 1:26:38 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 06:40:04.399192 mon.a mon.0 172.21.15.78:6789/0 155 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382522 2018-04-10 20:51:51 2018-04-12 05:08:31 2018-04-12 07:08:33 2:00:02 0:10:21 1:49:41 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382523 2018-04-10 20:51:52 2018-04-12 05:10:34 2018-04-12 08:46:39 3:36:05 3:21:19 0:14:46 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi043 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=wip-pdonnell-testing-20180410.184912 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

fail 2382524 2018-04-10 20:51:53 2018-04-12 05:10:34 2018-04-12 06:12:35 1:02:01 0:50:24 0:11:37 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 05:51:33.084283 mon.a mon.0 172.21.15.62:6789/0 365 : cluster [WRN] Health check failed: 29 slow ops, oldest one blocked for 643 sec (SLOW_OPS)" in cluster log

fail 2382525 2018-04-10 20:51:53 2018-04-12 05:12:19 2018-04-12 06:00:19 0:48:00 0:37:36 0:10:24 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 05:29:14.153520 mon.b mon.0 172.21.15.164:6789/0 136 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382526 2018-04-10 20:51:54 2018-04-12 05:12:19 2018-04-12 05:34:18 0:21:59 0:09:42 0:12:17 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 05:29:23.371025 mon.a mon.0 172.21.15.83:6789/0 198 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382527 2018-04-10 20:51:55 2018-04-12 05:12:19 2018-04-12 05:54:19 0:42:00 0:29:43 0:12:17 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
fail 2382528 2018-04-10 20:51:56 2018-04-12 05:12:21 2018-04-12 05:38:21 0:26:00 0:10:06 0:15:54 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 05:35:30.490402 mon.a mon.0 172.21.15.73:6789/0 200 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 126 sec (SLOW_OPS)" in cluster log

pass 2382529 2018-04-10 20:51:57 2018-04-12 05:12:22 2018-04-12 06:30:23 1:18:01 0:09:31 1:08:30 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382530 2018-04-10 20:51:57 2018-04-12 05:12:23 2018-04-12 05:32:22 0:19:59 0:08:08 0:11:51 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 05:28:32.590096 mon.b mon.0 172.21.15.86:6789/0 193 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382531 2018-04-10 20:51:58 2018-04-12 05:14:24 2018-04-12 06:06:24 0:52:00 0:39:26 0:12:34 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 05:44:21.549164 mon.a mon.0 172.21.15.18:6789/0 158 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 736 sec (SLOW_OPS)" in cluster log

pass 2382532 2018-04-10 20:51:59 2018-04-12 05:14:24 2018-04-12 05:46:24 0:32:00 0:20:16 0:11:44 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382533 2018-04-10 20:52:00 2018-04-12 05:14:24 2018-04-12 06:28:25 1:14:01 1:03:00 0:11:01 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 05:31:38.060133 mon.b mon.0 172.21.15.28:6789/0 133 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382534 2018-04-10 20:52:00 2018-04-12 05:14:25 2018-04-12 06:12:25 0:58:00 0:47:22 0:10:38 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 05:32:53.577601 mon.b mon.0 172.21.15.70:6789/0 193 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 149 sec (SLOW_OPS)" in cluster log

fail 2382535 2018-04-10 20:52:01 2018-04-12 05:14:25 2018-04-12 05:40:25 0:26:00 0:13:03 0:12:57 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 05:32:49.360558 mon.b mon.0 172.21.15.84:6789/0 237 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382536 2018-04-10 20:52:02 2018-04-12 05:14:25 2018-04-12 06:00:25 0:46:00 0:28:27 0:17:33 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 05:37:56.939147 mon.a mon.0 172.21.15.49:6789/0 330 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382537 2018-04-10 20:52:03 2018-04-12 05:14:25 2018-04-12 05:40:25 0:26:00 0:10:53 0:15:07 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 05:37:23.519535 mon.a mon.0 172.21.15.71:6789/0 213 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 105 sec (SLOW_OPS)" in cluster log

fail 2382538 2018-04-10 20:52:03 2018-04-12 05:14:31 2018-04-12 06:16:32 1:02:01 0:46:56 0:15:05 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 06:08:04.238849 mon.a mon.0 172.21.15.36:6789/0 207 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 95 sec (SLOW_OPS)" in cluster log

pass 2382539 2018-04-10 20:52:04 2018-04-12 05:14:32 2018-04-12 06:08:32 0:54:00 0:39:47 0:14:13 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
pass 2382540 2018-04-10 20:52:05 2018-04-12 05:16:32 2018-04-12 06:30:33 1:14:01 0:20:39 0:53:22 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
pass 2382541 2018-04-10 20:52:06 2018-04-12 05:16:33 2018-04-12 07:10:34 1:54:01 0:09:45 1:44:16 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382542 2018-04-10 20:52:07 2018-04-12 05:16:32 2018-04-12 06:46:34 1:30:02 0:18:55 1:11:07 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 06:34:48.852864 mon.a mon.0 172.21.15.49:6789/0 190 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382543 2018-04-10 20:52:07 2018-04-12 05:16:32 2018-04-12 06:38:33 1:22:01 0:10:07 1:11:54 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 06:35:12.319321 mon.a mon.0 172.21.15.41:6789/0 157 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382544 2018-04-10 20:52:08 2018-04-12 05:16:32 2018-04-12 06:24:33 1:08:01 0:09:52 0:58:09 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 2382545 2018-04-10 20:52:09 2018-04-12 05:16:32 2018-04-12 06:54:34 1:38:02 0:10:28 1:27:34 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382546 2018-04-10 20:52:10 2018-04-12 05:16:40 2018-04-12 06:12:40 0:56:00 0:39:45 0:16:15 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 05:38:27.470154 mon.b mon.0 172.21.15.94:6789/0 236 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382547 2018-04-10 20:52:10 2018-04-12 05:18:20 2018-04-12 05:52:19 0:33:59 0:20:45 0:13:14 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 05:47:02.795127 mon.a mon.0 172.21.15.82:6789/0 1159 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382548 2018-04-10 20:52:11 2018-04-12 05:18:22 2018-04-12 06:46:24 1:28:02 0:31:29 0:56:33 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 06:29:58.435764 mon.b mon.0 172.21.15.19:6789/0 180 : cluster [WRN] Health check failed: 8 slow ops, oldest one blocked for 603 sec (SLOW_OPS)" in cluster log

fail 2382549 2018-04-10 20:52:12 2018-04-12 05:18:23 2018-04-12 07:58:26 2:40:03 0:41:25 1:58:38 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 07:25:09.391069 mon.a mon.0 172.21.15.58:6789/0 165 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 156 sec (SLOW_OPS)" in cluster log

fail 2382550 2018-04-10 20:52:13 2018-04-12 05:18:23 2018-04-12 06:32:24 1:14:01 0:11:39 1:02:22 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 06:28:09.938672 mon.a mon.0 172.21.15.14:6789/0 206 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382551 2018-04-10 20:52:14 2018-04-12 05:18:29 2018-04-12 05:50:29 0:32:00 0:18:28 0:13:32 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
fail 2382552 2018-04-10 20:52:14 2018-04-12 05:18:30 2018-04-12 05:40:30 0:22:00 0:10:46 0:11:14 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 05:36:29.061828 mon.a mon.0 172.21.15.35:6789/0 274 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382553 2018-04-10 20:52:15 2018-04-12 05:18:32 2018-04-12 06:24:32 1:06:00 0:45:15 0:20:45 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 06:12:24.287434 mon.a mon.0 172.21.15.81:6789/0 321 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382554 2018-04-10 20:52:16 2018-04-12 05:18:33 2018-04-12 07:06:35 1:48:02 0:47:52 1:00:10 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 06:30:45.927795 mon.a mon.0 172.21.15.37:6789/0 329 : cluster [WRN] Health check failed: 8 slow ops, oldest one blocked for 471 sec (SLOW_OPS)" in cluster log

fail 2382555 2018-04-10 20:52:17 2018-04-12 05:18:34 2018-04-12 06:30:34 1:12:00 0:46:29 0:25:31 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 05:49:36.252210 mon.b mon.0 172.21.15.83:6789/0 136 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382556 2018-04-10 20:52:17 2018-04-12 05:20:30 2018-04-12 05:42:29 0:21:59 0:10:19 0:11:40 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 05:40:04.180465 mon.b mon.0 172.21.15.102:6789/0 173 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382557 2018-04-10 20:52:18 2018-04-12 05:20:30 2018-04-12 06:14:30 0:54:00 0:29:21 0:24:39 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 06:07:11.283603 mon.b mon.0 172.21.15.52:6789/0 144 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382558 2018-04-10 20:52:19 2018-04-12 05:20:30 2018-04-12 05:42:30 0:22:00 0:10:01 0:11:59 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 05:37:53.942513 mon.a mon.0 172.21.15.41:6789/0 193 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382559 2018-04-10 20:52:20 2018-04-12 05:20:32 2018-04-12 06:26:33 1:06:01 0:09:23 0:56:38 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382560 2018-04-10 20:52:20 2018-04-12 05:23:00 2018-04-12 05:42:59 0:19:59 0:08:42 0:11:17 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 05:38:40.489612 mon.b mon.0 172.21.15.1:6789/0 122 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

pass 2382561 2018-04-10 20:52:21 2018-04-12 05:23:01 2018-04-12 06:13:01 0:50:00 0:39:23 0:10:37 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
pass 2382562 2018-04-10 20:52:22 2018-04-12 05:23:00 2018-04-12 05:57:00 0:34:00 0:20:45 0:13:15 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382563 2018-04-10 20:52:22 2018-04-12 05:23:00 2018-04-12 06:39:01 1:16:01 1:04:26 0:11:35 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 05:52:41.402145 mon.b mon.0 172.21.15.29:6789/0 133 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382564 2018-04-10 20:52:23 2018-04-12 05:23:01 2018-04-12 06:33:01 1:10:00 0:51:21 0:18:39 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 05:47:49.618404 mon.a mon.0 172.21.15.59:6789/0 200 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382565 2018-04-10 20:52:24 2018-04-12 05:23:01 2018-04-12 05:57:00 0:33:59 0:13:39 0:20:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 05:51:13.857353 mon.b mon.0 172.21.15.8:6789/0 144 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 212 sec (SLOW_OPS)" in cluster log

fail 2382566 2018-04-10 20:52:25 2018-04-12 05:23:01 2018-04-12 06:07:00 0:43:59 0:26:45 0:17:14 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 05:53:37.335681 mon.a mon.0 172.21.15.37:6789/0 230 : cluster [WRN] Health check failed: 37 slow ops, oldest one blocked for 552 sec (SLOW_OPS)" in cluster log

pass 2382567 2018-04-10 20:52:26 2018-04-12 05:24:32 2018-04-12 06:04:32 0:40:00 0:10:07 0:29:53 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382568 2018-04-10 20:52:26 2018-04-12 05:24:33 2018-04-12 07:22:35 1:58:02 0:50:31 1:07:31 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 07:06:36.506226 mon.b mon.0 172.21.15.5:6789/0 179 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 40 sec (SLOW_OPS)" in cluster log

fail 2382569 2018-04-10 20:52:27 2018-04-12 05:24:33 2018-04-12 07:08:34 1:44:01 0:44:31 0:59:30 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 06:30:09.830392 mon.b mon.0 172.21.15.8:6789/0 147 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382570 2018-04-10 20:52:28 2018-04-12 05:24:33 2018-04-12 06:42:34 1:18:01 0:20:02 0:57:59 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 06:28:21.872164 mon.b mon.0 172.21.15.94:6789/0 308 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2382571 2018-04-10 20:52:29 2018-04-12 05:24:33 2018-04-12 06:36:33 1:12:00 0:09:19 1:02:41 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382572 2018-04-10 20:52:29 2018-04-12 05:26:31 2018-04-12 06:00:31 0:34:00 0:18:24 0:15:36 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 05:47:23.210295 mon.b mon.0 172.21.15.33:6789/0 168 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382573 2018-04-10 20:52:30 2018-04-12 05:26:32 2018-04-12 05:50:31 0:23:59 0:10:00 0:13:59 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 05:47:54.993384 mon.a mon.0 172.21.15.24:6789/0 160 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 142 sec (SLOW_OPS)" in cluster log

pass 2382574 2018-04-10 20:52:31 2018-04-12 05:26:34 2018-04-12 07:48:37 2:22:03 0:10:16 2:11:47 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 2382575 2018-04-10 20:52:32 2018-04-12 05:26:35 2018-04-12 06:08:35 0:42:00 0:09:45 0:32:15 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 2382576 2018-04-10 20:52:33 2018-04-12 05:30:23 2018-04-12 05:58:23 0:28:00 0:09:11 0:18:49 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
fail 2382577 2018-04-10 20:52:33 2018-04-12 05:30:23 2018-04-12 06:28:24 0:58:01 0:21:58 0:36:03 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 06:16:22.420476 mon.a mon.0 172.21.15.145:6789/0 578 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382578 2018-04-10 20:52:34 2018-04-12 05:30:25 2018-04-12 07:42:27 2:12:02 0:42:29 1:29:33 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 07:27:32.837193 mon.a mon.0 172.21.15.201:6789/0 349 : cluster [WRN] Health check failed: 18 slow ops, oldest one blocked for 1283 sec (SLOW_OPS)" in cluster log

fail 2382579 2018-04-10 20:52:35 2018-04-12 05:32:34 2018-04-12 07:48:37 2:16:03 0:40:51 1:35:12 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 07:12:56.062035 mon.b mon.0 172.21.15.167:6789/0 232 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382580 2018-04-10 20:52:36 2018-04-12 05:32:34 2018-04-12 06:54:35 1:22:01 0:10:24 1:11:37 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 06:51:04.685242 mon.b mon.0 172.21.15.86:6789/0 174 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382581 2018-04-10 20:52:37 2018-04-12 05:34:28 2018-04-12 06:04:28 0:30:00 0:18:56 0:11:04 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 05:51:49.014762 mon.b mon.0 172.21.15.19:6789/0 138 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382582 2018-04-10 20:52:37 2018-04-12 05:34:30 2018-04-12 06:20:30 0:46:00 0:11:01 0:34:59 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 06:16:08.876720 mon.b mon.0 172.21.15.66:6789/0 271 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382583 2018-04-10 20:52:38 2018-04-12 05:34:30 2018-04-12 06:36:31 1:02:01 0:44:48 0:17:13 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 06:20:50.919851 mon.a mon.0 172.21.15.44:6789/0 203 : cluster [WRN] Health check failed: 19 slow ops, oldest one blocked for 824 sec (SLOW_OPS)" in cluster log

fail 2382584 2018-04-10 20:52:39 2018-04-12 05:34:34 2018-04-12 06:50:35 1:16:01 0:50:48 0:25:13 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 06:06:05.279259 mon.b mon.0 172.21.15.24:6789/0 303 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382585 2018-04-10 20:52:40 2018-04-12 05:36:29 2018-04-12 06:34:30 0:58:01 0:47:48 0:10:13 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 05:52:27.889071 mon.b mon.0 172.21.15.69:6789/0 139 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382586 2018-04-10 20:52:40 2018-04-12 05:36:35 2018-04-12 06:54:36 1:18:01 0:10:22 1:07:39 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 06:50:03.804536 mon.a mon.0 172.21.15.14:6789/0 320 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 63 sec (SLOW_OPS)" in cluster log

fail 2382587 2018-04-10 20:52:41 2018-04-12 05:38:30 2018-04-12 06:10:30 0:32:00 0:20:34 0:11:26 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 05:56:50.622195 mon.a mon.0 172.21.15.35:6789/0 162 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382588 2018-04-10 20:52:42 2018-04-12 05:38:30 2018-04-12 06:00:30 0:22:00 0:09:58 0:12:02 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 2382589 2018-04-10 20:52:43 2018-04-12 05:38:30 2018-04-12 06:00:30 0:22:00 0:09:18 0:12:42 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382590 2018-04-10 20:52:43 2018-04-12 05:40:34 2018-04-12 06:00:34 0:20:00 0:09:13 0:10:47 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 05:56:47.610406 mon.b mon.0 172.21.15.1:6789/0 146 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

pass 2382591 2018-04-10 20:52:44 2018-04-12 05:40:34 2018-04-12 06:02:34 0:22:00 0:08:46 0:13:14 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
fail 2382592 2018-04-10 20:52:45 2018-04-12 05:40:34 2018-04-12 06:16:34 0:36:00 0:21:35 0:14:25 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 06:06:07.423056 mon.a mon.0 172.21.15.60:6789/0 1312 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382593 2018-04-10 20:52:46 2018-04-12 05:42:28 2018-04-12 07:12:29 1:30:01 1:05:41 0:24:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 2382594 2018-04-10 20:52:46 2018-04-12 05:42:28 2018-04-12 06:38:28 0:56:00 0:29:56 0:26:04 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 06:15:11.543201 mon.b mon.0 172.21.15.1:6789/0 177 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382595 2018-04-10 20:52:47 2018-04-12 05:42:30 2018-04-12 06:14:30 0:32:00 0:14:22 0:17:38 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 06:05:26.470234 mon.a mon.0 172.21.15.89:6789/0 259 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382596 2018-04-10 20:52:48 2018-04-12 05:42:31 2018-04-12 06:28:31 0:46:00 0:24:25 0:21:35 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 06:10:10.285266 mon.b mon.0 172.21.15.23:6789/0 170 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382597 2018-04-10 20:52:49 2018-04-12 05:43:00 2018-04-12 06:11:00 0:28:00 0:10:26 0:17:34 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 06:07:44.888573 mon.b mon.0 172.21.15.99:6789/0 170 : cluster [WRN] Health check failed: 11 slow ops, oldest one blocked for 139 sec (SLOW_OPS)" in cluster log

fail 2382598 2018-04-10 20:52:50 2018-04-12 05:46:25 2018-04-12 06:50:26 1:04:01 0:45:13 0:18:48 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 06:32:31.550213 mon.a mon.0 172.21.15.192:6789/0 199 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382599 2018-04-10 20:52:50 2018-04-12 05:46:25 2018-04-12 06:44:26 0:58:01 0:42:31 0:15:30 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 06:18:20.938325 mon.b mon.0 172.21.15.47:6789/0 145 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 738 sec (SLOW_OPS)" in cluster log

fail 2382600 2018-04-10 20:52:51 2018-04-12 05:46:25 2018-04-12 06:20:25 0:34:00 0:22:41 0:11:19 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 06:12:19.051454 mon.b mon.0 172.21.15.38:6789/0 320 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382601 2018-04-10 20:52:52 2018-04-12 05:46:35 2018-04-12 06:16:35 0:30:00 0:08:31 0:21:29 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382602 2018-04-10 20:52:53 2018-04-12 05:48:32 2018-04-12 06:34:32 0:46:00 0:25:26 0:20:34 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 06:27:54.783324 mon.b mon.0 172.21.15.86:6789/0 181 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382603 2018-04-10 20:52:54 2018-04-12 05:48:34 2018-04-12 06:10:34 0:22:00 0:10:16 0:11:44 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 06:07:31.068885 mon.b mon.0 172.21.15.97:6789/0 239 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382604 2018-04-10 20:52:54 2018-04-12 05:50:27 2018-04-12 06:50:27 1:00:00 0:09:49 0:50:11 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 06:46:23.820253 mon.b mon.0 172.21.15.64:6789/0 270 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382605 2018-04-10 20:52:55 2018-04-12 05:50:30 2018-04-12 06:18:29 0:27:59 0:08:49 0:19:10 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 2382606 2018-04-10 20:52:56 2018-04-12 05:50:33 2018-04-12 06:14:32 0:23:59 0:08:40 0:15:19 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
pass 2382607 2018-04-10 20:52:57 2018-04-12 05:52:18 2018-04-12 06:46:18 0:54:00 0:22:18 0:31:42 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382608 2018-04-10 20:52:58 2018-04-12 05:52:21 2018-04-12 07:44:22 1:52:01 0:25:48 1:26:13 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 07:35:05.807535 mon.a mon.0 172.21.15.69:6789/0 336 : cluster [WRN] Health check failed: 10 slow ops, oldest one blocked for 652 sec (SLOW_OPS)" in cluster log

fail 2382609 2018-04-10 20:52:58 2018-04-12 05:52:24 2018-04-12 06:48:25 0:56:01 0:43:07 0:12:54 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 06:10:03.679891 mon.b mon.0 172.21.15.176:6789/0 138 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382610 2018-04-10 20:52:59 2018-04-12 05:52:34 2018-04-12 06:20:34 0:28:00 0:10:35 0:17:25 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 06:16:26.880177 mon.a mon.0 172.21.15.118:6789/0 203 : cluster [WRN] Health check failed: 8 slow ops, oldest one blocked for 72 sec (SLOW_OPS)" in cluster log

pass 2382611 2018-04-10 20:53:00 2018-04-12 05:54:21 2018-04-12 06:52:22 0:58:01 0:19:33 0:38:28 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
fail 2382612 2018-04-10 20:53:01 2018-04-12 05:56:23 2018-04-12 06:20:23 0:24:00 0:10:40 0:13:20 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 06:18:12.030975 mon.a mon.0 172.21.15.114:6789/0 242 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 130 sec (SLOW_OPS)" in cluster log

fail 2382613 2018-04-10 20:53:01 2018-04-12 05:57:09 2018-04-12 07:27:10 1:30:01 0:45:02 0:44:59 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 07:09:12.235206 mon.b mon.0 172.21.15.83:6789/0 198 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382614 2018-04-10 20:53:02 2018-04-12 05:57:09 2018-04-12 07:01:10 1:04:01 0:51:27 0:12:34 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 06:16:28.096977 mon.b mon.0 172.21.15.6:6789/0 306 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382615 2018-04-10 20:53:03 2018-04-12 05:58:32 2018-04-12 07:16:33 1:18:01 0:38:02 0:39:59 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 07:05:04.096153 mon.b mon.0 172.21.15.28:6789/0 142 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 1327 sec (SLOW_OPS)" in cluster log

fail 2382616 2018-04-10 20:53:04 2018-04-12 06:00:28 2018-04-12 06:50:28 0:50:00 0:09:21 0:40:39 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 06:46:39.694908 mon.b mon.0 172.21.15.163:6789/0 176 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 39 sec (SLOW_OPS)" in cluster log

fail 2382617 2018-04-10 20:53:05 2018-04-12 06:00:29 2018-04-12 09:00:33 3:00:04 0:23:07 2:36:57 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 08:44:07.890758 mon.b mon.0 172.21.15.100:6789/0 223 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 145 sec (SLOW_OPS)" in cluster log

pass 2382618 2018-04-10 20:53:05 2018-04-12 06:00:28 2018-04-12 06:56:29 0:56:01 0:09:54 0:46:07 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 2382619 2018-04-10 20:53:06 2018-04-12 06:00:29 2018-04-12 07:26:30 1:26:01 0:09:06 1:16:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 2382620 2018-04-10 20:53:07 2018-04-12 06:00:29 2018-04-12 06:50:29 0:50:00 0:09:26 0:40:34 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382621 2018-04-10 20:53:08 2018-04-12 06:00:31 2018-04-12 06:22:31 0:22:00 0:09:07 0:12:53 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 06:19:21.583073 mon.a mon.0 172.21.15.74:6789/0 256 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 52 sec (SLOW_OPS)" in cluster log

fail 2382622 2018-04-10 20:53:08 2018-04-12 06:00:31 2018-04-12 06:52:32 0:52:01 0:21:13 0:30:48 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 06:40:39.628915 mon.a mon.0 172.21.15.71:6789/0 1310 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382623 2018-04-10 20:53:09 2018-04-12 06:00:32 2018-04-12 07:16:33 1:16:01 1:04:09 0:11:52 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 2382624 2018-04-10 20:53:10 2018-04-12 06:00:33 2018-04-12 07:08:34 1:08:01 0:27:34 0:40:27 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 06:48:10.100601 mon.b mon.0 172.21.15.70:6789/0 177 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382625 2018-04-10 20:53:11 2018-04-12 06:00:35 2018-04-12 06:30:35 0:30:00 0:14:21 0:15:39 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
fail 2382626 2018-04-10 20:53:11 2018-04-12 06:02:44 2018-04-12 06:40:44 0:38:00 0:26:52 0:11:08 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 06:21:00.472904 mon.a mon.0 172.21.15.55:6789/0 328 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382627 2018-04-10 20:53:12 2018-04-12 06:04:37 2018-04-12 06:30:37 0:26:00 0:10:02 0:15:58 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 06:26:44.535530 mon.a mon.0 172.21.15.62:6789/0 200 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382628 2018-04-10 20:53:13 2018-04-12 06:04:37 2018-04-12 07:08:38 1:04:01 0:46:10 0:17:51 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 06:36:07.770615 mon.a mon.0 172.21.15.26:6789/0 324 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382629 2018-04-10 20:53:14 2018-04-12 06:06:38 2018-04-12 06:58:38 0:52:00 0:38:20 0:13:40 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 06:37:37.687628 mon.b mon.0 172.21.15.167:6789/0 137 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382630 2018-04-10 20:53:14 2018-04-12 06:06:38 2018-04-12 07:18:39 1:12:01 0:56:07 0:15:54 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 06:28:35.730036 mon.a mon.0 172.21.15.129:6789/0 324 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382631 2018-04-10 20:53:15 2018-04-12 06:06:38 2018-04-12 06:28:37 0:21:59 0:09:04 0:12:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 06:24:28.497215 mon.a mon.0 172.21.15.181:6789/0 253 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382632 2018-04-10 20:53:16 2018-04-12 06:07:02 2018-04-12 07:09:02 1:02:00 0:24:17 0:37:43 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
fail 2382633 2018-04-10 20:53:17 2018-04-12 06:08:44 2018-04-12 06:36:43 0:27:59 0:10:16 0:17:43 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 06:34:43.017046 mon.a mon.0 172.21.15.60:6789/0 160 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 137 sec (SLOW_OPS)" in cluster log

pass 2382634 2018-04-10 20:53:18 2018-04-12 06:08:44 2018-04-12 06:28:43 0:19:59 0:09:25 0:10:34 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 2382635 2018-04-10 20:53:18 2018-04-12 06:08:45 2018-04-12 06:34:43 0:25:58 0:08:56 0:17:02 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 2382636 2018-04-10 20:53:19 2018-04-12 06:10:15 2018-04-12 07:02:15 0:52:00 0:09:29 0:42:31 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
fail 2382637 2018-04-10 20:53:20 2018-04-12 06:10:31 2018-04-12 07:54:33 1:44:02 0:21:56 1:22:06 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 07:42:40.927841 mon.b mon.0 172.21.15.78:6789/0 670 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382638 2018-04-10 20:53:20 2018-04-12 06:10:35 2018-04-12 06:54:35 0:44:00 0:27:27 0:16:33 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 06:32:45.660024 mon.b mon.0 172.21.15.143:6789/0 170 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382639 2018-04-10 20:53:21 2018-04-12 06:11:02 2018-04-12 07:07:02 0:56:00 0:42:54 0:13:06 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 06:29:06.916383 mon.b mon.0 172.21.15.89:6789/0 151 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382640 2018-04-10 20:53:22 2018-04-12 06:12:35 2018-04-12 07:14:35 1:02:00 0:10:43 0:51:17 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 07:09:58.482798 mon.b mon.0 172.21.15.160:6789/0 291 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382641 2018-04-10 20:53:23 2018-04-12 06:12:36 2018-04-12 06:56:36 0:44:00 0:18:40 0:25:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 06:42:16.819048 mon.b mon.0 172.21.15.160:6789/0 136 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382642 2018-04-10 20:53:23 2018-04-12 06:12:41 2018-04-12 06:46:41 0:34:00 0:10:31 0:23:29 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 06:44:13.500968 mon.b mon.0 172.21.15.81:6789/0 175 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 153 sec (SLOW_OPS)" in cluster log

fail 2382643 2018-04-10 20:53:24 2018-04-12 06:13:02 2018-04-12 09:39:06 3:26:04 0:45:06 2:40:58 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 09:08:20.915581 mon.a mon.0 172.21.15.60:6789/0 194 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382644 2018-04-10 20:53:25 2018-04-12 06:14:28 2018-04-12 07:18:28 1:04:00 0:51:43 0:12:17 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 06:32:20.543554 mon.a mon.0 172.21.15.131:6789/0 331 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382645 2018-04-10 20:53:26 2018-04-12 06:14:31 2018-04-12 07:18:32 1:04:01 0:41:33 0:22:28 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 06:42:42.522007 mon.a mon.0 172.21.15.175:6789/0 246 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382646 2018-04-10 20:53:27 2018-04-12 06:14:32 2018-04-12 09:10:35 2:56:03 0:10:38 2:45:25 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:06:47.209079 mon.a mon.0 172.21.15.103:6789/0 202 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 49 sec (SLOW_OPS)" in cluster log

pass 2382647 2018-04-10 20:53:27 2018-04-12 06:14:33 2018-04-12 08:16:36 2:02:03 0:22:44 1:39:19 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
fail 2382648 2018-04-10 20:53:28 2018-04-12 06:16:33 2018-04-12 06:48:33 0:32:00 0:10:44 0:21:16 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 06:44:10.157613 mon.b mon.0 172.21.15.23:6789/0 175 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382649 2018-04-10 20:53:29 2018-04-12 06:16:35 2018-04-12 07:04:36 0:48:01 0:09:10 0:38:51 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382650 2018-04-10 20:53:30 2018-04-12 06:16:36 2018-04-12 06:52:36 0:36:00 0:11:59 0:24:01 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 06:46:42.819860 mon.a mon.0 172.21.15.35:6789/0 170 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 120 sec (SLOW_OPS)" in cluster log

pass 2382651 2018-04-10 20:53:30 2018-04-12 06:18:31 2018-04-12 07:46:32 1:28:01 0:51:32 0:36:29 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
pass 2382652 2018-04-10 20:53:31 2018-04-12 06:18:38 2018-04-12 06:50:38 0:32:00 0:21:39 0:10:21 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382653 2018-04-10 20:53:32 2018-04-12 06:20:25 2018-04-12 08:00:26 1:40:01 1:11:10 0:28:51 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 06:54:14.255315 mon.b mon.0 172.21.15.169:6789/0 152 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382654 2018-04-10 20:53:33 2018-04-12 06:20:26 2018-04-12 07:10:26 0:50:00 0:37:36 0:12:24 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 06:40:47.539112 mon.a mon.0 172.21.15.74:6789/0 328 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 138 sec (SLOW_OPS)" in cluster log

fail 2382655 2018-04-10 20:53:33 2018-04-12 06:20:31 2018-04-12 08:14:33 1:54:02 0:13:12 1:40:50 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 08:08:33.682255 mon.a mon.0 172.21.15.145:6789/0 170 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 142 sec (SLOW_OPS)" in cluster log

fail 2382656 2018-04-10 20:53:34 2018-04-12 06:20:33 2018-04-12 08:10:35 1:50:02 0:28:31 1:21:31 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 08:01:13.333991 mon.a mon.0 172.21.15.73:6789/0 350 : cluster [WRN] Health check failed: 18 slow ops, oldest one blocked for 784 sec (SLOW_OPS)" in cluster log

pass 2382657 2018-04-10 20:53:35 2018-04-12 06:20:35 2018-04-12 07:08:36 0:48:01 0:10:39 0:37:22 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382658 2018-04-10 20:53:36 2018-04-12 06:22:32 2018-04-12 08:14:34 1:52:02 0:50:52 1:01:10 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 07:59:37.909256 mon.a mon.0 172.21.15.24:6789/0 190 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382659 2018-04-10 20:53:36 2018-04-12 06:24:43 2018-04-12 07:54:44 1:30:01 0:39:42 0:50:19 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
fail 2382660 2018-04-10 20:53:37 2018-04-12 06:24:43 2018-04-12 09:36:47 3:12:04 1:01:11 2:10:53 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 08:43:06.508959 mon.b mon.0 172.21.15.57:6789/0 169 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382661 2018-04-10 20:53:38 2018-04-12 06:26:35 2018-04-12 07:06:34 0:39:59 0:09:16 0:30:43 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 07:03:12.448999 mon.a mon.0 172.21.15.49:6789/0 163 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 51 sec (SLOW_OPS)" in cluster log

fail 2382662 2018-04-10 20:53:39 2018-04-12 06:28:26 2018-04-12 07:04:26 0:36:00 0:16:08 0:19:52 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 06:54:17.539639 mon.a mon.0 172.21.15.60:6789/0 195 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 57 sec (SLOW_OPS)" in cluster log

fail 2382663 2018-04-10 20:53:40 2018-04-12 06:28:27 2018-04-12 07:10:27 0:42:00 0:11:11 0:30:49 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 07:06:37.676344 mon.b mon.0 172.21.15.59:6789/0 151 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382664 2018-04-10 20:53:40 2018-04-12 06:28:37 2018-04-12 06:48:36 0:19:59 0:09:56 0:10:03 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382665 2018-04-10 20:53:41 2018-04-12 06:28:39 2018-04-12 08:32:42 2:04:03 0:09:21 1:54:42 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 08:30:01.265653 mon.a mon.0 172.21.15.24:6789/0 154 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 60 sec (SLOW_OPS)" in cluster log

pass 2382666 2018-04-10 20:53:42 2018-04-12 06:28:44 2018-04-12 07:30:45 1:02:01 0:50:51 0:11:10 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
fail 2382667 2018-04-10 20:53:43 2018-04-12 06:30:25 2018-04-12 07:12:24 0:41:59 0:22:04 0:19:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 07:00:32.156428 mon.b mon.0 172.21.15.41:6789/0 490 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382668 2018-04-10 20:53:43 2018-04-12 06:30:34 2018-04-12 08:26:37 1:56:03 0:44:50 1:11:13 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 08:15:33.120406 mon.a mon.0 172.21.15.74:6789/0 333 : cluster [WRN] Health check failed: 17 slow ops, oldest one blocked for 1637 sec (SLOW_OPS)" in cluster log

pass 2382669 2018-04-10 20:53:44 2018-04-12 06:30:36 2018-04-12 08:16:38 1:46:02 0:48:05 0:57:57 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
fail 2382670 2018-04-10 20:53:45 2018-04-12 06:30:36 2018-04-12 07:24:36 0:54:00 0:11:53 0:42:07 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 07:18:51.191296 mon.b mon.0 172.21.15.94:6789/0 181 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382671 2018-04-10 20:53:46 2018-04-12 06:30:38 2018-04-12 07:14:38 0:44:00 0:17:58 0:26:02 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 07:02:41.345877 mon.a mon.0 172.21.15.109:6789/0 175 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382672 2018-04-10 20:53:47 2018-04-12 06:32:26 2018-04-12 07:10:26 0:38:00 0:11:00 0:27:00 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 07:05:17.707480 mon.b mon.0 172.21.15.163:6789/0 160 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382673 2018-04-10 20:53:47 2018-04-12 06:33:02 2018-04-12 07:51:03 1:18:01 0:52:04 0:25:57 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 07:37:50.056176 mon.a mon.0 172.21.15.19:6789/0 316 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382674 2018-04-10 20:53:48 2018-04-12 06:34:31 2018-04-12 07:36:32 1:02:01 0:49:32 0:12:29 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 07:17:20.939971 mon.a mon.0 172.21.15.202:6789/0 262 : cluster [WRN] Health check failed: 32 slow ops, oldest one blocked for 1464 sec (SLOW_OPS)" in cluster log

pass 2382675 2018-04-10 20:53:49 2018-04-12 06:34:33 2018-04-12 08:02:35 1:28:02 0:40:20 0:47:42 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2382676 2018-04-10 20:53:50 2018-04-12 06:34:45 2018-04-12 07:58:46 1:24:01 0:10:32 1:13:29 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 07:57:00.816249 mon.a mon.0 172.21.15.109:6789/0 331 : cluster [WRN] Health check failed: 12 slow ops, oldest one blocked for 167 sec (SLOW_OPS)" in cluster log

pass 2382677 2018-04-10 20:53:50 2018-04-12 06:36:32 2018-04-12 07:18:32 0:42:00 0:31:30 0:10:30 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
fail 2382678 2018-04-10 20:53:51 2018-04-12 06:36:34 2018-04-12 07:12:34 0:36:00 0:11:17 0:24:43 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 07:09:35.474312 mon.b mon.0 172.21.15.38:6789/0 168 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382679 2018-04-10 20:53:52 2018-04-12 06:36:44 2018-04-12 08:32:47 1:56:03 0:10:07 1:45:56 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 2382680 2018-04-10 20:53:53 2018-04-12 06:38:39 2018-04-12 07:22:39 0:44:00 0:09:01 0:34:59 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382681 2018-04-10 20:53:53 2018-04-12 06:38:39 2018-04-12 07:52:40 1:14:01 0:47:37 0:26:24 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 07:09:43.283123 mon.b mon.0 172.21.15.86:6789/0 133 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2382682 2018-04-10 20:53:54 2018-04-12 06:38:39 2018-04-12 07:32:40 0:54:01 0:21:39 0:32:22 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
pass 2382683 2018-04-10 20:53:55 2018-04-12 06:39:02 2018-04-12 08:03:03 1:24:01 1:10:03 0:13:58 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 2382684 2018-04-10 20:53:56 2018-04-12 06:40:54 2018-04-12 07:32:54 0:52:00 0:29:15 0:22:45 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 07:10:16.325762 mon.a mon.0 172.21.15.36:6789/0 327 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382685 2018-04-10 20:53:56 2018-04-12 06:42:44 2018-04-12 07:32:45 0:50:01 0:14:13 0:35:48 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
fail 2382686 2018-04-10 20:53:57 2018-04-12 06:44:27 2018-04-12 07:52:28 1:08:01 0:26:23 0:41:38 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 07:42:48.873128 mon.b mon.0 172.21.15.28:6789/0 189 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 38 sec (SLOW_OPS)" in cluster log

pass 2382687 2018-04-10 20:53:58 2018-04-12 06:46:20 2018-04-12 07:30:20 0:44:00 0:10:38 0:33:22 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382688 2018-04-10 20:53:59 2018-04-12 06:46:25 2018-04-12 08:04:26 1:18:01 0:55:08 0:22:53 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 07:46:13.669701 mon.a mon.0 172.21.15.2:6789/0 192 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 44 sec (SLOW_OPS)" in cluster log

fail 2382689 2018-04-10 20:54:00 2018-04-12 06:46:35 2018-04-12 08:02:36 1:16:01 0:39:17 0:36:44 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 07:41:15.233801 mon.a mon.0 172.21.15.14:6789/0 164 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 741 sec (SLOW_OPS)" in cluster log

fail 2382690 2018-04-10 20:54:00 2018-04-12 06:46:42 2018-04-12 07:22:42 0:36:00 0:20:59 0:15:01 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 07:16:29.263116 mon.b mon.0 172.21.15.78:6789/0 178 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 567 sec (SLOW_OPS)" in cluster log

pass 2382691 2018-04-10 20:54:01 2018-04-12 07:34:46 2018-04-12 08:00:46 0:26:00 0:09:13 0:16:47 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382692 2018-04-10 20:54:02 2018-04-12 07:36:30 2018-04-12 08:12:30 0:36:00 0:18:11 0:17:49 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 08:08:06.171502 mon.b mon.0 172.21.15.100:6789/0 170 : cluster [WRN] Health check failed: 15 slow ops, oldest one blocked for 517 sec (SLOW_OPS)" in cluster log

fail 2382693 2018-04-10 20:54:03 2018-04-12 07:36:33 2018-04-12 09:32:35 1:56:02 0:10:17 1:45:45 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 09:30:33.479165 mon.a mon.0 172.21.15.178:6789/0 273 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 143 sec (SLOW_OPS)" in cluster log

fail 2382694 2018-04-10 20:54:03 2018-04-12 07:36:39 2018-04-12 08:04:39 0:28:00 0:10:01 0:17:59 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 08:00:49.065156 mon.b mon.0 172.21.15.151:6789/0 271 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382695 2018-04-10 20:54:04 2018-04-12 07:36:40 2018-04-12 08:08:40 0:32:00 0:08:48 0:23:12 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382696 2018-04-10 20:54:05 2018-04-12 07:38:38 2018-04-12 08:50:40 1:12:02 0:50:32 0:21:30 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 08:06:29.508447 mon.b mon.0 172.21.15.8:6789/0 134 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382697 2018-04-10 20:54:06 2018-04-12 07:38:43 2018-04-12 08:24:44 0:46:01 0:21:50 0:24:11 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382698 2018-04-10 20:54:07 2018-04-12 07:38:45 2018-04-12 08:42:46 1:04:01 0:36:07 0:27:54 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 08:12:02.212611 mon.b mon.0 172.21.15.26:6789/0 183 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382699 2018-04-10 20:54:07 2018-04-12 07:38:46 2018-04-12 09:20:47 1:42:01 0:44:03 0:57:58 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 08:43:11.156653 mon.b mon.0 172.21.15.88:6789/0 136 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382700 2018-04-10 20:54:08 2018-04-12 07:39:04 2018-04-12 08:03:04 0:24:00 0:11:52 0:12:08 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 07:57:37.093170 mon.b mon.0 172.21.15.174:6789/0 177 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382701 2018-04-10 20:54:09 2018-04-12 07:40:34 2018-04-12 08:26:35 0:46:01 0:16:40 0:29:21 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 08:14:48.654396 mon.a mon.0 172.21.15.112:6789/0 173 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 57 sec (SLOW_OPS)" in cluster log

pass 2382702 2018-04-10 20:54:10 2018-04-12 07:40:34 2018-04-12 08:46:35 1:06:01 0:10:30 0:55:31 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382703 2018-04-10 20:54:10 2018-04-12 07:40:34 2018-04-12 10:44:38 3:04:04 0:51:00 2:13:04 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 10:34:52.091365 mon.b mon.0 172.21.15.62:6789/0 190 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 185 sec (SLOW_OPS)" in cluster log

fail 2382704 2018-04-10 20:54:11 2018-04-12 07:40:38 2018-04-12 08:40:39 1:00:01 0:48:52 0:11:09 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 07:56:40.744616 mon.b mon.0 172.21.15.33:6789/0 170 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382705 2018-04-10 20:54:12 2018-04-12 07:40:39 2018-04-12 08:44:40 1:04:01 0:48:17 0:15:44 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 08:02:15.797960 mon.b mon.0 172.21.15.67:6789/0 145 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382706 2018-04-10 20:54:13 2018-04-12 07:40:41 2018-04-12 08:04:41 0:24:00 0:10:54 0:13:06 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 08:01:24.786612 mon.b mon.0 172.21.15.94:6789/0 176 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382707 2018-04-10 20:54:13 2018-04-12 07:41:05 2018-04-12 08:37:05 0:56:00 0:28:09 0:27:51 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 2382708 2018-04-10 20:54:14 2018-04-12 07:42:29 2018-04-12 08:12:29 0:30:00 0:10:22 0:19:38 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 2382709 2018-04-10 20:54:15 2018-04-12 07:42:36 2018-04-12 08:06:36 0:24:00 0:09:14 0:14:46 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 2382710 2018-04-10 20:54:16 2018-04-12 07:42:41 2018-04-12 08:04:41 0:22:00 0:09:32 0:12:28 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382711 2018-04-10 20:54:16 2018-04-12 07:42:44 2018-04-12 09:28:46 1:46:02 0:49:58 0:56:04 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 08:55:22.563272 mon.a mon.0 172.21.15.19:6789/0 248 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 616 sec (SLOW_OPS)" in cluster log

pass 2382712 2018-04-10 20:54:17 2018-04-12 07:44:32 2018-04-12 08:18:32 0:34:00 0:21:41 0:12:19 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382713 2018-04-10 20:54:18 2018-04-12 07:44:32 2018-04-12 09:18:33 1:34:01 1:07:10 0:26:51 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 08:39:42.434575 mon.a mon.0 172.21.15.129:6789/0 268 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 1338 sec (SLOW_OPS)" in cluster log

fail 2382714 2018-04-10 20:54:19 2018-04-12 07:44:37 2018-04-12 08:34:37 0:50:00 0:37:54 0:12:06 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 08:02:34.422937 mon.a mon.0 172.21.15.84:6789/0 314 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382715 2018-04-10 20:54:20 2018-04-12 07:44:39 2018-04-12 08:28:39 0:44:00 0:15:48 0:28:12 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 08:18:35.206732 mon.b mon.0 172.21.15.55:6789/0 133 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382716 2018-04-10 20:54:20 2018-04-12 07:44:41 2018-04-12 08:32:41 0:48:00 0:23:07 0:24:53 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 08:14:25.602670 mon.b mon.0 172.21.15.109:6789/0 170 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382717 2018-04-10 20:54:21 2018-04-12 07:44:45 2018-04-12 08:08:45 0:24:00 0:10:09 0:13:51 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 2382718 2018-04-10 20:54:22 2018-04-12 07:46:38 2018-04-12 08:50:39 1:04:01 0:48:13 0:15:48 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 2382719 2018-04-10 20:54:23 2018-04-12 07:46:38 2018-04-12 08:40:39 0:54:01 0:43:04 0:10:57 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 08:14:22.396410 mon.b mon.0 172.21.15.167:6789/0 145 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382720 2018-04-10 20:54:23 2018-04-12 07:46:39 2018-04-12 08:42:40 0:56:01 0:20:09 0:35:52 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 08:27:37.845907 mon.b mon.0 172.21.15.87:6789/0 168 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382721 2018-04-10 20:54:24 2018-04-12 07:48:47 2018-04-12 08:16:47 0:28:00 0:09:27 0:18:33 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 08:13:12.768997 mon.a mon.0 172.21.15.37:6789/0 167 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 49 sec (SLOW_OPS)" in cluster log

fail 2382722 2018-04-10 20:54:25 2018-04-12 07:48:47 2018-04-12 08:30:48 0:42:01 0:19:15 0:22:46 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 08:17:54.175202 mon.b mon.0 172.21.15.23:6789/0 164 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382723 2018-04-10 20:54:26 2018-04-12 07:48:47 2018-04-12 08:08:46 0:19:59 0:09:39 0:10:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 08:05:24.038125 mon.b mon.0 172.21.15.19:6789/0 233 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382724 2018-04-10 20:54:26 2018-04-12 07:48:47 2018-04-12 09:14:49 1:26:02 0:09:49 1:16:13 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382725 2018-04-10 20:54:27 2018-04-12 07:50:49 2018-04-12 08:18:49 0:28:00 0:10:51 0:17:09 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 08:13:18.757769 mon.b mon.0 172.21.15.81:6789/0 135 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

pass 2382726 2018-04-10 20:54:28 2018-04-12 07:50:49 2018-04-12 08:18:49 0:28:00 0:09:34 0:18:26 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
fail 2382727 2018-04-10 20:54:29 2018-04-12 07:50:50 2018-04-12 08:34:50 0:44:00 0:21:52 0:22:08 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 08:22:40.626686 mon.a mon.0 172.21.15.71:6789/0 740 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382728 2018-04-10 20:54:29 2018-04-12 07:51:05 2018-04-12 08:43:05 0:52:00 0:39:41 0:12:19 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 08:27:45.568482 mon.a mon.0 172.21.15.60:6789/0 346 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

pass 2382729 2018-04-10 20:54:30 2018-04-12 07:52:38 2018-04-12 09:02:39 1:10:01 0:52:33 0:17:28 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
fail 2382730 2018-04-10 20:54:31 2018-04-12 07:52:38 2018-04-12 08:58:39 1:06:01 0:09:54 0:56:07 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 08:54:41.653777 mon.b mon.0 172.21.15.33:6789/0 167 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2382731 2018-04-10 20:54:32 2018-04-12 07:52:37 2018-04-12 08:22:37 0:30:00 0:19:25 0:10:35 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
fail 2382732 2018-04-10 20:54:33 2018-04-12 07:52:38 2018-04-12 08:28:38 0:36:00 0:11:05 0:24:55 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 08:22:56.925307 mon.b mon.0 172.21.15.49:6789/0 277 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382733 2018-04-10 20:54:33 2018-04-12 07:52:38 2018-04-12 09:18:39 1:26:01 0:48:57 0:37:04 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 09:13:57.765563 mon.a mon.0 172.21.15.35:6789/0 350 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 794 sec (SLOW_OPS)" in cluster log

fail 2382734 2018-04-10 20:54:34 2018-04-12 07:52:38 2018-04-12 09:44:40 1:52:02 0:50:50 1:01:12 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 08:59:13.726898 mon.b mon.0 172.21.15.50:6789/0 171 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382735 2018-04-10 20:54:35 2018-04-12 07:52:38 2018-04-12 09:36:40 1:44:02 0:50:59 0:53:03 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 08:51:34.027516 mon.b mon.0 172.21.15.59:6789/0 142 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382736 2018-04-10 20:54:36 2018-04-12 07:52:41 2018-04-12 08:14:41 0:22:00 0:10:10 0:11:50 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 08:11:18.921239 mon.b mon.0 172.21.15.78:6789/0 185 : cluster [WRN] Health check failed: 8 slow ops, oldest one blocked for 116 sec (SLOW_OPS)" in cluster log

pass 2382737 2018-04-10 20:54:36 2018-04-12 07:53:13 2018-04-12 08:43:14 0:50:01 0:21:03 0:28:58 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
fail 2382738 2018-04-10 20:54:37 2018-04-12 07:54:38 2018-04-12 09:38:40 1:44:02 0:10:07 1:33:55 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 09:34:53.778570 mon.a mon.0 172.21.15.35:6789/0 199 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382739 2018-04-10 20:54:38 2018-04-12 07:54:38 2018-04-12 08:26:38 0:32:00 0:09:26 0:22:34 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 2382740 2018-04-10 20:54:39 2018-04-12 07:54:41 2018-04-12 08:14:41 0:20:00 0:10:11 0:09:49 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382741 2018-04-10 20:54:39 2018-04-12 07:54:45 2018-04-12 08:34:46 0:40:01 0:09:32 0:30:29 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 08:29:54.991846 mon.b mon.0 172.21.15.145:6789/0 143 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382742 2018-04-10 20:54:40 2018-04-12 07:54:46 2018-04-12 08:36:46 0:42:00 0:20:53 0:21:07 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382743 2018-04-10 20:54:41 2018-04-12 07:55:13 2018-04-12 09:17:14 1:22:01 1:07:08 0:14:53 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 08:15:00.543677 mon.a mon.0 172.21.15.131:6789/0 161 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382744 2018-04-10 20:54:42 2018-04-12 07:56:49 2018-04-12 09:06:50 1:10:01 0:24:52 0:45:09 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 08:47:06.824495 mon.b mon.0 172.21.15.40:6789/0 166 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382745 2018-04-10 20:54:42 2018-04-12 07:56:49 2018-04-12 08:28:50 0:32:01 0:14:20 0:17:41 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 08:20:54.664284 mon.b mon.0 172.21.15.79:6789/0 235 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382746 2018-04-10 20:54:43 2018-04-12 07:58:36 2018-04-12 08:48:36 0:50:00 0:30:01 0:19:59 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 08:23:43.891577 mon.b mon.0 172.21.15.69:6789/0 305 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382747 2018-04-10 20:54:44 2018-04-12 07:58:36 2018-04-12 09:36:38 1:38:02 0:10:21 1:27:41 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382748 2018-04-10 20:54:45 2018-04-12 07:58:39 2018-04-12 10:06:41 2:08:02 1:01:45 1:06:17 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 09:32:15.328920 mon.a mon.0 172.21.15.23:6789/0 189 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382749 2018-04-10 20:54:45 2018-04-12 07:58:47 2018-04-12 09:10:48 1:12:01 0:41:46 0:30:15 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 08:45:55.755269 mon.a mon.0 172.21.15.6:6789/0 269 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 740 sec (SLOW_OPS)" in cluster log

fail 2382750 2018-04-10 20:54:46 2018-04-12 08:00:37 2018-04-12 09:52:39 1:52:02 0:21:45 1:30:17 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 09:43:22.813220 mon.a mon.0 172.21.15.5:6789/0 211 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382751 2018-04-10 20:54:47 2018-04-12 08:00:37 2018-04-12 08:30:37 0:30:00 0:08:43 0:21:17 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382752 2018-04-10 20:54:48 2018-04-12 08:00:41 2018-04-12 08:42:41 0:42:00 0:24:27 0:17:33 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 08:36:51.270494 mon.a mon.0 172.21.15.5:6789/0 191 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382753 2018-04-10 20:54:49 2018-04-12 08:00:47 2018-04-12 08:42:47 0:42:00 0:10:00 0:32:00 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382754 2018-04-10 20:54:49 2018-04-12 08:00:47 2018-04-12 08:26:47 0:26:00 0:10:22 0:15:38 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 08:23:29.868585 mon.b mon.0 172.21.15.17:6789/0 462 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382755 2018-04-10 20:54:50 2018-04-12 08:02:36 2018-04-12 09:06:38 1:04:02 0:10:04 0:53:58 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:01:20.553971 mon.b mon.0 172.21.15.18:6789/0 122 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382756 2018-04-10 20:54:51 2018-04-12 08:02:37 2018-04-12 09:06:38 1:04:01 0:08:58 0:55:03 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 09:01:53.993647 mon.a mon.0 172.21.15.112:6789/0 160 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382757 2018-04-10 20:54:52 2018-04-12 08:02:45 2018-04-12 08:36:45 0:34:00 0:21:38 0:12:22 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 08:24:48.914235 mon.b mon.0 172.21.15.94:6789/0 638 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382758 2018-04-10 20:54:52 2018-04-12 08:03:13 2018-04-12 08:49:14 0:46:01 0:35:15 0:10:46 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 08:38:47.246925 mon.a mon.0 172.21.15.2:6789/0 342 : cluster [WRN] Health check failed: 17 slow ops, oldest one blocked for 1142 sec (SLOW_OPS)" in cluster log

fail 2382759 2018-04-10 20:54:53 2018-04-12 08:03:13 2018-04-12 09:11:14 1:08:01 0:44:52 0:23:09 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 08:33:57.717828 mon.a mon.0 172.21.15.37:6789/0 269 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 189 sec (SLOW_OPS)" in cluster log

fail 2382760 2018-04-10 20:54:54 2018-04-12 08:04:35 2018-04-12 08:34:35 0:30:00 0:10:59 0:19:01 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 08:30:25.576031 mon.a mon.0 172.21.15.110:6789/0 318 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382761 2018-04-10 20:54:55 2018-04-12 08:04:41 2018-04-12 08:44:41 0:40:00 0:19:18 0:20:42 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 08:30:12.408669 mon.b mon.0 172.21.15.50:6789/0 141 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382762 2018-04-10 20:54:55 2018-04-12 08:04:42 2018-04-12 08:32:42 0:28:00 0:10:13 0:17:47 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382763 2018-04-10 20:54:56 2018-04-12 08:04:42 2018-04-12 09:18:43 1:14:01 0:49:54 0:24:07 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 08:49:33.834476 mon.b mon.0 172.21.15.1:6789/0 313 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382764 2018-04-10 20:54:57 2018-04-12 08:06:39 2018-04-12 10:36:42 2:30:03 0:49:26 1:40:37 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 10:07:04.691459 mon.a mon.0 172.21.15.59:6789/0 330 : cluster [WRN] Health check failed: 17 slow ops, oldest one blocked for 896 sec (SLOW_OPS)" in cluster log

fail 2382765 2018-04-10 20:54:57 2018-04-12 08:06:39 2018-04-12 09:22:40 1:16:01 0:38:25 0:37:36 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 09:11:23.482878 mon.b mon.0 172.21.15.84:6789/0 139 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 1352 sec (SLOW_OPS)" in cluster log

fail 2382766 2018-04-10 20:54:58 2018-04-12 08:08:40 2018-04-12 08:30:40 0:22:00 0:10:59 0:11:01 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 08:26:33.851306 mon.a mon.0 172.21.15.14:6789/0 208 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 39 sec (SLOW_OPS)" in cluster log

fail 2382767 2018-04-10 20:54:59 2018-04-12 08:08:41 2018-04-12 08:48:41 0:40:00 0:23:05 0:16:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 08:43:33.796434 mon.a mon.0 172.21.15.140:6789/0 160 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382768 2018-04-10 20:55:00 2018-04-12 08:08:41 2018-04-12 08:28:41 0:20:00 0:09:42 0:10:18 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382769 2018-04-10 20:55:00 2018-04-12 08:08:46 2018-04-12 08:52:47 0:44:01 0:09:24 0:34:37 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 08:49:51.108513 mon.b mon.0 172.21.15.77:6789/0 203 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382770 2018-04-10 20:55:01 2018-04-12 08:08:48 2018-04-12 09:26:49 1:18:01 0:10:38 1:07:23 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:22:16.635748 mon.a mon.0 172.21.15.24:6789/0 189 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382771 2018-04-10 20:55:02 2018-04-12 08:10:39 2018-04-12 08:44:40 0:34:01 0:09:11 0:24:50 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
pass 2382772 2018-04-10 20:55:03 2018-04-12 08:10:39 2018-04-12 09:44:41 1:34:02 0:19:31 1:14:31 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382773 2018-04-10 20:55:03 2018-04-12 08:12:37 2018-04-12 10:02:39 1:50:02 1:14:10 0:35:52 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 08:54:36.055581 mon.b mon.0 172.21.15.167:6789/0 245 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382774 2018-04-10 20:55:04 2018-04-12 08:12:38 2018-04-12 09:20:38 1:08:00 0:28:38 0:39:22 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 08:59:18.961155 mon.a mon.0 172.21.15.5:6789/0 196 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382775 2018-04-10 20:55:05 2018-04-12 08:12:38 2018-04-12 10:22:41 2:10:03 0:15:04 1:54:59 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 10:17:44.266669 mon.a mon.0 172.21.15.15:6789/0 270 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382776 2018-04-10 20:55:06 2018-04-12 08:12:41 2018-04-12 08:54:42 0:42:01 0:31:32 0:10:29 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 08:29:15.525864 mon.b mon.0 172.21.15.138:6789/0 167 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382777 2018-04-10 20:55:06 2018-04-12 08:12:47 2018-04-12 08:50:48 0:38:01 0:11:11 0:26:50 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 08:46:33.073295 mon.a mon.0 172.21.15.103:6789/0 330 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 45 sec (SLOW_OPS)" in cluster log

fail 2382778 2018-04-10 20:55:07 2018-04-12 08:14:43 2018-04-12 09:20:44 1:06:01 0:50:05 0:15:56 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 09:03:22.276890 mon.a mon.0 172.21.15.56:6789/0 334 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

pass 2382779 2018-04-10 20:55:08 2018-04-12 08:14:43 2018-04-12 09:28:44 1:14:01 0:38:03 0:35:58 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
fail 2382780 2018-04-10 20:55:09 2018-04-12 08:14:43 2018-04-12 09:54:45 1:40:02 1:00:46 0:39:16 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 08:58:50.562899 mon.b mon.0 172.21.15.99:6789/0 181 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382781 2018-04-10 20:55:09 2018-04-12 08:14:43 2018-04-12 08:34:43 0:20:00 0:08:41 0:11:19 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 08:32:29.515477 mon.a mon.0 172.21.15.114:6789/0 167 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382782 2018-04-10 20:55:10 2018-04-12 08:14:43 2018-04-12 08:56:43 0:42:00 0:25:12 0:16:48 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 08:49:53.229080 mon.b mon.0 172.21.15.36:6789/0 175 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 38 sec (SLOW_OPS)" in cluster log

fail 2382783 2018-04-10 20:55:11 2018-04-12 08:14:57 2018-04-12 09:38:58 1:24:01 0:10:14 1:13:47 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 09:33:41.379008 mon.a mon.0 172.21.15.129:6789/0 262 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382784 2018-04-10 20:55:12 2018-04-12 08:16:46 2018-04-12 08:46:46 0:30:00 0:09:35 0:20:25 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 2382785 2018-04-10 20:55:12 2018-04-12 08:16:46 2018-04-12 08:46:46 0:30:00 0:12:09 0:17:51 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
pass 2382786 2018-04-10 20:55:13 2018-04-12 08:16:48 2018-04-12 09:14:49 0:58:01 0:09:34 0:48:27 smithi master multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
fail 2382787 2018-04-10 20:55:14 2018-04-12 08:18:43 2018-04-12 09:06:43 0:48:00 0:21:51 0:26:09 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 08:54:15.026237 mon.b mon.0 172.21.15.86:6789/0 479 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382788 2018-04-10 20:55:15 2018-04-12 08:18:43 2018-04-12 09:38:44 1:20:01 0:32:54 0:47:07 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 09:26:30.616307 mon.b mon.0 172.21.15.94:6789/0 187 : cluster [WRN] Health check failed: 17 slow ops, oldest one blocked for 974 sec (SLOW_OPS)" in cluster log

fail 2382789 2018-04-10 20:55:15 2018-04-12 08:18:51 2018-04-12 09:30:51 1:12:00 0:50:41 0:21:19 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 08:45:50.532462 mon.a mon.0 172.21.15.136:6789/0 170 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382790 2018-04-10 20:55:16 2018-04-12 08:18:51 2018-04-12 09:06:51 0:48:00 0:10:45 0:37:15 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 09:01:49.711183 mon.b mon.0 172.21.15.67:6789/0 172 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382791 2018-04-10 20:55:17 2018-04-12 08:22:48 2018-04-12 08:58:48 0:36:00 0:18:25 0:17:35 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 08:45:37.017609 mon.b mon.0 172.21.15.79:6789/0 239 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382792 2018-04-10 20:55:18 2018-04-12 08:24:53 2018-04-12 09:24:54 1:00:01 0:10:23 0:49:38 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382793 2018-04-10 20:55:18 2018-04-12 08:26:44 2018-04-12 10:32:46 2:06:02 0:50:33 1:15:29 smithi master centos multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 10:22:33.027396 mon.a mon.0 172.21.15.73:6789/0 330 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 825 sec (SLOW_OPS)" in cluster log

fail 2382794 2018-04-10 20:55:19 2018-04-12 08:26:45 2018-04-12 10:08:46 1:42:01 0:51:21 0:50:40 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 09:32:15.812962 mon.b mon.0 172.21.15.44:6789/0 179 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382795 2018-04-10 20:55:20 2018-04-12 08:26:44 2018-04-12 09:56:46 1:30:02 0:41:34 0:48:28 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 09:21:30.706429 mon.b mon.0 172.21.15.64:6789/0 136 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382796 2018-04-10 20:55:20 2018-04-12 08:26:48 2018-04-12 08:58:48 0:32:00 0:10:54 0:21:06 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 08:53:27.459071 mon.b mon.0 172.21.15.52:6789/0 174 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 64 sec (SLOW_OPS)" in cluster log

pass 2382797 2018-04-10 20:55:21 2018-04-12 08:28:49 2018-04-12 09:06:49 0:38:00 0:23:49 0:14:11 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
fail 2382798 2018-04-10 20:55:22 2018-04-12 08:28:49 2018-04-12 08:54:48 0:25:59 0:11:00 0:14:59 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 08:50:31.531368 mon.a mon.0 172.21.15.145:6789/0 194 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2382799 2018-04-10 20:55:23 2018-04-12 08:28:49 2018-04-12 08:50:48 0:21:59 0:10:46 0:11:13 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
pass 2382800 2018-04-10 20:55:23 2018-04-12 08:28:49 2018-04-12 08:54:48 0:25:59 0:11:04 0:14:55 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382801 2018-04-10 20:55:24 2018-04-12 08:28:51 2018-04-12 10:14:52 1:46:01 0:43:58 1:02:03 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 09:37:32.436122 mon.b mon.0 172.21.15.135:6789/0 295 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382802 2018-04-10 20:55:25 2018-04-12 08:30:47 2018-04-12 09:06:47 0:36:00 0:21:49 0:14:11 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 08:53:33.868894 mon.a mon.0 172.21.15.24:6789/0 677 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382803 2018-04-10 20:55:26 2018-04-12 08:30:47 2018-04-12 09:50:48 1:20:01 1:07:15 0:12:46 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 09:26:13.987430 mon.a mon.0 172.21.15.14:6789/0 238 : cluster [WRN] Health check failed: 43 slow ops, oldest one blocked for 1484 sec (SLOW_OPS)" in cluster log

fail 2382804 2018-04-10 20:55:26 2018-04-12 08:30:49 2018-04-12 09:06:49 0:36:00 0:25:08 0:10:52 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 08:46:33.838064 mon.b mon.0 172.21.15.109:6789/0 142 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382805 2018-04-10 20:55:27 2018-04-12 08:32:51 2018-04-12 09:28:51 0:56:00 0:14:13 0:41:47 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 09:21:00.561220 mon.b mon.0 172.21.15.17:6789/0 178 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382806 2018-04-10 20:55:28 2018-04-12 08:32:51 2018-04-12 10:40:53 2:08:02 0:21:09 1:46:53 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 10:32:25.919282 mon.a mon.0 172.21.15.5:6789/0 168 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382807 2018-04-10 20:55:29 2018-04-12 08:32:51 2018-04-12 09:32:51 1:00:00 0:10:10 0:49:50 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382808 2018-04-10 20:55:29 2018-04-12 08:32:51 2018-04-12 10:36:53 2:04:02 0:32:41 1:31:21 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 10:17:06.808919 mon.a mon.0 172.21.15.37:6789/0 241 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382809 2018-04-10 20:55:30 2018-04-12 08:34:46 2018-04-12 11:30:49 2:56:03 0:42:02 2:14:01 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 11:05:55.311563 mon.b mon.0 172.21.15.19:6789/0 171 : cluster [WRN] Health check failed: 9 slow ops, oldest one blocked for 746 sec (SLOW_OPS)" in cluster log

pass 2382810 2018-04-10 20:55:31 2018-04-12 08:34:46 2018-04-12 09:56:46 1:22:00 0:58:58 0:23:02 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2382811 2018-04-10 20:55:32 2018-04-12 08:34:46 2018-04-12 09:08:45 0:33:59 0:10:24 0:23:35 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:03:40.470865 mon.b mon.0 172.21.15.44:6789/0 164 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382812 2018-04-10 20:55:32 2018-04-12 08:34:47 2018-04-12 09:06:47 0:32:00 0:14:58 0:17:02 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 2382813 2018-04-10 20:55:33 2018-04-12 08:34:51 2018-04-12 10:08:52 1:34:01 0:10:35 1:23:26 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 2382814 2018-04-10 20:55:34 2018-04-12 08:36:48 2018-04-12 11:40:51 3:04:03 0:09:20 2:54:43 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382815 2018-04-10 20:55:35 2018-04-12 08:36:48 2018-04-12 09:18:48 0:42:00 0:11:02 0:30:58 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:14:43.695504 mon.b mon.0 172.21.15.33:6789/0 175 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382816 2018-04-10 20:55:35 2018-04-12 08:37:07 2018-04-12 10:51:09 2:14:02 0:45:01 1:29:01 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 10:28:26.316800 mon.a mon.0 172.21.15.14:6789/0 216 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 344 sec (SLOW_OPS)" in cluster log

pass 2382817 2018-04-10 20:55:36 2018-04-12 08:40:51 2018-04-12 09:36:51 0:56:00 0:22:20 0:33:40 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382818 2018-04-10 20:55:37 2018-04-12 08:40:51 2018-04-12 09:50:52 1:10:01 1:00:22 0:09:39 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 08:56:45.209110 mon.a mon.0 172.21.15.97:6789/0 167 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382819 2018-04-10 20:55:38 2018-04-12 08:42:42 2018-04-12 10:16:43 1:34:01 0:59:52 0:34:09 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 09:25:22.484564 mon.a mon.0 172.21.15.38:6789/0 205 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382820 2018-04-10 20:55:38 2018-04-11 16:09:02 2018-04-11 16:41:02 0:32:00 0:10:53 0:21:07 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-11 16:34:52.103007 mon.a mon.0 172.21.15.165:6789/0 256 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382821 2018-04-10 20:55:39 2018-04-12 08:42:42 2018-04-12 09:16:42 0:34:00 0:18:52 0:15:08 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
fail 2382822 2018-04-10 20:55:40 2018-04-12 08:42:47 2018-04-12 09:04:47 0:22:00 0:10:05 0:11:55 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 09:01:43.541805 mon.a mon.0 172.21.15.47:6789/0 239 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382823 2018-04-10 20:55:41 2018-04-12 08:42:48 2018-04-12 10:14:50 1:32:02 0:33:41 0:58:21 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 09:57:21.071801 mon.b mon.0 172.21.15.26:6789/0 137 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382824 2018-04-10 20:55:42 2018-04-12 08:43:07 2018-04-12 09:47:07 1:04:00 0:51:50 0:12:10 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 09:23:37.043415 mon.a mon.0 172.21.15.192:6789/0 170 : cluster [WRN] Health check failed: 11 slow ops, oldest one blocked for 1470 sec (SLOW_OPS)" in cluster log

fail 2382825 2018-04-10 20:55:42 2018-04-12 08:43:24 2018-04-12 10:05:25 1:22:01 0:38:17 0:43:44 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 09:33:59.280569 mon.a mon.0 172.21.15.33:6789/0 198 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382826 2018-04-10 20:55:43 2018-04-12 08:44:51 2018-04-12 09:04:50 0:19:59 0:09:04 0:10:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:02:33.942412 mon.b mon.0 172.21.15.17:6789/0 139 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 112 sec (SLOW_OPS)" in cluster log

fail 2382827 2018-04-10 20:55:44 2018-04-12 08:44:51 2018-04-12 09:28:51 0:44:00 0:29:10 0:14:50 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 09:06:53.995594 mon.b mon.0 172.21.15.69:6789/0 170 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382828 2018-04-10 20:55:45 2018-04-12 08:44:51 2018-04-12 09:10:50 0:25:59 0:10:29 0:15:30 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 2382829 2018-04-10 20:55:46 2018-04-12 08:46:44 2018-04-12 09:24:44 0:38:00 0:10:02 0:27:58 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382830 2018-04-10 20:55:46 2018-04-12 08:46:44 2018-04-12 09:36:45 0:50:01 0:09:59 0:40:02 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:33:06.921344 mon.b mon.0 172.21.15.61:6789/0 181 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382831 2018-04-10 20:55:47 2018-04-12 08:46:47 2018-04-12 09:40:48 0:54:01 0:38:22 0:15:39 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 09:19:51.657755 mon.a mon.0 172.21.15.15:6789/0 320 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2382832 2018-04-10 20:55:48 2018-04-12 08:46:48 2018-04-12 09:32:48 0:46:00 0:19:00 0:27:00 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382833 2018-04-10 20:55:49 2018-04-12 08:48:45 2018-04-12 10:16:47 1:28:02 1:05:02 0:23:00 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 09:17:54.379397 mon.b mon.0 172.21.15.174:6789/0 169 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382834 2018-04-10 20:55:49 2018-04-12 08:48:45 2018-04-12 10:24:47 1:36:02 0:30:07 1:05:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 10:01:20.036133 mon.b mon.0 172.21.15.17:6789/0 145 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382835 2018-04-10 20:55:50 2018-04-12 08:49:15 2018-04-12 09:45:15 0:56:00 0:16:30 0:39:30 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 09:35:44.938294 mon.b mon.0 172.21.15.1:6789/0 166 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382836 2018-04-10 20:55:51 2018-04-12 08:50:49 2018-04-12 10:00:49 1:10:00 0:23:05 0:46:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
pass 2382837 2018-04-10 20:55:52 2018-04-12 08:50:49 2018-04-12 09:28:49 0:38:00 0:09:37 0:28:23 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382838 2018-04-10 20:55:53 2018-04-12 08:50:50 2018-04-12 10:06:50 1:16:00 0:40:13 0:35:47 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 09:55:41.112428 mon.b mon.0 172.21.15.36:6789/0 153 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 614 sec (SLOW_OPS)" in cluster log

fail 2382839 2018-04-10 20:55:53 2018-04-12 08:50:50 2018-04-12 09:48:51 0:58:01 0:40:46 0:17:15 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 09:13:51.166023 mon.a mon.0 172.21.15.41:6789/0 321 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382840 2018-04-10 20:55:54 2018-04-12 08:52:57 2018-04-12 09:50:58 0:58:01 0:20:11 0:37:50 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2382841 2018-04-10 20:55:55 2018-04-12 08:54:43 2018-04-12 10:24:45 1:30:02 0:10:04 1:19:58 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 10:21:38.544126 mon.a mon.0 172.21.15.28:6789/0 338 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 53 sec (SLOW_OPS)" in cluster log

fail 2382842 2018-04-10 20:55:56 2018-04-12 08:54:49 2018-04-12 10:56:52 2:02:03 0:17:18 1:44:45 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 10:44:43.957403 mon.a mon.0 172.21.15.33:6789/0 249 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 58 sec (SLOW_OPS)" in cluster log

fail 2382843 2018-04-10 20:55:56 2018-04-12 08:54:50 2018-04-12 09:16:49 0:21:59 0:10:40 0:11:19 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 09:14:06.726688 mon.b mon.0 172.21.15.52:6789/0 174 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 35 sec (SLOW_OPS)" in cluster log

fail 2382844 2018-04-10 20:55:57 2018-04-12 08:56:45 2018-04-12 09:56:45 1:00:00 0:09:01 0:50:59 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 09:53:51.416083 mon.a mon.0 172.21.15.60:6789/0 250 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382845 2018-04-10 20:55:58 2018-04-12 08:58:40 2018-04-12 12:12:44 3:14:04 0:09:14 3:04:50 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 12:09:38.343083 mon.b mon.0 172.21.15.47:6789/0 276 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 52 sec (SLOW_OPS)" in cluster log

fail 2382846 2018-04-10 20:55:59 2018-04-12 08:58:49 2018-04-12 09:48:49 0:50:00 0:39:59 0:10:01 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 09:29:01.980794 mon.a mon.0 172.21.15.79:6789/0 209 : cluster [WRN] Health check failed: 17 slow ops, oldest one blocked for 897 sec (SLOW_OPS)" in cluster log

fail 2382847 2018-04-10 20:55:59 2018-04-12 08:58:49 2018-04-12 09:36:49 0:38:00 0:22:22 0:15:38 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 09:32:31.231474 mon.a mon.0 172.21.15.112:6789/0 1392 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382848 2018-04-10 20:56:00 2018-04-12 09:00:34 2018-04-12 10:46:36 1:46:02 0:41:08 1:04:54 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 10:31:26.832180 mon.b mon.0 172.21.15.94:6789/0 156 : cluster [WRN] Health check failed: 11 slow ops, oldest one blocked for 1290 sec (SLOW_OPS)" in cluster log

pass 2382849 2018-04-10 20:56:01 2018-04-12 09:02:40 2018-04-12 10:22:41 1:20:01 1:03:51 0:16:10 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
fail 2382850 2018-04-10 20:56:02 2018-04-12 09:04:58 2018-04-12 10:08:58 1:04:00 0:11:11 0:52:49 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 10:03:56.905777 mon.b mon.0 172.21.15.19:6789/0 143 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 74 sec (SLOW_OPS)" in cluster log

pass 2382851 2018-04-10 20:56:02 2018-04-12 09:04:58 2018-04-12 10:18:59 1:14:01 0:19:26 0:54:35 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
fail 2382852 2018-04-10 20:56:03 2018-04-12 09:06:41 2018-04-12 10:18:40 1:11:59 0:10:24 1:01:35 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 10:14:26.472504 mon.a mon.0 172.21.15.154:6789/0 149 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382853 2018-04-10 20:56:04 2018-04-12 09:06:41 2018-04-12 11:22:42 2:16:01 0:37:02 1:38:59 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 11:13:33.642911 mon.a mon.0 172.21.15.73:6789/0 167 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382854 2018-04-10 20:56:05 2018-04-12 09:06:44 2018-04-12 10:34:45 1:28:01 0:47:17 0:40:44 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 09:52:47.774511 mon.a mon.0 172.21.15.129:6789/0 157 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382855 2018-04-10 20:56:05 2018-04-12 09:06:48 2018-04-12 10:56:50 1:50:02 0:49:11 1:00:51 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2382856 2018-04-10 20:56:06 2018-04-12 09:06:48 2018-04-12 10:06:49 1:00:01 0:09:25 0:50:36 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 10:03:54.520603 mon.a mon.0 172.21.15.79:6789/0 156 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 87 sec (SLOW_OPS)" in cluster log

fail 2382857 2018-04-10 20:56:07 2018-04-12 09:06:50 2018-04-12 09:50:50 0:44:00 0:29:36 0:14:24 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 09:43:43.793755 mon.a mon.0 172.21.15.37:6789/0 224 : cluster [WRN] Health check failed: 15 slow ops, oldest one blocked for 998 sec (SLOW_OPS)" in cluster log

fail 2382858 2018-04-10 20:56:08 2018-04-12 09:06:50 2018-04-12 10:36:52 1:30:02 0:10:37 1:19:25 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 10:32:23.509960 mon.a mon.0 172.21.15.38:6789/0 238 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382859 2018-04-10 20:56:08 2018-04-12 09:06:52 2018-04-12 09:44:52 0:38:00 0:09:24 0:28:36 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 09:41:57.315280 mon.a mon.0 172.21.15.24:6789/0 314 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382860 2018-04-10 20:56:09 2018-04-12 09:06:52 2018-04-12 09:54:52 0:48:00 0:09:41 0:38:19 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:52:09.725395 mon.b mon.0 172.21.15.83:6789/0 320 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 77 sec (SLOW_OPS)" in cluster log

fail 2382861 2018-04-10 20:56:10 2018-04-12 09:08:55 2018-04-12 10:14:55 1:06:00 0:40:25 0:25:35 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 10:03:35.414828 mon.a mon.0 172.21.15.47:6789/0 226 : cluster [WRN] Health check failed: 22 slow ops, oldest one blocked for 1462 sec (SLOW_OPS)" in cluster log

pass 2382862 2018-04-10 20:56:11 2018-04-12 09:10:44 2018-04-12 09:44:44 0:34:00 0:21:07 0:12:53 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382863 2018-04-10 20:56:11 2018-04-12 09:10:49 2018-04-12 10:46:51 1:36:02 1:04:43 0:31:19 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 10:04:05.296067 mon.b mon.0 172.21.15.164:6789/0 176 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382864 2018-04-10 20:56:12 2018-04-12 09:10:52 2018-04-12 11:04:53 1:54:01 0:28:56 1:25:05 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 10:42:45.005568 mon.a mon.0 172.21.15.17:6789/0 176 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 134 sec (SLOW_OPS)" in cluster log

fail 2382865 2018-04-10 20:56:13 2018-04-12 09:11:15 2018-04-12 10:23:16 1:12:01 0:15:35 0:56:26 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 10:16:37.675119 mon.a mon.0 172.21.15.18:6789/0 207 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382866 2018-04-10 20:56:14 2018-04-12 09:15:01 2018-04-12 12:17:04 3:02:03 0:18:43 2:43:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 12:02:43.896366 mon.a mon.0 172.21.15.88:6789/0 156 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382867 2018-04-10 20:56:14 2018-04-12 09:15:01 2018-04-12 11:03:02 1:48:01 0:10:13 1:37:48 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 10:59:01.992445 mon.b mon.0 172.21.15.63:6789/0 175 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382868 2018-04-10 20:56:15 2018-04-12 09:16:52 2018-04-12 11:10:54 1:54:02 0:30:56 1:23:06 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 11:01:19.709218 mon.b mon.0 172.21.15.144:6789/0 235 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382869 2018-04-10 20:56:16 2018-04-12 09:16:52 2018-04-12 10:20:53 1:04:01 0:42:04 0:21:57 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 09:56:01.888476 mon.b mon.0 172.21.15.114:6789/0 177 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382870 2018-04-10 20:56:17 2018-04-12 09:17:15 2018-04-12 10:01:15 0:44:00 0:19:32 0:24:28 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 09:48:01.131601 mon.b mon.0 172.21.15.86:6789/0 148 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382871 2018-04-10 20:56:17 2018-04-12 09:18:43 2018-04-12 10:14:44 0:56:01 0:09:55 0:46:06 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 10:11:40.446917 mon.a mon.0 172.21.15.82:6789/0 207 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382872 2018-04-10 20:56:18 2018-04-12 09:18:44 2018-04-12 10:00:44 0:42:00 0:17:40 0:24:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 09:47:52.627831 mon.b mon.0 172.21.15.169:6789/0 139 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382873 2018-04-10 20:56:19 2018-04-12 09:18:45 2018-04-12 09:40:44 0:21:59 0:10:14 0:11:45 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 09:37:31.121726 mon.b mon.0 172.21.15.88:6789/0 168 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382874 2018-04-10 20:56:20 2018-04-12 09:18:50 2018-04-12 09:40:49 0:21:59 0:09:28 0:12:31 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 09:37:45.610108 mon.a mon.0 172.21.15.62:6789/0 253 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 41 sec (SLOW_OPS)" in cluster log

fail 2382875 2018-04-10 20:56:20 2018-04-12 09:20:48 2018-04-12 09:56:48 0:36:00 0:10:04 0:25:56 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:52:37.746511 mon.a mon.0 172.21.15.82:6789/0 202 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382876 2018-04-10 20:56:21 2018-04-12 09:20:48 2018-04-12 10:20:48 1:00:00 0:09:38 0:50:22 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 10:18:16.648235 mon.b mon.0 172.21.15.167:6789/0 177 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382877 2018-04-10 20:56:22 2018-04-12 09:20:49 2018-04-12 09:58:49 0:38:00 0:22:23 0:15:37 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 09:54:50.499548 mon.a mon.0 172.21.15.18:6789/0 1910 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382878 2018-04-10 20:56:23 2018-04-12 09:22:36 2018-04-12 10:16:36 0:54:00 0:40:57 0:13:03 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 10:02:30.973560 mon.a mon.0 172.21.15.101:6789/0 174 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382879 2018-04-10 20:56:23 2018-04-12 09:22:41 2018-04-12 10:28:42 1:06:01 0:40:20 0:25:41 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 09:55:30.382817 mon.a mon.0 172.21.15.35:6789/0 340 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 132 sec (SLOW_OPS)" in cluster log

fail 2382880 2018-04-10 20:56:24 2018-04-12 09:24:54 2018-04-12 10:26:54 1:02:00 0:10:42 0:51:18 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 10:22:22.859274 mon.a mon.0 172.21.15.23:6789/0 246 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382881 2018-04-10 20:56:25 2018-04-12 09:24:55 2018-04-12 09:58:55 0:34:00 0:20:10 0:13:50 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 09:44:24.465897 mon.a mon.0 172.21.15.120:6789/0 203 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

pass 2382882 2018-04-10 20:56:26 2018-04-12 09:27:01 2018-04-12 10:05:01 0:38:00 0:09:50 0:28:10 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382883 2018-04-10 20:56:26 2018-04-12 09:28:45 2018-04-12 10:34:46 1:06:01 0:43:56 0:22:05 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 10:04:08.210687 mon.a mon.0 172.21.15.140:6789/0 147 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382884 2018-04-10 20:56:27 2018-04-12 09:28:47 2018-04-12 10:44:48 1:16:01 0:47:28 0:28:33 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 10:03:39.148188 mon.a mon.0 172.21.15.41:6789/0 153 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382885 2018-04-10 20:56:28 2018-04-12 09:28:50 2018-04-12 10:34:51 1:06:01 0:48:22 0:17:39 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2382886 2018-04-10 20:56:29 2018-04-12 09:28:52 2018-04-12 09:48:51 0:19:59 0:09:02 0:10:57 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:45:40.741846 mon.b mon.0 172.21.15.19:6789/0 240 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382887 2018-04-10 20:56:29 2018-04-12 09:28:52 2018-04-12 10:24:52 0:56:00 0:20:51 0:35:09 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 10:09:21.670361 mon.a mon.0 172.21.15.118:6789/0 325 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382888 2018-04-10 20:56:30 2018-04-12 09:30:53 2018-04-12 09:56:53 0:26:00 0:09:35 0:16:25 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 09:52:07.931148 mon.b mon.0 172.21.15.94:6789/0 152 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382889 2018-04-10 20:56:31 2018-04-12 09:32:37 2018-04-12 09:56:36 0:23:59 0:10:03 0:13:56 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382890 2018-04-10 20:56:31 2018-04-12 09:32:52 2018-04-12 10:28:52 0:56:00 0:10:00 0:46:00 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 10:24:14.957731 mon.b mon.0 172.21.15.145:6789/0 221 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382891 2018-04-10 20:56:32 2018-04-12 09:32:53 2018-04-12 10:26:53 0:54:00 0:10:14 0:43:46 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 10:22:15.097387 mon.a mon.0 172.21.15.8:6789/0 338 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 43 sec (SLOW_OPS)" in cluster log

pass 2382892 2018-04-10 20:56:33 2018-04-12 09:36:39 2018-04-12 10:38:40 1:02:01 0:20:50 0:41:11 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382893 2018-04-10 20:56:34 2018-04-12 09:36:41 2018-04-12 10:56:43 1:20:02 1:06:00 0:14:02 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 10:34:51.307113 mon.a mon.0 172.21.15.88:6789/0 408 : cluster [WRN] Health check failed: 30 slow ops, oldest one blocked for 1407 sec (SLOW_OPS)" in cluster log

fail 2382894 2018-04-10 20:56:34 2018-04-12 09:36:46 2018-04-12 10:30:46 0:54:00 0:24:20 0:29:40 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 10:12:31.767456 mon.a mon.0 172.21.15.112:6789/0 258 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382895 2018-04-10 20:56:35 2018-04-12 09:36:48 2018-04-12 10:30:49 0:54:01 0:15:52 0:38:09 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 10:21:10.207422 mon.a mon.0 172.21.15.33:6789/0 206 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382896 2018-04-10 20:56:36 2018-04-12 09:36:50 2018-04-12 10:10:50 0:34:00 0:20:45 0:13:15 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
fail 2382897 2018-04-10 20:56:36 2018-04-12 09:36:52 2018-04-12 10:14:53 0:38:01 0:10:23 0:27:38 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 10:09:19.502573 mon.b mon.0 172.21.15.83:6789/0 200 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382898 2018-04-10 20:56:37 2018-04-12 09:38:47 2018-04-12 10:42:48 1:04:01 0:43:41 0:20:20 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 10:15:17.235806 mon.b mon.0 172.21.15.63:6789/0 144 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382899 2018-04-10 20:56:38 2018-04-12 09:38:47 2018-04-12 10:36:48 0:58:01 0:43:12 0:14:49 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 10:15:34.525371 mon.a mon.0 172.21.15.50:6789/0 228 : cluster [WRN] Health check failed: 23 slow ops, oldest one blocked for 978 sec (SLOW_OPS)" in cluster log

fail 2382900 2018-04-10 20:56:39 2018-04-12 09:39:00 2018-04-12 10:43:00 1:04:00 0:20:18 0:43:42 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 10:35:12.975552 mon.b mon.0 172.21.15.47:6789/0 136 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382901 2018-04-10 20:56:39 2018-04-12 09:39:07 2018-04-12 09:59:07 0:20:00 0:09:37 0:10:23 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 09:56:05.723922 mon.b mon.0 172.21.15.15:6789/0 177 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2382902 2018-04-10 20:56:40 2018-04-12 09:40:51 2018-04-12 10:16:51 0:36:00 0:23:58 0:12:02 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
pass 2382903 2018-04-10 20:56:41 2018-04-12 09:40:51 2018-04-12 10:38:52 0:58:01 0:11:14 0:46:47 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382904 2018-04-10 20:56:42 2018-04-12 09:40:51 2018-04-12 10:34:52 0:54:01 0:09:12 0:44:49 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 10:31:21.806074 mon.b mon.0 172.21.15.143:6789/0 213 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382905 2018-04-10 20:56:42 2018-04-12 09:40:51 2018-04-12 10:02:51 0:22:00 0:09:40 0:12:20 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 10:00:17.460039 mon.b mon.0 172.21.15.28:6789/0 181 : cluster [WRN] Health check failed: 14 slow ops, oldest one blocked for 93 sec (SLOW_OPS)" in cluster log

fail 2382906 2018-04-10 20:56:43 2018-04-12 09:44:42 2018-04-12 10:16:41 0:31:59 0:09:32 0:22:27 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 10:12:41.696967 mon.a mon.0 172.21.15.2:6789/0 322 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 57 sec (SLOW_OPS)" in cluster log

fail 2382907 2018-04-10 20:56:44 2018-04-12 09:44:42 2018-04-12 10:20:42 0:36:00 0:22:17 0:13:43 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 10:09:51.988977 mon.a mon.0 172.21.15.74:6789/0 660 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382908 2018-04-10 20:56:45 2018-04-12 09:44:45 2018-04-12 11:10:47 1:26:02 0:46:04 0:39:58 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 11:00:34.174832 mon.b mon.0 172.21.15.109:6789/0 147 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 1908 sec (SLOW_OPS)" in cluster log

fail 2382909 2018-04-10 20:56:45 2018-04-12 09:44:53 2018-04-12 11:10:54 1:26:01 1:03:37 0:22:24 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 10:15:17.633622 mon.a mon.0 172.21.15.60:6789/0 216 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 148 sec (SLOW_OPS)" in cluster log

fail 2382910 2018-04-10 20:56:46 2018-04-12 09:45:25 2018-04-12 11:45:27 2:00:02 0:10:20 1:49:42 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 11:40:57.656986 mon.b mon.0 172.21.15.149:6789/0 238 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382911 2018-04-10 20:56:47 2018-04-12 09:47:18 2018-04-12 11:39:21 1:52:03 0:19:06 1:32:57 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 11:25:46.956566 mon.b mon.0 172.21.15.109:6789/0 181 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 53 sec (SLOW_OPS)" in cluster log

pass 2382912 2018-04-10 20:56:48 2018-04-12 09:49:00 2018-04-12 10:11:00 0:22:00 0:09:50 0:12:10 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382913 2018-04-10 20:56:48 2018-04-12 09:49:01 2018-04-12 11:13:02 1:24:01 0:44:28 0:39:33 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 10:57:04.232816 mon.b mon.0 172.21.15.174:6789/0 162 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 206 sec (SLOW_OPS)" in cluster log

pass 2382914 2018-04-10 20:56:49 2018-04-12 09:49:00 2018-04-12 11:09:01 1:20:01 0:51:09 0:28:52 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
fail 2382915 2018-04-10 20:56:50 2018-04-12 09:50:58 2018-04-12 12:43:02 2:52:04 0:40:31 2:11:33 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 12:35:58.175026 mon.b mon.0 172.21.15.58:6789/0 169 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382916 2018-04-10 20:56:51 2018-04-12 09:50:58 2018-04-12 10:40:59 0:50:01 0:09:41 0:40:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 10:37:08.577568 mon.b mon.0 172.21.15.45:6789/0 152 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382917 2018-04-10 20:56:52 2018-04-12 09:50:58 2018-04-12 10:48:59 0:58:01 0:23:29 0:34:32 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 10:31:16.115734 mon.a mon.0 172.21.15.82:6789/0 190 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382918 2018-04-10 20:56:52 2018-04-12 09:50:59 2018-04-12 11:05:00 1:14:01 0:10:40 1:03:21 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:01:30.384661 mon.a mon.0 172.21.15.41:6789/0 171 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382919 2018-04-10 20:56:53 2018-04-12 09:52:43 2018-04-12 10:38:43 0:46:00 0:09:58 0:36:02 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 10:34:55.327876 mon.b mon.0 172.21.15.157:6789/0 274 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382920 2018-04-10 20:56:54 2018-04-12 09:54:46 2018-04-12 12:18:49 2:24:03 0:10:33 2:13:30 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 12:15:08.839574 mon.b mon.0 172.21.15.50:6789/0 254 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382921 2018-04-10 20:56:55 2018-04-12 09:54:54 2018-04-12 11:46:55 1:52:01 0:10:07 1:41:54 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 11:42:37.119655 mon.b mon.0 172.21.15.110:6789/0 173 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 37 sec (SLOW_OPS)" in cluster log

pass 2382922 2018-04-10 20:56:55 2018-04-12 09:56:46 2018-04-12 11:04:47 1:08:01 0:19:33 0:48:28 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382923 2018-04-10 20:56:56 2018-04-12 09:56:47 2018-04-12 12:32:50 2:36:03 1:07:30 1:28:33 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 12:08:37.791641 mon.b mon.0 172.21.15.36:6789/0 210 : cluster [WRN] Health check failed: 17 slow ops, oldest one blocked for 1563 sec (SLOW_OPS)" in cluster log

fail 2382924 2018-04-10 20:56:57 2018-04-12 09:56:47 2018-04-12 11:00:47 1:04:00 0:27:57 0:36:03 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 10:40:51.248879 mon.b mon.0 172.21.15.49:6789/0 142 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 127 sec (SLOW_OPS)" in cluster log

fail 2382925 2018-04-10 20:56:57 2018-04-12 09:56:47 2018-04-12 10:26:47 0:30:00 0:16:03 0:13:57 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 10:20:56.847297 mon.a mon.0 172.21.15.139:6789/0 340 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 154 sec (SLOW_OPS)" in cluster log

pass 2382926 2018-04-10 20:56:58 2018-04-12 09:56:49 2018-04-12 10:44:49 0:48:00 0:24:32 0:23:28 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
fail 2382927 2018-04-10 20:56:59 2018-04-12 09:56:54 2018-04-12 10:22:53 0:25:59 0:09:39 0:16:20 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 10:17:41.809387 mon.b mon.0 172.21.15.45:6789/0 125 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382928 2018-04-10 20:56:59 2018-04-12 09:58:51 2018-04-12 11:02:51 1:04:00 0:49:05 0:14:55 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 10:27:43.712513 mon.a mon.0 172.21.15.69:6789/0 160 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382929 2018-04-10 20:57:00 2018-04-12 09:58:56 2018-04-12 11:06:57 1:08:01 0:38:22 0:29:39 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 10:46:38.881669 mon.a mon.0 172.21.15.74:6789/0 321 : cluster [WRN] Health check failed: 9 slow ops, oldest one blocked for 742 sec (SLOW_OPS)" in cluster log

pass 2382930 2018-04-10 20:57:01 2018-04-12 09:59:16 2018-04-12 11:23:17 1:24:01 0:58:55 0:25:06 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2382931 2018-04-10 20:57:01 2018-04-12 10:00:54 2018-04-12 11:24:55 1:24:01 0:10:13 1:13:48 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:21:28.004502 mon.b mon.0 172.21.15.119:6789/0 174 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382932 2018-04-10 20:57:02 2018-04-12 10:00:54 2018-04-12 11:04:54 1:04:00 0:24:40 0:39:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 10:46:28.340637 mon.b mon.0 172.21.15.35:6789/0 235 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382933 2018-04-10 20:57:03 2018-04-12 10:01:17 2018-04-12 11:13:18 1:12:01 0:11:37 1:00:24 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:08:27.786354 mon.b mon.0 172.21.15.4:6789/0 307 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 111 sec (SLOW_OPS)" in cluster log

pass 2382934 2018-04-10 20:57:04 2018-04-12 10:02:50 2018-04-12 10:36:50 0:34:00 0:09:01 0:24:59 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382935 2018-04-10 20:57:04 2018-04-12 10:02:52 2018-04-12 10:24:52 0:22:00 0:09:40 0:12:20 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 10:20:16.969805 mon.b mon.0 172.21.15.49:6789/0 222 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382936 2018-04-10 20:57:05 2018-04-12 10:05:11 2018-04-12 10:43:11 0:38:00 0:09:28 0:28:32 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 10:40:20.098255 mon.b mon.0 172.21.15.1:6789/0 165 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382937 2018-04-10 20:57:06 2018-04-12 10:05:27 2018-04-12 10:47:27 0:42:00 0:22:15 0:19:45 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382938 2018-04-10 20:57:07 2018-04-12 10:06:45 2018-04-12 11:16:46 1:10:01 0:32:53 0:37:08 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 11:06:13.401057 mon.b mon.0 172.21.15.26:6789/0 240 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382939 2018-04-10 20:57:07 2018-04-12 10:06:50 2018-04-12 12:04:52 1:58:02 1:07:13 0:50:49 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 11:06:06.005322 mon.b mon.0 172.21.15.78:6789/0 183 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382940 2018-04-10 20:57:08 2018-04-12 10:06:52 2018-04-12 10:34:51 0:27:59 0:10:07 0:17:52 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 10:30:37.247290 mon.a mon.0 172.21.15.26:6789/0 258 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 81 sec (SLOW_OPS)" in cluster log

fail 2382941 2018-04-10 20:57:09 2018-04-12 10:08:55 2018-04-12 10:50:56 0:42:01 0:18:54 0:23:07 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 10:37:56.841747 mon.b mon.0 172.21.15.4:6789/0 163 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2382942 2018-04-10 20:57:10 2018-04-12 10:08:56 2018-04-12 10:40:55 0:31:59 0:10:15 0:21:44 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 10:37:17.950509 mon.b mon.0 172.21.15.70:6789/0 222 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382943 2018-04-10 20:57:10 2018-04-12 10:08:59 2018-04-12 12:23:02 2:14:03 0:45:48 1:28:15 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 11:53:11.560043 mon.a mon.0 172.21.15.120:6789/0 153 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382944 2018-04-10 20:57:11 2018-04-12 10:10:52 2018-04-12 11:36:53 1:26:01 0:51:54 0:34:07 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 11:16:12.740015 mon.a mon.0 172.21.15.38:6789/0 264 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382945 2018-04-10 20:57:12 2018-04-12 10:11:01 2018-04-12 11:05:02 0:54:01 0:40:56 0:13:05 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 10:50:17.627596 mon.a mon.0 172.21.15.2:6789/0 333 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 1187 sec (SLOW_OPS)" in cluster log

fail 2382946 2018-04-10 20:57:13 2018-04-12 10:14:55 2018-04-12 11:02:55 0:48:00 0:09:57 0:38:03 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 10:58:14.180418 mon.b mon.0 172.21.15.187:6789/0 233 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382947 2018-04-10 20:57:13 2018-04-12 10:14:55 2018-04-12 11:16:55 1:02:00 0:24:29 0:37:31 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 11:12:12.985316 mon.b mon.0 172.21.15.1:6789/0 176 : cluster [WRN] Health check failed: 9 slow ops, oldest one blocked for 801 sec (SLOW_OPS)" in cluster log

fail 2382948 2018-04-10 20:57:14 2018-04-12 10:14:55 2018-04-12 11:24:56 1:10:01 0:09:19 1:00:42 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/yes.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:21:21.868656 mon.b mon.0 172.21.15.17:6789/0 135 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382949 2018-04-10 20:57:15 2018-04-12 10:14:55 2018-04-12 11:54:56 1:40:01 0:09:35 1:30:26 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 11:50:49.227120 mon.b mon.0 172.21.15.47:6789/0 275 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 41 sec (SLOW_OPS)" in cluster log

fail 2382950 2018-04-10 20:57:16 2018-04-12 10:14:56 2018-04-12 10:44:56 0:30:00 0:13:01 0:16:59 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 10:40:46.280462 mon.b mon.0 172.21.15.66:6789/0 313 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 170 sec (SLOW_OPS)" in cluster log

fail 2382951 2018-04-10 20:57:18 2018-04-12 10:16:41 2018-04-12 11:26:42 1:10:01 0:50:56 0:19:05 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 10:54:11.995114 mon.b mon.0 172.21.15.8:6789/0 188 : cluster [WRN] Health check failed: 11 slow ops, oldest one blocked for 814 sec (SLOW_OPS)" in cluster log

fail 2382952 2018-04-10 20:57:21 2018-04-12 10:16:43 2018-04-12 11:16:43 1:00:00 0:21:04 0:38:56 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 11:04:39.857301 mon.b mon.0 172.21.15.40:6789/0 812 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382953 2018-04-10 20:57:22 2018-04-12 10:16:44 2018-04-12 11:48:45 1:32:01 1:17:55 0:14:06 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 11:23:11.183726 mon.a mon.0 172.21.15.15:6789/0 228 : cluster [WRN] Health check failed: 12 slow ops, oldest one blocked for 2740 sec (SLOW_OPS)" in cluster log

fail 2382954 2018-04-10 20:57:23 2018-04-12 10:16:51 2018-04-12 10:56:51 0:40:00 0:26:18 0:13:42 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 10:37:42.464887 mon.b mon.0 172.21.15.167:6789/0 143 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 132 sec (SLOW_OPS)" in cluster log

fail 2382955 2018-04-10 20:57:24 2018-04-12 10:16:52 2018-04-12 11:20:53 1:04:01 0:14:56 0:49:05 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 11:11:05.199941 mon.b mon.0 172.21.15.33:6789/0 166 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382956 2018-04-10 20:57:24 2018-04-12 10:18:42 2018-04-12 10:58:42 0:40:00 0:23:01 0:16:59 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
fail 2382957 2018-04-10 20:57:25 2018-04-12 10:19:00 2018-04-12 11:15:00 0:56:00 0:10:25 0:45:35 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:11:03.559420 mon.b mon.0 172.21.15.167:6789/0 202 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382958 2018-04-10 20:57:26 2018-04-12 10:20:44 2018-04-12 11:26:44 1:06:00 0:40:11 0:25:49 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 2382959 2018-04-10 20:57:27 2018-04-12 10:20:49 2018-04-12 11:14:50 0:54:01 0:41:20 0:12:41 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 10:40:11.143908 mon.a mon.0 172.21.15.24:6789/0 325 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2382960 2018-04-10 20:57:27 2018-04-12 10:20:54 2018-04-12 11:46:55 1:26:01 0:57:11 0:28:50 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2382961 2018-04-10 20:57:28 2018-04-12 10:22:51 2018-04-12 11:26:51 1:04:00 0:10:08 0:53:52 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:22:21.270137 mon.b mon.0 172.21.15.149:6789/0 178 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382962 2018-04-10 20:57:29 2018-04-12 10:22:51 2018-04-12 10:52:50 0:29:59 0:16:10 0:13:49 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 10:43:47.851801 mon.a mon.0 172.21.15.23:6789/0 152 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 57 sec (SLOW_OPS)" in cluster log

pass 2382963 2018-04-10 20:57:30 2018-04-12 10:22:54 2018-04-12 10:50:54 0:28:00 0:10:35 0:17:25 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 2382964 2018-04-10 20:57:31 2018-04-12 10:23:18 2018-04-12 11:15:18 0:52:00 0:09:10 0:42:50 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382965 2018-04-10 20:57:31 2018-04-12 10:24:46 2018-04-12 11:34:47 1:10:01 0:11:59 0:58:02 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:29:28.586709 mon.a mon.0 172.21.15.74:6789/0 212 : cluster [WRN] Health check failed: 11 slow ops, oldest one blocked for 120 sec (SLOW_OPS)" in cluster log

fail 2382966 2018-04-10 20:57:32 2018-04-12 10:24:48 2018-04-12 11:34:49 1:10:01 0:50:41 0:19:20 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 10:58:41.377763 mon.a mon.0 172.21.15.102:6789/0 327 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382967 2018-04-10 20:57:33 2018-04-12 10:24:53 2018-04-12 12:40:55 2:16:02 0:22:34 1:53:28 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 12:38:19.670138 mon.b mon.0 172.21.15.167:6789/0 1329 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382968 2018-04-10 20:57:34 2018-04-12 10:24:54 2018-04-12 11:18:54 0:54:00 0:34:56 0:19:04 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 2382969 2018-04-10 20:57:35 2018-04-12 10:26:49 2018-04-12 11:42:50 1:16:01 0:56:18 0:19:43 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 10:53:53.165878 mon.a mon.0 172.21.15.37:6789/0 333 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 182 sec (SLOW_OPS)" in cluster log

fail 2382970 2018-04-10 20:57:35 2018-04-12 10:26:54 2018-04-12 11:42:55 1:16:01 0:10:43 1:05:18 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 11:36:56.397974 mon.a mon.0 172.21.15.56:6789/0 251 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2382971 2018-04-10 20:57:36 2018-04-12 10:26:56 2018-04-12 11:16:56 0:50:00 0:21:53 0:28:07 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 11:00:20.105852 mon.b mon.0 172.21.15.47:6789/0 301 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382972 2018-04-10 20:57:37 2018-04-12 10:28:44 2018-04-12 10:56:43 0:27:59 0:10:24 0:17:35 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 10:52:52.778781 mon.b mon.0 172.21.15.6:6789/0 154 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382973 2018-04-10 20:57:38 2018-04-12 10:28:54 2018-04-12 12:08:55 1:40:01 0:40:39 0:59:22 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
pass 2382974 2018-04-10 20:57:39 2018-04-12 10:30:48 2018-04-12 12:08:50 1:38:02 0:53:33 0:44:29 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
fail 2382975 2018-04-10 20:57:39 2018-04-12 10:30:50 2018-04-12 11:30:50 1:00:00 0:43:17 0:16:43 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 11:15:25.865052 mon.a mon.0 172.21.15.59:6789/0 328 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2382976 2018-04-10 20:57:40 2018-04-12 10:32:48 2018-04-12 10:58:48 0:26:00 0:08:42 0:17:18 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2382977 2018-04-10 20:57:41 2018-04-12 10:34:47 2018-04-12 11:56:48 1:22:01 0:29:31 0:52:30 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 11:49:13.838913 mon.a mon.0 172.21.15.129:6789/0 218 : cluster [WRN] Health check failed: 12 slow ops, oldest one blocked for 985 sec (SLOW_OPS)" in cluster log

pass 2382978 2018-04-10 20:57:42 2018-04-12 10:34:48 2018-04-12 10:56:47 0:21:59 0:09:11 0:12:48 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 2382979 2018-04-10 20:57:43 2018-04-12 10:34:52 2018-04-12 11:00:52 0:26:00 0:09:58 0:16:02 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2382980 2018-04-10 20:57:43 2018-04-12 10:34:52 2018-04-12 11:24:53 0:50:01 0:10:58 0:39:03 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:20:15.571962 mon.b mon.0 172.21.15.63:6789/0 274 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2382981 2018-04-10 20:57:44 2018-04-12 10:34:53 2018-04-12 12:08:55 1:34:02 0:48:16 0:45:46 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
pass 2382982 2018-04-10 20:57:45 2018-04-12 10:36:51 2018-04-12 11:06:51 0:30:00 0:20:05 0:09:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2382983 2018-04-10 20:57:46 2018-04-12 10:36:51 2018-04-12 12:14:53 1:38:02 1:08:32 0:29:30 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 11:44:06.202632 mon.a mon.0 172.21.15.23:6789/0 223 : cluster [WRN] Health check failed: 21 slow ops, oldest one blocked for 876 sec (SLOW_OPS)" in cluster log

pass 2382984 2018-04-10 20:57:47 2018-04-12 10:36:51 2018-04-12 11:30:52 0:54:01 0:29:33 0:24:28 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
fail 2382985 2018-04-10 20:57:47 2018-04-12 10:36:54 2018-04-12 11:50:55 1:14:01 0:16:13 0:57:48 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 11:41:40.969483 mon.b mon.0 172.21.15.63:6789/0 169 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2382986 2018-04-10 20:57:48 2018-04-12 10:36:54 2018-04-12 12:48:57 2:12:03 0:25:16 1:46:47 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 12:30:20.679887 mon.b mon.0 172.21.15.55:6789/0 149 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382987 2018-04-10 20:57:49 2018-04-12 10:38:41 2018-04-12 11:04:41 0:26:00 0:10:15 0:15:45 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:01:32.007165 mon.a mon.0 172.21.15.67:6789/0 210 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 60 sec (SLOW_OPS)" in cluster log

pass 2382988 2018-04-10 20:57:50 2018-04-12 10:38:44 2018-04-12 13:08:47 2:30:03 0:44:03 1:46:00 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
fail 2382989 2018-04-10 20:57:50 2018-04-12 10:38:53 2018-04-12 11:40:53 1:02:00 0:40:38 0:21:22 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 11:17:54.788994 mon.a mon.0 172.21.15.101:6789/0 194 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2382990 2018-04-10 20:57:51 2018-04-12 10:41:06 2018-04-12 11:47:06 1:06:00 0:20:17 0:45:43 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2382991 2018-04-10 20:57:52 2018-04-12 10:41:06 2018-04-12 11:41:06 1:00:00 0:10:08 0:49:52 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:38:13.209962 mon.b mon.0 172.21.15.71:6789/0 183 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 68 sec (SLOW_OPS)" in cluster log

fail 2382992 2018-04-10 20:57:53 2018-04-12 10:41:06 2018-04-12 11:59:07 1:18:01 0:17:42 1:00:19 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 11:46:03.688123 mon.b mon.0 172.21.15.61:6789/0 136 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2382993 2018-04-10 20:57:54 2018-04-12 10:42:50 2018-04-12 11:24:50 0:42:00 0:10:59 0:31:01 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2382994 2018-04-10 20:57:54 2018-04-12 10:43:02 2018-04-12 11:07:01 0:23:59 0:09:44 0:14:15 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 11:02:27.331438 mon.b mon.0 172.21.15.110:6789/0 203 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2382995 2018-04-10 20:57:55 2018-04-12 10:43:13 2018-04-12 11:27:13 0:44:00 0:10:28 0:33:32 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:22:08.625453 mon.b mon.0 172.21.15.110:6789/0 280 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2382996 2018-04-10 20:57:56 2018-04-12 10:44:40 2018-04-12 11:52:40 1:08:00 0:52:00 0:16:00 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 11:21:21.586495 mon.a mon.0 172.21.15.112:6789/0 206 : cluster [WRN] Health check failed: 13 slow ops, oldest one blocked for 967 sec (SLOW_OPS)" in cluster log

pass 2382997 2018-04-10 20:57:57 2018-04-12 10:44:50 2018-04-12 11:18:49 0:33:59 0:21:53 0:12:06 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
pass 2382998 2018-04-10 20:57:58 2018-04-12 10:44:50 2018-04-12 12:04:51 1:20:01 0:53:38 0:26:23 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
fail 2382999 2018-04-10 20:57:59 2018-04-12 10:44:58 2018-04-12 11:58:58 1:14:00 1:03:24 0:10:36 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 11:01:59.509687 mon.b mon.0 172.21.15.94:6789/0 182 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2383000 2018-04-10 20:57:59 2018-04-12 10:46:37 2018-04-12 11:16:37 0:30:00 0:11:03 0:18:57 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 11:13:09.513124 mon.a mon.0 172.21.15.88:6789/0 254 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 84 sec (SLOW_OPS)" in cluster log

fail 2383001 2018-04-10 20:58:00 2018-04-12 10:46:53 2018-04-12 11:24:53 0:38:00 0:19:06 0:18:54 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 11:19:04.401576 mon.a mon.0 172.21.15.120:6789/0 351 : cluster [WRN] Health check failed: 18 slow ops, oldest one blocked for 437 sec (SLOW_OPS)" in cluster log

fail 2383002 2018-04-10 20:58:01 2018-04-12 10:47:28 2018-04-12 11:47:29 1:00:01 0:10:41 0:49:20 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:42:11.595388 mon.b mon.0 172.21.15.99:6789/0 225 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2383003 2018-04-10 20:58:02 2018-04-12 10:49:08 2018-04-12 13:09:10 2:20:02 0:36:33 1:43:29 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 12:46:24.347699 mon.b mon.0 172.21.15.94:6789/0 251 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2383004 2018-04-10 20:58:02 2018-04-12 10:51:02 2018-04-12 11:55:03 1:04:01 0:48:27 0:15:34 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
fail 2383005 2018-04-10 20:58:03 2018-04-12 10:51:02 2018-04-12 12:09:03 1:18:01 0:47:00 0:31:01 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 11:27:04.219865 mon.b mon.0 172.21.15.64:6789/0 173 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2383006 2018-04-10 20:58:04 2018-04-12 10:51:10 2018-04-12 11:19:10 0:28:00 0:09:14 0:18:46 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2383007 2018-04-10 20:58:05 2018-04-12 10:52:52 2018-04-12 11:36:52 0:44:00 0:29:58 0:14:02 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 11:30:25.620351 mon.b mon.0 172.21.15.6:6789/0 193 : cluster [WRN] Health check failed: 9 slow ops, oldest one blocked for 995 sec (SLOW_OPS)" in cluster log

fail 2383008 2018-04-10 20:58:06 2018-04-12 10:56:53 2018-04-12 11:56:53 1:00:00 0:10:05 0:49:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:52:44.656736 mon.b mon.0 172.21.15.6:6789/0 139 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2383009 2018-04-10 20:58:06 2018-04-12 10:56:53 2018-04-12 11:26:53 0:30:00 0:09:29 0:20:31 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 11:22:28.935532 mon.b mon.0 172.21.15.66:6789/0 452 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383010 2018-04-10 20:58:07 2018-04-12 10:56:53 2018-04-12 11:22:53 0:26:00 0:09:54 0:16:06 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/none.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:19:39.177474 mon.a mon.0 172.21.15.69:6789/0 223 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 108 sec (SLOW_OPS)" in cluster log

fail 2383011 2018-04-10 20:58:08 2018-04-12 10:56:53 2018-04-12 12:40:55 1:44:02 0:49:10 0:54:52 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 12:15:13.752640 mon.a mon.0 172.21.15.37:6789/0 358 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 245 sec (SLOW_OPS)" in cluster log

pass 2383012 2018-04-10 20:58:09 2018-04-12 10:56:53 2018-04-12 11:46:53 0:50:00 0:21:19 0:28:41 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2383013 2018-04-10 20:58:09 2018-04-12 10:56:53 2018-04-12 12:28:55 1:32:02 1:13:59 0:18:03 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 11:37:31.599296 mon.a mon.0 172.21.15.67:6789/0 205 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 1001 sec (SLOW_OPS)" in cluster log

fail 2383014 2018-04-10 20:58:10 2018-04-12 10:58:44 2018-04-12 11:38:44 0:40:00 0:26:58 0:13:02 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 11:19:49.281859 mon.a mon.0 172.21.15.178:6789/0 163 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 98 sec (SLOW_OPS)" in cluster log

fail 2383015 2018-04-10 20:58:11 2018-04-12 10:58:49 2018-04-12 11:28:49 0:30:00 0:16:34 0:13:26 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 11:22:41.454888 mon.b mon.0 172.21.15.169:6789/0 188 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383016 2018-04-10 20:58:12 2018-04-12 11:00:49 2018-04-12 11:48:49 0:48:00 0:22:30 0:25:30 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 11:31:52.977481 mon.a mon.0 172.21.15.26:6789/0 251 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 67 sec (SLOW_OPS)" in cluster log

fail 2383017 2018-04-10 20:58:12 2018-04-12 11:00:53 2018-04-12 11:28:53 0:28:00 0:10:15 0:17:45 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:24:05.553692 mon.b mon.0 172.21.15.44:6789/0 200 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2383018 2018-04-10 20:58:13 2018-04-12 11:03:01 2018-04-12 12:13:02 1:10:01 0:40:39 0:29:22 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 12:03:50.381871 mon.a mon.0 172.21.15.4:6789/0 161 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 181 sec (SLOW_OPS)" in cluster log

fail 2383019 2018-04-10 20:58:14 2018-04-12 11:03:01 2018-04-12 12:13:02 1:10:01 0:44:12 0:25:49 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 11:47:21.624919 mon.a mon.0 172.21.15.82:6789/0 218 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 739 sec (SLOW_OPS)" in cluster log

pass 2383020 2018-04-10 20:58:15 2018-04-12 11:03:04 2018-04-12 12:09:04 1:06:00 0:19:31 0:46:29 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2383021 2018-04-10 20:58:15 2018-04-12 11:04:51 2018-04-12 11:58:51 0:54:00 0:10:01 0:43:59 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:56:06.040170 mon.a mon.0 172.21.15.101:6789/0 211 : cluster [WRN] Health check failed: 9 slow ops, oldest one blocked for 75 sec (SLOW_OPS)" in cluster log

pass 2383022 2018-04-10 20:58:16 2018-04-12 11:04:51 2018-04-12 11:46:51 0:42:00 0:18:13 0:23:47 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
fail 2383023 2018-04-10 20:58:17 2018-04-12 11:04:55 2018-04-12 11:34:55 0:30:00 0:10:02 0:19:58 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:30:06.721115 mon.a mon.0 172.21.15.24:6789/0 330 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383024 2018-04-10 20:58:18 2018-04-12 11:04:56 2018-04-12 13:00:57 1:56:01 0:09:24 1:46:37 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 12:56:51.316014 mon.a mon.0 172.21.15.37:6789/0 250 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 43 sec (SLOW_OPS)" in cluster log

fail 2383025 2018-04-10 20:58:19 2018-04-12 11:05:01 2018-04-12 11:35:01 0:30:00 0:13:04 0:16:56 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:32:18.286094 mon.a mon.0 172.21.15.31:6789/0 353 : cluster [WRN] Health check failed: 15 slow ops, oldest one blocked for 250 sec (SLOW_OPS)" in cluster log

fail 2383026 2018-04-10 20:58:19 2018-04-12 11:05:03 2018-04-12 11:45:03 0:40:00 0:11:17 0:28:43 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 11:42:06.014229 mon.b mon.0 172.21.15.28:6789/0 182 : cluster [WRN] Health check failed: 14 slow ops, oldest one blocked for 178 sec (SLOW_OPS)" in cluster log

fail 2383027 2018-04-10 20:58:20 2018-04-12 11:07:03 2018-04-12 12:51:04 1:44:01 0:22:43 1:21:18 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 12:39:05.838060 mon.b mon.0 172.21.15.61:6789/0 812 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2383028 2018-04-10 20:58:21 2018-04-12 11:07:02 2018-04-12 12:09:03 1:02:01 0:44:24 0:17:37 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 11:30:10.726747 mon.b mon.0 172.21.15.167:6789/0 147 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2383029 2018-04-10 20:58:22 2018-04-12 11:07:03 2018-04-12 12:55:04 1:48:01 1:00:34 0:47:27 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 12:01:31.503344 mon.b mon.0 172.21.15.110:6789/0 179 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2383030 2018-04-10 20:58:22 2018-04-12 11:09:10 2018-04-12 11:51:10 0:42:00 0:11:13 0:30:47 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 11:46:35.033625 mon.b mon.0 172.21.15.19:6789/0 144 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383031 2018-04-10 20:58:23 2018-04-12 11:10:56 2018-04-12 12:26:57 1:16:01 0:19:43 0:56:18 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 12:20:47.225262 mon.b mon.0 172.21.15.114:6789/0 192 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2383032 2018-04-10 20:58:24 2018-04-12 11:10:56 2018-04-12 12:06:57 0:56:01 0:09:57 0:46:04 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 12:03:51.504289 mon.a mon.0 172.21.15.118:6789/0 178 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 103 sec (SLOW_OPS)" in cluster log

fail 2383033 2018-04-10 20:58:25 2018-04-12 11:10:56 2018-04-12 12:24:57 1:14:01 0:46:41 0:27:20 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 12:06:00.239009 mon.b mon.0 172.21.15.8:6789/0 247 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2383034 2018-04-10 20:58:26 2018-04-12 11:13:03 2018-04-12 12:29:04 1:16:01 0:50:04 0:25:57 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 11:43:50.116921 mon.b mon.0 172.21.15.157:6789/0 230 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2383035 2018-04-10 20:58:26 2018-04-12 11:13:19 2018-04-12 13:47:22 2:34:03 0:50:07 1:43:56 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
fail 2383036 2018-04-10 20:58:27 2018-04-12 11:15:03 2018-04-12 11:35:02 0:19:59 0:09:04 0:10:55 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:31:06.879826 mon.b mon.0 172.21.15.47:6789/0 130 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383037 2018-04-10 20:58:28 2018-04-12 11:15:03 2018-04-12 11:57:03 0:42:00 0:21:53 0:20:07 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 11:53:02.339469 mon.b mon.0 172.21.15.17:6789/0 185 : cluster [WRN] Health check failed: 15 slow ops, oldest one blocked for 696 sec (SLOW_OPS)" in cluster log

fail 2383038 2018-04-10 20:58:29 2018-04-12 11:15:20 2018-04-12 12:51:21 1:36:01 0:09:46 1:26:15 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 12:47:15.385862 mon.a mon.0 172.21.15.97:6789/0 157 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2383039 2018-04-10 20:58:29 2018-04-12 11:16:39 2018-04-12 12:28:40 1:12:01 0:09:52 1:02:09 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
Failure Reason:

"2018-04-12 12:25:25.003806 mon.b mon.0 172.21.15.40:6789/0 457 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2383040 2018-04-10 20:58:30 2018-04-12 11:16:44 2018-04-12 11:50:44 0:34:00 0:13:13 0:20:47 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:45:23.436668 mon.a mon.0 172.21.15.104:6789/0 275 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 110 sec (SLOW_OPS)" in cluster log

fail 2383041 2018-04-10 20:58:31 2018-04-12 11:16:47 2018-04-12 12:58:49 1:42:02 0:10:36 1:31:26 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 12:56:17.985282 mon.a mon.0 172.21.15.59:6789/0 207 : cluster [WRN] Health check failed: 13 slow ops, oldest one blocked for 148 sec (SLOW_OPS)" in cluster log

fail 2383042 2018-04-10 20:58:32 2018-04-12 11:16:57 2018-04-12 12:38:58 1:22:01 0:20:44 1:01:17 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 12:27:42.045688 mon.b mon.0 172.21.15.64:6789/0 602 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2383043 2018-04-10 20:58:33 2018-04-12 11:16:57 2018-04-12 13:18:59 2:02:02 1:13:36 0:48:26 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 12:57:07.640694 mon.b mon.0 172.21.15.38:6789/0 189 : cluster [WRN] Health check failed: 12 slow ops, oldest one blocked for 2668 sec (SLOW_OPS)" in cluster log

fail 2383044 2018-04-10 20:58:33 2018-04-12 11:18:59 2018-04-12 11:58:59 0:40:00 0:23:50 0:16:10 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 11:41:14.672566 mon.a mon.0 172.21.15.73:6789/0 159 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 116 sec (SLOW_OPS)" in cluster log

pass 2383045 2018-04-10 20:58:34 2018-04-12 11:18:59 2018-04-12 12:21:00 1:02:01 0:16:38 0:45:23 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
fail 2383046 2018-04-10 20:58:35 2018-04-12 11:19:11 2018-04-12 12:15:12 0:56:01 0:24:01 0:32:00 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 11:56:57.924162 mon.a mon.0 172.21.15.69:6789/0 248 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2383047 2018-04-10 20:58:36 2018-04-12 11:21:03 2018-04-12 12:13:03 0:52:00 0:12:00 0:40:00 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 12:08:53.215138 mon.b mon.0 172.21.15.44:6789/0 233 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2383048 2018-04-10 20:58:36 2018-04-12 11:22:52 2018-04-12 12:48:53 1:26:01 0:48:55 0:37:06 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 12:25:25.309792 mon.b mon.0 172.21.15.70:6789/0 160 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2383049 2018-04-10 20:58:37 2018-04-12 11:22:54 2018-04-12 13:08:56 1:46:02 0:43:01 1:03:01 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 12:51:43.115392 mon.a mon.0 172.21.15.129:6789/0 243 : cluster [WRN] Health check failed: 19 slow ops, oldest one blocked for 1220 sec (SLOW_OPS)" in cluster log

fail 2383050 2018-04-10 20:58:38 2018-04-12 11:23:18 2018-04-12 11:55:18 0:32:00 0:20:02 0:11:58 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 11:46:43.779049 mon.a mon.0 172.21.15.119:6789/0 252 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 426 sec (SLOW_OPS)" in cluster log

pass 2383051 2018-04-10 20:58:38 2018-04-12 11:24:59 2018-04-12 13:01:00 1:36:01 0:09:09 1:26:52 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
fail 2383052 2018-04-10 20:58:39 2018-04-12 11:24:59 2018-04-12 12:47:00 1:22:01 0:24:32 0:57:29 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 12:29:34.430023 mon.a mon.0 172.21.15.23:6789/0 259 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2383053 2018-04-10 20:58:40 2018-04-12 11:24:59 2018-04-12 12:25:00 1:00:01 0:10:53 0:49:08 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 12:19:29.234689 mon.b mon.0 172.21.15.49:6789/0 170 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2383054 2018-04-10 20:58:41 2018-04-12 11:24:59 2018-04-12 12:53:00 1:28:01 0:09:18 1:18:43 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2383055 2018-04-10 20:58:41 2018-04-12 11:24:59 2018-04-12 12:00:59 0:36:00 0:12:41 0:23:19 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:54:28.846201 mon.a mon.0 172.21.15.109:6789/0 207 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 59 sec (SLOW_OPS)" in cluster log

fail 2383056 2018-04-10 20:58:42 2018-04-12 11:26:52 2018-04-12 12:06:52 0:40:00 0:10:23 0:29:37 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 12:02:26.130465 mon.b mon.0 172.21.15.99:6789/0 171 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 81 sec (SLOW_OPS)" in cluster log

pass 2383057 2018-04-10 20:58:43 2018-04-12 11:26:52 2018-04-12 12:04:52 0:38:00 0:22:17 0:15:43 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2383058 2018-04-10 20:58:44 2018-04-12 11:26:53 2018-04-12 12:30:53 1:04:00 0:48:17 0:15:43 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 12:22:19.378314 mon.a mon.0 172.21.15.74:6789/0 166 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 2027 sec (SLOW_OPS)" in cluster log

fail 2383059 2018-04-10 20:58:45 2018-04-12 11:26:54 2018-04-12 12:58:55 1:32:01 1:11:28 0:20:33 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 11:55:45.761597 mon.a mon.0 172.21.15.165:6789/0 346 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 173 sec (SLOW_OPS)" in cluster log

fail 2383060 2018-04-10 20:58:45 2018-04-12 11:27:14 2018-04-12 11:53:13 0:25:59 0:10:11 0:15:48 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 11:49:11.318269 mon.a mon.0 172.21.15.24:6789/0 253 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 78 sec (SLOW_OPS)" in cluster log

fail 2383061 2018-04-10 20:58:46 2018-04-12 11:28:50 2018-04-12 12:00:50 0:32:00 0:20:15 0:11:45 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 11:53:25.419066 mon.a mon.0 172.21.15.136:6789/0 354 : cluster [WRN] Health check failed: 14 slow ops, oldest one blocked for 484 sec (SLOW_OPS)" in cluster log

fail 2383062 2018-04-10 20:58:47 2018-04-12 11:28:54 2018-04-12 11:54:54 0:26:00 0:10:22 0:15:38 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 11:52:25.199083 mon.b mon.0 172.21.15.174:6789/0 260 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 85 sec (SLOW_OPS)" in cluster log

fail 2383063 2018-04-10 20:58:48 2018-04-12 11:30:50 2018-04-12 13:12:52 1:42:02 0:47:43 0:54:19 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 12:54:34.062726 mon.a mon.0 172.21.15.15:6789/0 166 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 732 sec (SLOW_OPS)" in cluster log

fail 2383064 2018-04-10 20:58:48 2018-04-12 11:30:52 2018-04-12 12:40:52 1:10:00 0:46:36 0:23:24 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 12:00:01.366445 mon.a mon.0 172.21.15.35:6789/0 250 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383065 2018-04-10 20:58:49 2018-04-11 16:09:06 2018-04-11 16:59:06 0:50:00 0:38:45 0:11:15 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-11 16:27:04.668215 mon.a mon.0 172.21.15.94:6789/0 325 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

fail 2383066 2018-04-10 20:58:50 2018-04-12 11:30:54 2018-04-12 11:56:53 0:25:59 0:09:27 0:16:32 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 11:53:03.853472 mon.b mon.0 172.21.15.31:6789/0 141 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2383067 2018-04-10 20:58:51 2018-04-12 11:34:48 2018-04-12 12:22:49 0:48:01 0:24:14 0:23:47 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 12:17:08.876562 mon.a mon.0 172.21.15.26:6789/0 342 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 721 sec (SLOW_OPS)" in cluster log

pass 2383068 2018-04-10 20:58:51 2018-04-12 11:34:50 2018-04-12 12:04:50 0:30:00 0:09:44 0:20:16 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
pass 2383069 2018-04-10 20:58:52 2018-04-12 11:34:56 2018-04-12 12:18:56 0:44:00 0:10:11 0:33:49 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2383070 2018-04-10 20:58:53 2018-04-12 11:35:02 2018-04-12 12:13:02 0:38:00 0:13:26 0:24:34 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 12:06:56.768116 mon.b mon.0 172.21.15.19:6789/0 268 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 43 sec (SLOW_OPS)" in cluster log

fail 2383071 2018-04-10 20:58:54 2018-04-12 11:35:03 2018-04-12 12:29:04 0:54:01 0:10:36 0:43:25 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 12:25:28.643031 mon.b mon.0 172.21.15.71:6789/0 301 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 47 sec (SLOW_OPS)" in cluster log

pass 2383072 2018-04-10 20:58:55 2018-04-12 11:37:04 2018-04-12 12:41:05 1:04:01 0:20:24 0:43:37 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
fail 2383073 2018-04-10 20:58:55 2018-04-12 11:37:04 2018-04-12 13:39:06 2:02:02 1:15:52 0:46:10 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 12:30:22.423853 mon.b mon.0 172.21.15.19:6789/0 184 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2383074 2018-04-10 20:58:56 2018-04-12 11:38:54 2018-04-12 13:24:56 1:46:02 0:54:26 0:51:36 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
fail 2383075 2018-04-10 20:58:57 2018-04-12 11:39:23 2018-04-12 12:19:22 0:39:59 0:16:31 0:23:28 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
Failure Reason:

"2018-04-12 12:13:28.592540 mon.a mon.0 172.21.15.24:6789/0 351 : cluster [WRN] Health check failed: 12 slow ops, oldest one blocked for 283 sec (SLOW_OPS)" in cluster log

fail 2383076 2018-04-10 20:58:58 2018-04-12 11:41:00 2018-04-12 12:35:00 0:54:00 0:21:47 0:32:13 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 12:27:01.103793 mon.b mon.0 172.21.15.173:6789/0 142 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2383077 2018-04-10 20:58:58 2018-04-12 11:41:00 2018-04-12 12:29:01 0:48:01 0:10:15 0:37:46 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 12:24:32.123652 mon.b mon.0 172.21.15.13:6789/0 212 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2383078 2018-04-10 20:58:59 2018-04-12 11:41:07 2018-04-12 12:39:08 0:58:01 0:45:30 0:12:31 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 12:34:28.270946 mon.a mon.0 172.21.15.56:6789/0 174 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383079 2018-04-10 20:59:00 2018-04-12 11:42:51 2018-04-12 12:36:52 0:54:01 0:39:12 0:14:49 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
Failure Reason:

"2018-04-12 12:18:58.172619 mon.a mon.0 172.21.15.5:6789/0 221 : cluster [WRN] Health check failed: 19 slow ops, oldest one blocked for 971 sec (SLOW_OPS)" in cluster log

fail 2383080 2018-04-10 20:59:01 2018-04-12 11:42:56 2018-04-12 13:14:58 1:32:02 0:59:42 0:32:20 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 12:31:33.835074 mon.a mon.0 172.21.15.99:6789/0 155 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 645 sec (SLOW_OPS)" in cluster log

fail 2383081 2018-04-10 20:59:01 2018-04-12 11:45:15 2018-04-12 12:19:15 0:34:00 0:10:12 0:23:48 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 12:15:10.536781 mon.b mon.0 172.21.15.94:6789/0 168 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383082 2018-04-10 20:59:02 2018-04-12 11:45:29 2018-04-12 12:33:29 0:48:00 0:23:46 0:24:14 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 12:15:45.106116 mon.b mon.0 172.21.15.109:6789/0 149 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

fail 2383083 2018-04-10 20:59:03 2018-04-12 11:46:52 2018-04-12 12:16:52 0:30:00 0:10:59 0:19:01 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 12:13:53.501710 mon.a mon.0 172.21.15.129:6789/0 331 : cluster [WRN] Health check failed: 10 slow ops, oldest one blocked for 152 sec (SLOW_OPS)" in cluster log

pass 2383084 2018-04-10 20:59:04 2018-04-12 11:46:55 2018-04-12 12:22:54 0:35:59 0:09:03 0:26:56 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_exports.yaml} 3
fail 2383085 2018-04-10 20:59:05 2018-04-12 11:46:57 2018-04-12 12:12:56 0:25:59 0:12:52 0:13:07 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/9-mds-3-standby.yaml mount/kclient.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 12:05:07.901737 mon.a mon.0 172.21.15.15:6789/0 259 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

fail 2383086 2018-04-10 20:59:05 2018-04-12 11:46:57 2018-04-12 12:14:56 0:27:59 0:10:38 0:17:21 smithi master multimds/verify/{begin.yaml clusters/9-mds.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_fsstress.yaml validater/lockdep.yaml} 3
Failure Reason:

"2018-04-12 12:11:50.387542 mon.a mon.0 172.21.15.119:6789/0 318 : cluster [WRN] Health check failed: 12 slow ops, oldest one blocked for 92 sec (SLOW_OPS)" in cluster log

fail 2383087 2018-04-10 20:59:06 2018-04-12 11:47:07 2018-04-12 12:27:07 0:40:00 0:22:27 0:17:33 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cephfs_test_snapshots.yaml} 3
Failure Reason:

"2018-04-12 12:22:37.716695 mon.a mon.0 172.21.15.87:6789/0 1422 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

fail 2383088 2018-04-10 20:59:07 2018-04-12 11:47:30 2018-04-12 12:39:30 0:52:00 0:38:17 0:13:43 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-04-12 12:27:31.899727 mon.b mon.0 172.21.15.63:6789/0 236 : cluster [WRN] Health check failed: 9 slow ops, oldest one blocked for 1253 sec (SLOW_OPS)" in cluster log

fail 2383089 2018-04-10 20:59:08 2018-04-12 11:48:47 2018-04-12 13:30:48 1:42:01 1:00:33 0:41:28 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_misc.yaml} 3
Failure Reason:

"2018-04-12 12:35:56.781464 mon.b mon.0 172.21.15.50:6789/0 179 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2383090 2018-04-10 20:59:08 2018-04-12 11:48:50 2018-04-12 12:12:50 0:24:00 0:10:52 0:13:08 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_norstats.yaml} 3
fail 2383091 2018-04-10 20:59:09 2018-04-12 11:50:54 2018-04-12 13:18:55 1:28:01 0:19:18 1:08:43 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_blogbench.yaml} 3
Failure Reason:

"2018-04-12 13:06:08.995815 mon.b mon.0 172.21.15.61:6789/0 307 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2383092 2018-04-10 20:59:10 2018-04-12 11:50:56 2018-04-12 12:30:56 0:40:00 0:10:39 0:29:21 smithi master multimds/thrash/{begin.yaml ceph-thrash/default.yaml clusters/3-mds-2-standby.yaml mount/fuse.yaml msgr-failures/none.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml thrash/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} thrash_debug.yaml} tasks/cfuse_workunit_suites_pjd.yaml} 3
fail 2383093 2018-04-10 20:59:10 2018-04-12 11:51:11 2018-04-12 13:05:12 1:14:01 0:44:25 0:29:36 smithi master centos multimds/verify/{begin.yaml clusters/3-mds.yaml mount/fuse.yaml objectstore-ec/filestore-xfs.yaml overrides/{fuse-default-perm-no.yaml verify/{debug.yaml frag_enable.yaml mon-debug.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml}} tasks/cfuse_workunit_suites_dbench.yaml validater/valgrind.yaml} 3
Failure Reason:

"2018-04-12 12:47:56.571735 mon.a mon.0 172.21.15.1:6789/0 155 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

pass 2383094 2018-04-10 20:59:11 2018-04-12 11:52:50 2018-04-12 12:56:56 1:04:06 0:51:47 0:12:19 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_dbench.yaml} 3
fail 2383095 2018-04-10 20:59:12 2018-04-12 11:53:15 2018-04-12 13:39:16 1:46:01 0:39:30 1:06:31 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore-comp.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-04-12 13:06:03.041997 mon.a mon.0 172.21.15.73:6789/0 205 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383096 2018-04-10 20:59:13 2018-04-12 11:54:55 2018-04-12 13:06:56 1:12:01 0:09:51 1:02:10 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsstress.yaml} 3
Failure Reason:

"2018-04-12 13:04:20.891691 mon.a mon.0 172.21.15.2:6789/0 258 : cluster [WRN] Health check failed: 3 slow ops, oldest one blocked for 63 sec (SLOW_OPS)" in cluster log

fail 2383097 2018-04-10 20:59:13 2018-04-12 11:54:58 2018-04-12 13:06:58 1:12:00 0:23:31 0:48:29 smithi master multimds/basic/{begin.yaml clusters/9-mds.yaml inline/yes.yaml mount/fuse.yaml objectstore-ec/bluestore.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_fsx.yaml} 3
Failure Reason:

"2018-04-12 12:50:28.491483 mon.b mon.0 172.21.15.173:6789/0 166 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

fail 2383098 2018-04-10 20:59:14 2018-04-12 11:55:04 2018-04-12 13:01:04 1:06:00 0:09:19 0:56:41 smithi master multimds/basic/{begin.yaml clusters/3-mds.yaml inline/no.yaml mount/kclient.yaml objectstore-ec/filestore-xfs.yaml overrides/{basic/{debug.yaml frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} fuse-default-perm-no.yaml} q_check_counter/check_counter.yaml tasks/cfuse_workunit_suites_pjd.yaml} 3
Failure Reason:

"2018-04-12 12:57:22.078061 mon.b mon.0 172.21.15.31:6789/0 141 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log