Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 2097919 2018-01-22 03:25:30 2018-01-25 05:45:23 2018-01-25 06:15:23 0:30:00 0:09:42 0:20:18 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2097923 2018-01-22 03:25:30 2018-01-25 05:45:23 2018-01-25 07:33:25 1:48:02 0:50:22 0:57:40 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
pass 2097926 2018-01-22 03:25:31 2018-01-25 05:57:33 2018-01-25 06:21:33 0:24:00 0:10:38 0:13:22 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 2097929 2018-01-22 03:25:32 2018-01-25 05:57:33 2018-01-25 07:09:35 1:12:02 0:54:37 0:17:25 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
pass 2097932 2018-01-22 03:25:32 2018-01-25 05:59:25 2018-01-25 06:53:26 0:54:01 0:23:22 0:30:39 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 2097933 2018-01-22 03:25:33 2018-01-25 06:01:02 2018-01-25 07:17:02 1:16:00 0:16:06 0:59:54 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
pass 2097936 2018-01-22 03:25:34 2018-01-24 21:56:06 2018-01-24 22:42:06 0:46:00 0:30:03 0:15:57 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2097939 2018-01-22 03:25:34 2018-01-25 06:47:24 2018-01-25 07:23:24 0:36:00 0:24:39 0:11:21 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2097942 2018-01-22 03:25:35 2018-01-25 06:47:32 2018-01-25 11:25:37 4:38:05 0:16:05 4:22:00 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
pass 2097945 2018-01-22 03:25:35 2018-01-25 06:49:16 2018-01-25 08:59:18 2:10:02 0:38:19 1:31:43 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 2097948 2018-01-22 03:25:36 2018-01-25 06:49:25 2018-01-25 08:11:26 1:22:01 0:15:57 1:06:04 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 2097951 2018-01-22 03:25:37 2018-01-25 06:51:18 2018-01-25 09:39:21 2:48:03 0:36:06 2:11:57 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
pass 2097954 2018-01-22 03:25:37 2018-01-25 06:53:27 2018-01-25 09:01:29 2:08:02 0:49:02 1:19:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
fail 2097957 2018-01-22 03:25:38 2018-01-25 06:55:16 2018-01-25 08:07:17 1:12:01 0:25:19 0:46:42 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 07:51:36.297244 mon.a mon.0 172.21.15.165:6789/0 108 : cluster [WRN] Health check failed: 18878 slow ops, oldest one blocked for 68 sec (SLOW_OPS)" in cluster log

pass 2097960 2018-01-22 03:25:39 2018-01-25 06:57:26 2018-01-25 08:53:28 1:56:02 0:10:22 1:45:40 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 2097963 2018-01-22 03:25:39 2018-01-25 06:59:15 2018-01-25 07:39:15 0:40:00 0:13:15 0:26:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 2097964 2018-01-22 03:25:40 2018-01-25 06:59:27 2018-01-25 08:15:28 1:16:01 0:26:26 0:49:35 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2097965 2018-01-22 03:25:41 2018-01-25 07:01:33 2018-01-25 10:31:37 3:30:04 0:18:09 3:11:55 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/damage.yaml whitelist_health.yaml} 4
fail 2097966 2018-01-22 03:25:41 2018-01-25 07:05:25 2018-01-25 08:33:26 1:28:01 1:15:16 0:12:45 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 07:26:52.715364 mon.a mon.0 172.21.15.77:6789/0 163 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

pass 2097967 2018-01-22 03:25:42 2018-01-25 07:05:25 2018-01-25 07:37:25 0:32:00 0:17:19 0:14:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 2097968 2018-01-22 03:25:43 2018-01-25 07:05:26 2018-01-25 09:07:28 2:02:02 0:22:27 1:39:35 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 2097969 2018-01-22 03:25:43 2018-01-25 07:05:28 2018-01-25 08:37:30 1:32:02 0:21:02 1:11:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 2097970 2018-01-22 03:25:44 2018-01-25 07:05:30 2018-01-25 08:13:32 1:08:02 0:17:18 0:50:44 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 2097971 2018-01-22 03:25:45 2018-01-25 07:07:26 2018-01-25 10:49:30 3:42:04 0:19:32 3:22:32 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 2097972 2018-01-22 03:25:45 2018-01-25 07:09:26 2018-01-25 07:35:26 0:26:00 0:10:29 0:15:31 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 2097973 2018-01-22 03:25:46 2018-01-25 07:09:26 2018-01-25 07:47:26 0:38:00 0:16:44 0:21:16 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2097974 2018-01-22 03:25:46 2018-01-25 07:09:27 2018-01-25 09:15:29 2:06:02 0:29:35 1:36:27 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-comp.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
pass 2097975 2018-01-22 03:25:47 2018-01-25 07:09:36 2018-01-25 07:43:36 0:34:00 0:17:27 0:16:33 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
pass 2097976 2018-01-22 03:25:48 2018-01-25 07:11:24 2018-01-25 07:39:24 0:28:00 0:18:22 0:09:38 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 2097977 2018-01-22 03:25:48 2018-01-25 07:13:33 2018-01-25 07:57:33 0:44:00 0:26:32 0:17:28 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 2097978 2018-01-22 03:25:49 2018-01-25 07:17:16 2018-01-25 08:15:16 0:58:00 0:21:30 0:36:30 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
pass 2097979 2018-01-22 03:25:50 2018-01-25 07:17:16 2018-01-25 08:03:16 0:46:00 0:35:58 0:10:02 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2097980 2018-01-22 03:25:50 2018-01-23 18:28:44 2018-01-23 19:00:44 0:32:00 0:20:09 0:11:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2097981 2018-01-22 03:25:51 2018-01-25 07:17:29 2018-01-25 08:49:30 1:32:01 0:10:04 1:21:57 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
fail 2097982 2018-01-22 03:25:52 2018-01-25 07:17:29 2018-01-25 09:03:30 1:46:01 0:51:40 0:54:21 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2018-01-25 08:38:24.867720 mon.b mon.0 172.21.15.1:6789/0 354 : cluster [WRN] Health check failed: 5547 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2097983 2018-01-22 03:25:52 2018-01-25 07:17:32 2018-01-25 08:09:32 0:52:00 0:34:47 0:17:13 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
dead 2097984 2018-01-22 03:25:53 2018-01-25 07:19:14 2018-01-25 19:21:46 12:02:32 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
pass 2097985 2018-01-22 03:25:53 2018-01-25 07:19:14 2018-01-25 08:11:14 0:52:00 0:41:24 0:10:36 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
pass 2097986 2018-01-22 03:25:54 2018-01-25 07:19:21 2018-01-25 08:53:22 1:34:01 0:24:04 1:09:57 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 2097987 2018-01-22 03:25:55 2018-01-25 07:21:19 2018-01-25 08:35:19 1:14:00 0:10:39 1:03:21 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 2097988 2018-01-22 03:25:55 2018-01-25 07:21:20 2018-01-25 08:37:21 1:16:01 0:14:41 1:01:20 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 2097989 2018-01-22 03:25:56 2018-01-25 07:21:28 2018-01-25 07:57:28 0:36:00 0:22:19 0:13:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2097990 2018-01-22 03:25:57 2018-01-25 07:21:29 2018-01-25 09:59:32 2:38:03 0:12:07 2:25:56 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 2097991 2018-01-22 03:25:57 2018-01-25 07:21:38 2018-01-25 07:59:38 0:38:00 0:22:40 0:15:20 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2018-01-25 07:50:04.114445 mon.a mon.0 172.21.15.85:6789/0 121 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

pass 2097992 2018-01-22 03:25:58 2018-01-25 07:23:31 2018-01-25 08:35:31 1:12:00 0:16:19 0:55:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
fail 2097993 2018-01-22 03:25:59 2018-01-25 07:25:31 2018-01-25 11:07:35 3:42:04 0:12:29 3:29:35 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/strays.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_files_throttle (tasks.cephfs.test_strays.TestStrays)

pass 2097994 2018-01-22 03:25:59 2018-01-25 07:25:38 2018-01-25 08:07:38 0:42:00 0:17:40 0:24:20 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 2097995 2018-01-22 03:26:00 2018-01-25 07:25:38 2018-01-25 08:07:38 0:42:00 0:20:59 0:21:01 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 2097996 2018-01-22 03:26:00 2018-01-25 07:27:31 2018-01-25 11:55:36 4:28:05 0:15:10 4:12:55 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 2097997 2018-01-22 03:26:01 2018-01-25 07:28:47 2018-01-25 08:00:47 0:32:00 0:12:59 0:19:01 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 2097998 2018-01-22 03:26:02 2018-01-25 07:32:58 2018-01-25 08:00:57 0:27:59 0:16:20 0:11:39 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2097999 2018-01-22 03:26:02 2018-01-25 07:33:26 2018-01-25 11:51:31 4:18:05 0:51:15 3:26:50 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
pass 2098000 2018-01-22 03:26:03 2018-01-25 07:35:40 2018-01-25 10:03:43 2:28:03 0:10:29 2:17:34 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 2098001 2018-01-22 03:26:04 2018-01-25 07:37:40 2018-01-25 08:51:41 1:14:01 0:30:10 0:43:51 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
pass 2098002 2018-01-22 03:26:04 2018-01-25 07:37:40 2018-01-25 08:45:40 1:08:00 0:34:12 0:33:48 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 2098003 2018-01-22 03:26:05 2018-01-25 07:39:30 2018-01-25 08:35:31 0:56:01 0:11:45 0:44:16 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
pass 2098004 2018-01-22 03:26:06 2018-01-25 07:39:30 2018-01-25 08:39:31 1:00:01 0:30:26 0:29:35 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2098005 2018-01-22 03:26:06 2018-01-25 07:39:30 2018-01-25 08:17:30 0:38:00 0:19:48 0:18:12 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2098006 2018-01-22 03:26:07 2018-01-25 07:39:30 2018-01-25 09:47:33 2:08:03 0:16:36 1:51:27 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
fail 2098007 2018-01-22 03:26:07 2018-01-25 07:39:36 2018-01-25 08:55:37 1:16:01 1:04:42 0:11:19 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2018-01-25 08:31:33.922924 mon.b mon.0 172.21.15.161:6789/0 316 : cluster [WRN] Health check failed: 1458 slow ops, oldest one blocked for 34 sec (SLOW_OPS)" in cluster log

pass 2098008 2018-01-22 03:26:08 2018-01-25 07:43:36 2018-01-25 08:27:36 0:44:00 0:18:01 0:25:59 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 2098009 2018-01-22 03:26:09 2018-01-25 07:43:36 2018-01-25 11:21:40 3:38:04 0:34:53 3:03:11 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
pass 2098010 2018-01-22 03:26:09 2018-01-25 07:43:37 2018-01-25 08:57:37 1:14:00 0:41:07 0:32:53 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
fail 2098011 2018-01-22 03:26:10 2018-01-25 07:45:28 2018-01-25 08:53:29 1:08:01 0:56:56 0:11:05 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 08:09:42.488467 mon.b mon.0 172.21.15.53:6789/0 142 : cluster [WRN] Health check failed: 6008 slow ops, oldest one blocked for 35 sec (SLOW_OPS)" in cluster log

pass 2098012 2018-01-22 03:26:11 2018-01-25 07:47:30 2018-01-25 09:17:31 1:30:01 0:10:03 1:19:58 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 2098013 2018-01-22 03:26:11 2018-01-25 07:47:30 2018-01-25 08:21:30 0:34:00 0:12:02 0:21:58 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 2098014 2018-01-22 03:26:12 2018-01-25 07:47:30 2018-01-25 08:29:30 0:42:00 0:23:14 0:18:46 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2098015 2018-01-22 03:26:12 2018-01-25 07:51:26 2018-01-25 08:29:26 0:38:00 0:22:02 0:15:58 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/damage.yaml whitelist_health.yaml} 4
fail 2098016 2018-01-22 03:26:13 2018-01-25 07:59:29 2018-01-25 09:41:31 1:42:02 0:20:27 1:21:35 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 09:26:36.698663 mon.b mon.0 172.21.15.6:6789/0 135 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

pass 2098017 2018-01-22 03:26:14 2018-01-25 07:59:39 2018-01-25 09:09:40 1:10:01 0:12:09 0:57:52 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 2098018 2018-01-22 03:26:14 2018-01-25 08:01:01 2018-01-25 10:11:03 2:10:02 0:21:49 1:48:13 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 2098019 2018-01-22 03:26:15 2018-01-25 08:01:01 2018-01-25 09:07:02 1:06:01 0:11:13 0:54:48 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 2098020 2018-01-22 03:26:16 2018-01-25 08:03:29 2018-01-25 08:35:28 0:31:59 0:19:20 0:12:39 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 2098021 2018-01-22 03:26:16 2018-01-25 08:03:29 2018-01-25 12:55:36 4:52:07 0:19:41 4:32:26 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 2098022 2018-01-22 03:26:17 2018-01-25 08:03:29 2018-01-25 08:37:29 0:34:00 0:11:02 0:22:58 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 2098023 2018-01-22 03:26:18 2018-01-25 08:07:31 2018-01-25 08:35:30 0:27:59 0:09:49 0:18:10 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2098024 2018-01-22 03:26:18 2018-01-25 08:07:39 2018-01-25 10:57:42 2:50:03 0:30:25 2:19:38 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
pass 2098025 2018-01-22 03:26:19 2018-01-25 08:07:39 2018-01-25 11:01:42 2:54:03 0:12:01 2:42:02 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
pass 2098026 2018-01-22 03:26:19 2018-01-25 08:09:30 2018-01-25 09:11:30 1:02:00 0:21:40 0:40:20 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
fail 2098027 2018-01-22 03:26:20 2018-01-25 08:09:34 2018-01-25 08:55:34 0:46:00 0:34:10 0:11:50 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-01-25 08:37:37.196798 mon.b mon.0 172.21.15.30:6789/0 151 : cluster [WRN] Health check failed: 49 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2098028 2018-01-22 03:26:21 2018-01-25 08:11:28 2018-01-25 08:55:28 0:44:00 0:18:20 0:25:40 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
pass 2098029 2018-01-22 03:26:21 2018-01-25 08:11:28 2018-01-25 08:53:28 0:42:00 0:28:55 0:13:05 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2098030 2018-01-22 03:26:22 2018-01-25 08:11:31 2018-01-25 09:03:31 0:52:00 0:19:00 0:33:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2098031 2018-01-22 03:26:23 2018-01-25 08:13:46 2018-01-25 09:11:47 0:58:01 0:13:31 0:44:30 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
pass 2098032 2018-01-22 03:26:23 2018-01-25 08:14:10 2018-01-25 09:44:14 1:30:04 0:40:39 0:49:25 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 2098033 2018-01-22 03:26:24 2018-01-25 08:15:31 2018-01-25 09:41:32 1:26:01 0:54:24 0:31:37 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 2098034 2018-01-22 03:26:24 2018-01-25 08:15:31 2018-01-25 11:09:34 2:54:03 0:07:36 2:46:27 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi041 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true'

pass 2098035 2018-01-22 03:26:25 2018-01-25 08:15:31 2018-01-25 09:19:32 1:04:01 0:36:56 0:27:05 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
fail 2098036 2018-01-22 03:26:26 2018-01-25 08:17:38 2018-01-25 09:41:39 1:24:01 1:14:28 0:09:33 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 08:37:22.883427 mon.b mon.0 172.21.15.11:6789/0 107 : cluster [WRN] Health check failed: 17954 slow ops, oldest one blocked for 85 sec (SLOW_OPS)" in cluster log

pass 2098037 2018-01-22 03:26:27 2018-01-25 08:17:38 2018-01-25 10:19:40 2:02:02 0:10:05 1:51:57 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 2098038 2018-01-22 03:26:28 2018-01-25 08:20:34 2018-01-25 08:44:34 0:24:00 0:09:41 0:14:19 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 2098039 2018-01-22 03:26:29 2018-01-25 08:21:43 2018-01-25 11:03:46 2:42:03 0:24:24 2:17:39 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2098040 2018-01-22 03:26:30 2018-01-25 08:25:43 2018-01-25 09:15:43 0:50:00 0:11:59 0:38:01 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 2098041 2018-01-22 03:26:30 2018-01-25 08:27:43 2018-01-25 08:53:42 0:25:59 0:11:21 0:14:38 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2018-01-25 08:48:38.999889 mon.a mon.0 172.21.15.165:6789/0 115 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

pass 2098042 2018-01-22 03:26:31 2018-01-25 08:29:27 2018-01-25 08:53:27 0:24:00 0:09:30 0:14:30 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 2098043 2018-01-22 03:26:32 2018-01-25 08:29:27 2018-01-25 09:47:28 1:18:01 0:27:30 0:50:31 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/strays.yaml whitelist_health.yaml} 4
fail 2098044 2018-01-22 03:26:32 2018-01-25 08:29:31 2018-01-25 09:05:31 0:36:00 0:21:07 0:14:53 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2018-01-25 08:51:58.838033 mon.b mon.0 172.21.15.94:6789/0 101 : cluster [WRN] Health check failed: 90 slow ops, oldest one blocked for 30 sec (SLOW_OPS)" in cluster log

pass 2098045 2018-01-22 03:26:33 2018-01-25 08:33:42 2018-01-25 08:55:42 0:22:00 0:10:40 0:11:20 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 2098046 2018-01-22 03:26:33 2018-01-25 08:33:42 2018-01-25 09:49:43 1:16:01 0:13:26 1:02:35 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 2098047 2018-01-22 03:26:34 2018-01-25 08:35:29 2018-01-25 08:55:29 0:20:00 0:08:30 0:11:30 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 2098048 2018-01-22 03:26:35 2018-01-25 08:35:29 2018-01-25 09:09:29 0:34:00 0:11:55 0:22:05 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2098049 2018-01-22 03:26:35 2018-01-25 08:35:29 2018-01-25 11:01:32 2:26:03 0:47:51 1:38:12 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/filestore-xfs.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
pass 2098050 2018-01-22 03:26:36 2018-01-25 08:35:29 2018-01-25 10:03:31 1:28:02 0:10:39 1:17:23 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 2098051 2018-01-22 03:26:37 2018-01-25 08:35:31 2018-01-25 09:45:32 1:10:01 0:58:33 0:11:28 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
pass 2098052 2018-01-22 03:26:37 2018-01-25 08:35:32 2018-01-25 09:11:32 0:36:00 0:23:24 0:12:36 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 2098053 2018-01-22 03:26:38 2018-01-25 08:35:32 2018-01-25 10:07:34 1:32:02 0:09:21 1:22:41 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
pass 2098054 2018-01-22 03:26:39 2018-01-25 08:37:34 2018-01-25 09:29:35 0:52:01 0:23:38 0:28:23 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2098055 2018-01-22 03:26:39 2018-01-25 08:37:35 2018-01-25 09:13:35 0:36:00 0:18:44 0:17:16 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2098056 2018-01-22 03:26:40 2018-01-25 08:37:34 2018-01-25 09:41:35 1:04:01 0:16:12 0:47:49 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
pass 2098057 2018-01-22 03:26:40 2018-01-25 08:39:37 2018-01-25 09:55:38 1:16:01 0:42:06 0:33:55 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 2098058 2018-01-22 03:26:41 2018-01-25 08:39:38 2018-01-25 09:11:38 0:32:00 0:11:09 0:20:51 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 2098059 2018-01-22 03:26:42 2018-01-25 08:41:22 2018-01-25 09:49:23 1:08:01 0:33:41 0:34:20 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
pass 2098060 2018-01-22 03:26:42 2018-01-25 08:45:49 2018-01-25 09:47:49 1:02:00 0:43:45 0:18:15 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
fail 2098061 2018-01-22 03:26:43 2018-01-25 08:47:36 2018-01-25 09:21:36 0:34:00 0:18:43 0:15:17 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 09:12:08.337191 mon.b mon.0 172.21.15.165:6789/0 97 : cluster [WRN] Health check failed: 51 slow ops, oldest one blocked for 32 sec (SLOW_OPS)" in cluster log

pass 2098062 2018-01-22 03:26:44 2018-01-25 08:49:44 2018-01-25 09:13:44 0:24:00 0:10:19 0:13:41 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 2098063 2018-01-22 03:26:44 2018-01-25 08:49:44 2018-01-25 09:11:44 0:22:00 0:09:52 0:12:08 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 2098064 2018-01-22 03:26:45 2018-01-25 08:51:57 2018-01-25 09:27:57 0:36:00 0:25:08 0:10:52 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2098065 2018-01-22 03:26:45 2018-01-25 08:53:32 2018-01-25 09:41:33 0:48:01 0:18:31 0:29:30 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/damage.yaml whitelist_health.yaml} 4
fail 2098066 2018-01-22 03:26:46 2018-01-25 08:53:32 2018-01-25 10:13:33 1:20:01 0:24:09 0:55:52 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 09:54:53.854690 mon.a mon.0 172.21.15.41:6789/0 120 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 2098067 2018-01-22 03:26:47 2018-01-25 08:53:32 2018-01-25 09:15:32 0:22:00 0:09:26 0:12:34 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 2098068 2018-01-22 03:26:47 2018-01-25 08:53:32 2018-01-25 10:17:34 1:24:02 0:20:55 1:03:07 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 2098069 2018-01-22 03:26:48 2018-01-25 08:53:32 2018-01-25 09:25:32 0:32:00 0:11:15 0:20:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 2098070 2018-01-22 03:26:49 2018-01-25 08:53:32 2018-01-25 09:59:33 1:06:01 0:10:12 0:55:49 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 2098071 2018-01-22 03:26:49 2018-01-25 08:53:44 2018-01-25 10:49:45 1:56:01 0:18:51 1:37:10 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 2098072 2018-01-22 03:26:50 2018-01-25 08:55:44 2018-01-25 10:01:45 1:06:01 0:08:23 0:57:38 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 2098073 2018-01-22 03:26:50 2018-01-25 08:55:44 2018-01-25 10:27:46 1:32:02 0:09:50 1:22:12 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2098074 2018-01-22 03:26:51 2018-01-25 08:55:44 2018-01-25 13:13:49 4:18:05 0:36:04 3:42:01 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
pass 2098075 2018-01-22 03:26:52 2018-01-25 08:55:44 2018-01-25 10:09:45 1:14:01 0:12:55 1:01:06 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
pass 2098076 2018-01-22 03:26:52 2018-01-25 08:55:44 2018-01-25 09:55:45 1:00:01 0:12:36 0:47:25 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 2098077 2018-01-22 03:26:53 2018-01-25 08:57:52 2018-01-25 09:43:52 0:46:00 0:23:38 0:22:22 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 2098078 2018-01-22 03:26:54 2018-01-25 08:57:52 2018-01-25 14:11:58 5:14:06 0:17:31 4:56:35 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
pass 2098079 2018-01-22 03:26:54 2018-01-25 08:59:33 2018-01-25 09:45:34 0:46:01 0:30:16 0:15:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2098080 2018-01-22 03:26:55 2018-01-25 09:01:44 2018-01-25 09:39:44 0:38:00 0:18:28 0:19:32 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2098081 2018-01-22 03:26:55 2018-01-25 09:03:45 2018-01-25 14:49:52 5:46:07 0:10:26 5:35:41 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
pass 2098082 2018-01-22 03:26:56 2018-01-25 09:03:45 2018-01-25 11:01:47 1:58:02 0:36:05 1:21:57 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 2098083 2018-01-22 03:26:57 2018-01-25 09:05:47 2018-01-25 09:37:47 0:32:00 0:19:55 0:12:05 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 2098084 2018-01-22 03:26:57 2018-01-25 09:09:36 2018-01-25 10:45:38 1:36:02 0:10:54 1:25:08 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_full_different_file (tasks.cephfs.test_full.TestClusterFull)

pass 2098085 2018-01-22 03:26:58 2018-01-25 09:09:41 2018-01-25 10:01:41 0:52:00 0:40:10 0:11:50 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
fail 2098086 2018-01-22 03:26:59 2018-01-25 09:11:44 2018-01-25 09:53:44 0:42:00 0:19:24 0:22:36 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 09:40:41.565625 mon.b mon.0 172.21.15.94:6789/0 105 : cluster [WRN] Health check failed: 4479 slow ops, oldest one blocked for 33 sec (SLOW_OPS)" in cluster log

pass 2098087 2018-01-22 03:26:59 2018-01-25 09:11:44 2018-01-25 09:57:45 0:46:01 0:09:59 0:36:02 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 2098088 2018-01-22 03:27:00 2018-01-25 09:11:44 2018-01-25 09:33:44 0:22:00 0:09:54 0:12:06 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 2098089 2018-01-22 03:27:00 2018-01-25 09:11:45 2018-01-25 09:43:45 0:32:00 0:18:09 0:13:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2098090 2018-01-22 03:27:01 2018-01-25 09:11:48 2018-01-25 10:11:48 1:00:00 0:11:58 0:48:02 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 2098091 2018-01-22 03:27:02 2018-01-25 09:13:49 2018-01-25 11:33:51 2:20:02 0:11:58 2:08:04 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2018-01-25 11:30:12.935553 mon.a mon.0 172.21.15.36:6789/0 25 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 2098092 2018-01-22 03:27:02 2018-01-25 09:13:49 2018-01-25 09:35:49 0:22:00 0:09:30 0:12:30 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 2098093 2018-01-22 03:27:03 2018-01-25 09:15:44 2018-01-25 10:19:44 1:04:00 0:27:16 0:36:44 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/strays.yaml whitelist_health.yaml} 4
pass 2098094 2018-01-22 03:27:04 2018-01-25 09:15:44 2018-01-25 09:39:48 0:24:04 0:11:38 0:12:26 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 2098095 2018-01-22 03:27:04 2018-01-25 09:15:44 2018-01-25 09:47:44 0:32:00 0:10:03 0:21:57 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 2098096 2018-01-22 03:27:05 2018-01-25 09:17:50 2018-01-25 11:45:52 2:28:02 0:14:48 2:13:14 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 2098097 2018-01-22 03:27:06 2018-01-25 09:19:46 2018-01-25 09:45:46 0:26:00 0:08:15 0:17:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 2098098 2018-01-22 03:27:06 2018-01-25 09:21:50 2018-01-25 10:21:51 1:00:01 0:09:06 0:50:55 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2098099 2018-01-22 03:27:07 2018-01-25 09:23:33 2018-01-25 12:47:37 3:24:04 0:47:29 2:36:35 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-comp.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
pass 2098100 2018-01-22 03:27:08 2018-01-25 09:25:44 2018-01-25 14:49:50 5:24:06 0:10:49 5:13:17 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 2098101 2018-01-22 03:27:08 2018-01-25 09:25:44 2018-01-25 10:07:44 0:42:00 0:19:33 0:22:27 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
pass 2098102 2018-01-22 03:27:09 2018-01-25 09:28:13 2018-01-25 10:06:13 0:38:00 0:22:30 0:15:30 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 2098103 2018-01-22 03:27:09 2018-01-25 09:29:44 2018-01-25 10:37:44 1:08:00 0:09:53 0:58:07 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
pass 2098104 2018-01-22 03:27:10 2018-01-25 09:34:01 2018-01-25 11:42:03 2:08:02 0:30:06 1:37:56 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2098105 2018-01-22 03:27:11 2018-01-25 09:36:05 2018-01-25 10:08:04 0:31:59 0:18:53 0:13:06 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2098106 2018-01-22 03:27:11 2018-01-25 09:37:30 2018-01-25 11:53:33 2:16:03 0:24:01 1:52:02 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
pass 2098107 2018-01-22 03:27:12 2018-01-25 09:37:48 2018-01-25 10:43:49 1:06:01 0:46:26 0:19:35 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 2098108 2018-01-22 03:27:13 2018-01-25 09:39:38 2018-01-25 10:23:38 0:44:00 0:21:00 0:23:00 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 2098109 2018-01-22 03:27:13 2018-01-25 09:39:38 2018-01-25 11:11:40 1:32:02 0:34:06 0:57:56 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
pass 2098110 2018-01-22 03:27:14 2018-01-25 09:39:38 2018-01-25 10:31:39 0:52:01 0:38:47 0:13:14 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
pass 2098111 2018-01-22 03:27:14 2018-01-25 09:39:45 2018-01-25 10:15:45 0:36:00 0:23:54 0:12:06 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 2098112 2018-01-22 03:27:15 2018-01-25 09:39:49 2018-01-25 11:19:51 1:40:02 0:11:04 1:28:58 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 2098113 2018-01-22 03:27:16 2018-01-25 09:41:27 2018-01-25 10:01:26 0:19:59 0:09:41 0:10:18 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 2098114 2018-01-22 03:27:16 2018-01-25 09:41:32 2018-01-25 10:29:32 0:48:00 0:18:49 0:29:11 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2098115 2018-01-22 03:27:17 2018-01-25 09:41:33 2018-01-25 10:35:33 0:54:00 0:18:21 0:35:39 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/damage.yaml whitelist_health.yaml} 4
fail 2098116 2018-01-22 03:27:18 2018-01-25 09:41:34 2018-01-25 11:09:35 1:28:01 1:11:42 0:16:19 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 10:02:21.334381 mon.b mon.0 172.21.15.64:6789/0 152 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

pass 2098117 2018-01-22 03:27:18 2018-01-25 09:41:36 2018-01-25 10:03:36 0:22:00 0:09:47 0:12:13 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 2098118 2018-01-22 03:27:19 2018-01-25 09:41:40 2018-01-25 12:11:43 2:30:03 0:23:58 2:06:05 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 2098119 2018-01-22 03:27:20 2018-01-25 09:44:02 2018-01-25 10:08:01 0:23:59 0:10:42 0:13:17 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 2098120 2018-01-22 03:27:20 2018-01-25 09:44:02 2018-01-25 10:22:02 0:38:00 0:10:19 0:27:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 2098121 2018-01-22 03:27:21 2018-01-25 09:44:15 2018-01-25 10:40:15 0:56:00 0:19:16 0:36:44 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 2098122 2018-01-22 03:27:22 2018-01-25 09:45:47 2018-01-25 10:23:47 0:38:00 0:08:46 0:29:14 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 2098123 2018-01-22 03:27:22 2018-01-25 09:45:47 2018-01-25 10:15:46 0:29:59 0:09:18 0:20:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2098124 2018-01-22 03:27:23 2018-01-25 09:45:47 2018-01-25 12:19:49 2:34:02 0:36:44 1:57:18 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
pass 2098125 2018-01-22 03:27:24 2018-01-25 09:47:43 2018-01-25 10:23:43 0:36:00 0:12:35 0:23:25 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
pass 2098126 2018-01-22 03:27:24 2018-01-25 09:47:45 2018-01-25 10:15:45 0:28:00 0:11:39 0:16:21 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
fail 2098127 2018-01-22 03:27:25 2018-01-25 09:47:50 2018-01-25 11:19:51 1:32:01 0:27:55 1:04:06 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2018-01-25 11:00:08.250057 mon.b mon.0 172.21.15.94:6789/0 143 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 2098128 2018-01-22 03:27:25 2018-01-25 09:49:40 2018-01-25 10:35:40 0:46:00 0:18:21 0:27:39 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
pass 2098129 2018-01-22 03:27:26 2018-01-25 09:49:44 2018-01-25 10:33:44 0:44:00 0:25:40 0:18:20 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2098130 2018-01-22 03:27:27 2018-01-25 09:54:01 2018-01-25 11:36:02 1:42:01 0:18:51 1:23:10 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2098131 2018-01-22 03:27:27 2018-01-25 09:55:52 2018-01-25 11:07:53 1:12:01 0:09:50 1:02:11 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
pass 2098132 2018-01-22 03:27:28 2018-01-25 09:55:52 2018-01-25 10:51:52 0:56:00 0:46:47 0:09:13 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 2098133 2018-01-22 03:27:29 2018-01-25 09:57:59 2018-01-25 11:34:02 1:36:03 1:12:47 0:23:16 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
dead 2098134 2018-01-22 03:27:29 2018-01-25 09:59:48 2018-01-25 22:02:18 12:02:30 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
pass 2098135 2018-01-22 03:27:30 2018-01-25 09:59:48 2018-01-25 12:15:50 2:16:02 0:46:05 1:29:57 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
fail 2098136 2018-01-22 03:27:31 2018-01-25 10:01:33 2018-01-25 12:27:35 2:26:02 1:11:05 1:14:57 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 11:24:57.717826 mon.a mon.0 172.21.15.68:6789/0 125 : cluster [WRN] Health check failed: 16585 slow ops, oldest one blocked for 79 sec (SLOW_OPS)" in cluster log

pass 2098137 2018-01-22 03:27:31 2018-01-25 10:01:42 2018-01-25 12:53:45 2:52:03 0:10:21 2:41:42 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 2098138 2018-01-22 03:27:32 2018-01-25 10:01:46 2018-01-25 10:25:45 0:23:59 0:09:14 0:14:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 2098139 2018-01-22 03:27:32 2018-01-25 10:03:44 2018-01-25 11:33:45 1:30:01 0:14:49 1:15:12 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2098140 2018-01-22 03:27:33 2018-01-25 10:03:44 2018-01-25 10:27:44 0:24:00 0:11:25 0:12:35 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 2098141 2018-01-22 03:27:34 2018-01-25 10:03:44 2018-01-25 11:57:46 1:54:02 0:11:17 1:42:45 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2018-01-25 11:52:56.096540 mon.a mon.1 172.21.15.145:6789/0 84 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 2098142 2018-01-22 03:27:34 2018-01-25 10:06:28 2018-01-25 11:04:29 0:58:01 0:09:32 0:48:29 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 2098143 2018-01-22 03:27:35 2018-01-25 10:07:47 2018-01-25 12:35:50 2:28:03 0:26:17 2:01:46 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/strays.yaml whitelist_health.yaml} 4
pass 2098144 2018-01-22 03:27:36 2018-01-25 10:07:47 2018-01-25 10:45:47 0:38:00 0:11:02 0:26:58 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 2098145 2018-01-22 03:27:36 2018-01-25 10:07:47 2018-01-25 10:33:47 0:26:00 0:10:28 0:15:32 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 2098146 2018-01-22 03:27:37 2018-01-25 10:07:48 2018-01-25 12:31:50 2:24:02 0:13:24 2:10:38 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 2098147 2018-01-22 03:27:37 2018-01-25 10:08:02 2018-01-25 10:34:02 0:26:00 0:08:32 0:17:28 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 2098148 2018-01-22 03:27:38 2018-01-25 10:08:05 2018-01-25 11:34:06 1:26:01 0:08:46 1:17:15 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2098149 2018-01-22 03:27:39 2018-01-25 10:10:00 2018-01-25 11:56:03 1:46:03 0:48:19 0:57:44 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
pass 2098150 2018-01-22 03:27:39 2018-01-25 10:11:17 2018-01-25 11:19:18 1:08:01 0:10:32 0:57:29 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 2098151 2018-01-22 03:27:40 2018-01-25 10:11:49 2018-01-25 11:57:51 1:46:02 0:27:13 1:18:49 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
pass 2098152 2018-01-22 03:27:41 2018-01-25 10:13:39 2018-01-25 10:53:39 0:40:00 0:27:49 0:12:11 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 2098153 2018-01-22 03:27:41 2018-01-25 10:15:59 2018-01-25 10:51:59 0:36:00 0:09:32 0:26:28 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
pass 2098154 2018-01-22 03:27:42 2018-01-25 10:15:59 2018-01-25 11:07:59 0:52:00 0:26:10 0:25:50 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2098155 2018-01-22 03:27:43 2018-01-25 10:15:59 2018-01-25 11:03:59 0:48:00 0:18:53 0:29:07 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2098156 2018-01-22 03:27:43 2018-01-25 10:17:31 2018-01-25 10:47:31 0:30:00 0:15:46 0:14:14 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
pass 2098157 2018-01-22 03:27:44 2018-01-25 10:17:35 2018-01-25 11:17:35 1:00:00 0:40:53 0:19:07 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 2098158 2018-01-22 03:27:44 2018-01-25 10:19:54 2018-01-25 10:43:54 0:24:00 0:11:55 0:12:05 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 2098159 2018-01-22 03:27:45 2018-01-25 10:19:55 2018-01-25 11:43:55 1:24:00 0:38:45 0:45:15 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
pass 2098160 2018-01-22 03:27:46 2018-01-25 10:22:08 2018-01-25 11:10:09 0:48:01 0:36:57 0:11:04 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
fail 2098161 2018-01-22 03:27:46 2018-01-25 10:22:27 2018-01-25 12:10:28 1:48:01 0:53:48 0:54:13 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 11:24:06.421377 mon.b mon.0 172.21.15.53:6789/0 142 : cluster [WRN] Health check failed: 4875 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2098162 2018-01-22 03:27:47 2018-01-25 10:23:52 2018-01-25 10:55:52 0:32:00 0:10:18 0:21:42 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 2098163 2018-01-22 03:27:48 2018-01-25 10:23:52 2018-01-25 10:49:52 0:26:00 0:09:41 0:16:19 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
dead 2098164 2018-01-22 03:27:49 2018-01-25 10:23:52 2018-01-25 22:26:22 12:02:30 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2098165 2018-01-22 03:27:49 2018-01-25 10:25:59 2018-01-25 18:34:09 8:08:10 0:19:17 7:48:53 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/damage.yaml whitelist_health.yaml} 4
fail 2098166 2018-01-22 03:27:50 2018-01-25 10:27:58 2018-01-25 12:00:00 1:32:02 0:24:56 1:07:06 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2018-01-25 11:45:38.797948 mon.b mon.0 172.21.15.82:6789/0 111 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

pass 2098167 2018-01-22 03:27:50 2018-01-25 10:27:58 2018-01-25 11:58:00 1:30:02 0:16:15 1:13:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 2098168 2018-01-22 03:27:51 2018-01-25 10:29:46 2018-01-25 11:57:47 1:28:01 0:25:32 1:02:29 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
fail 2098169 2018-01-22 03:27:52 2018-01-25 10:31:43 2018-01-25 12:13:44 1:42:01 0:21:09 1:20:52 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2018-01-25 12:01:19.825211 mon.b mon.0 172.21.15.49:6789/0 144 : cluster [WRN] Health check failed: 9 slow ops, oldest one blocked for 31 sec (SLOW_OPS)" in cluster log

pass 2098170 2018-01-22 03:27:52 2018-01-25 10:31:43 2018-01-25 11:19:43 0:48:00 0:10:13 0:37:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 2098171 2018-01-22 03:27:53 2018-01-25 10:33:59 2018-01-25 11:30:00 0:56:01 0:19:38 0:36:23 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 2098172 2018-01-22 03:27:54 2018-01-25 10:33:59 2018-01-25 11:24:00 0:50:01 0:08:37 0:41:24 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 2098173 2018-01-22 03:27:54 2018-01-25 10:34:03 2018-01-25 11:00:03 0:26:00 0:09:17 0:16:43 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 2098174 2018-01-22 03:27:55 2018-01-25 10:35:47 2018-01-25 11:23:47 0:48:00 0:27:10 0:20:50 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/filestore-xfs.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
pass 2098175 2018-01-22 03:27:56 2018-01-25 10:35:47 2018-01-25 11:25:47 0:50:00 0:12:40 0:37:20 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
pass 2098176 2018-01-22 03:27:56 2018-01-25 10:37:59 2018-01-25 11:57:59 1:20:00 0:24:14 0:55:46 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 2098177 2018-01-22 03:27:57 2018-01-25 10:40:30 2018-01-25 12:02:30 1:22:00 0:29:10 0:52:50 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 2098178 2018-01-22 03:27:58 2018-01-25 10:43:56 2018-01-25 15:14:01 4:30:05 0:18:00 4:12:05 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
pass 2098179 2018-01-22 03:27:58 2018-01-25 10:43:56 2018-01-25 11:25:56 0:42:00 0:23:30 0:18:30 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_misc.yaml} 3
pass 2098180 2018-01-22 03:27:59 2018-01-25 10:45:53 2018-01-25 11:19:52 0:33:59 0:18:53 0:15:06 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 2098181 2018-01-22 03:27:59 2018-01-25 10:45:53 2018-01-25 13:13:55 2:28:02 0:09:21 2:18:41 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
pass 2098182 2018-01-22 03:28:00 2018-01-25 10:47:45 2018-01-25 11:37:46 0:50:01 0:40:57 0:09:04 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 2098183 2018-01-22 03:28:01 2018-01-25 10:49:45 2018-01-25 11:59:45 1:10:00 0:29:12 0:40:48 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 2098184 2018-01-22 03:28:01 2018-01-25 10:49:47 2018-01-25 12:15:48 1:26:01 0:07:37 1:18:24 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi200 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true'

pass 2098185 2018-01-22 03:28:02 2018-01-25 10:49:53 2018-01-25 11:47:54 0:58:01 0:41:14 0:16:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
pass 2098186 2018-01-22 03:28:03 2018-01-25 10:52:06 2018-01-25 11:30:06 0:38:00 0:18:54 0:19:06 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 2098187 2018-01-22 03:28:03 2018-01-25 10:52:06 2018-01-25 12:42:08 1:50:02 0:10:07 1:39:55 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 2098188 2018-01-22 03:28:04 2018-01-25 10:53:53 2018-01-25 11:13:52 0:19:59 0:09:07 0:10:52 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 2098189 2018-01-22 03:28:05 2018-01-25 10:55:05 2018-01-25 12:33:06 1:38:01 0:24:34 1:13:27 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 2098190 2018-01-22 03:28:05 2018-01-25 10:55:54 2018-01-25 12:07:54 1:12:00 0:12:15 0:59:45 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 2098191 2018-01-22 03:28:06 2018-01-25 10:57:32 2018-01-25 12:03:33 1:06:01 0:28:25 0:37:36 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2018-01-25 11:47:12.552508 mon.b mon.0 172.21.15.92:6789/0 108 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

pass 2098192 2018-01-22 03:28:07 2018-01-25 10:57:43 2018-01-25 11:23:43 0:26:00 0:09:23 0:16:37 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 2098193 2018-01-22 03:28:07 2018-01-25 10:59:34 2018-01-25 12:33:35 1:34:01 0:26:26 1:07:35 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/strays.yaml whitelist_health.yaml} 4
pass 2098194 2018-01-22 03:28:08 2018-01-25 10:59:34 2018-01-25 11:43:34 0:44:00 0:14:43 0:29:17 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 2098195 2018-01-22 03:28:09 2018-01-25 10:59:44 2018-01-25 12:01:45 1:02:01 0:10:10 0:51:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 2098196 2018-01-22 03:28:09 2018-01-25 11:00:04 2018-01-25 12:18:05 1:18:01 0:13:36 1:04:25 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 2098197 2018-01-22 03:28:10 2018-01-25 11:01:46 2018-01-25 11:19:46 0:18:00 0:07:58 0:10:02 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_trivial_sync.yaml} 3