Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 1792296 2017-10-30 20:58:47 2017-10-30 21:06:17 2017-10-30 21:30:17 0:24:00 0:13:53 0:10:07 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792299 2017-10-30 20:58:48 2017-10-30 21:06:44 2017-10-30 23:42:47 2:36:03 0:50:23 1:45:40 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
Failure Reason:

"2017-10-30 23:12:19.731239 mon.a mon.0 172.21.15.64:6789/0 145 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792302 2017-10-30 20:58:48 2017-10-30 21:08:22 2017-10-31 01:02:27 3:54:05 0:12:20 3:41:45 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 1792305 2017-10-30 20:58:49 2017-10-30 21:08:40 2017-10-30 22:02:41 0:54:01 0:47:32 0:06:29 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1792308 2017-10-30 20:58:50 2017-10-30 21:11:08 2017-10-30 21:53:08 0:42:00 0:31:43 0:10:17 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-10-30 21:25:30.595377 mon.b mon.0 172.21.15.53:6789/0 128 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792311 2017-10-30 20:58:50 2017-10-30 21:12:08 2017-10-30 21:38:08 0:26:00 0:14:18 0:11:42 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
fail 1792314 2017-10-30 20:58:51 2017-10-30 21:12:24 2017-10-30 21:58:24 0:46:00 0:36:23 0:09:37 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-30 21:25:55.781240 mon.a mon.0 172.21.15.89:6789/0 123 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792317 2017-10-30 20:58:52 2017-10-30 21:12:44 2017-10-30 21:46:44 0:34:00 0:17:45 0:16:15 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1792320 2017-10-30 20:58:52 2017-10-30 21:14:19 2017-10-30 22:48:20 1:34:01 0:20:29 1:13:32 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
fail 1792324 2017-10-30 20:58:53 2017-10-30 21:14:38 2017-10-30 23:04:39 1:50:01 0:43:10 1:06:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-30 22:33:12.127867 mon.b mon.0 172.21.15.77:6789/0 228 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792327 2017-10-30 20:58:54 2017-10-30 21:16:24 2017-10-30 22:16:24 1:00:00 0:12:01 0:47:59 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 1792330 2017-10-30 20:58:55 2017-10-30 21:18:37 2017-10-30 23:08:38 1:50:01 0:38:43 1:11:18 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
fail 1792332 2017-10-30 20:58:55 2017-10-30 21:20:15 2017-10-30 22:28:16 1:08:01 0:38:08 0:29:53 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-30 22:09:21.101578 mon.a mon.0 172.21.15.96:6789/0 127 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792335 2017-10-30 20:58:56 2017-10-30 21:20:32 2017-10-30 21:48:32 0:28:00 0:20:20 0:07:40 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1792338 2017-10-30 20:58:56 2017-10-30 21:23:53 2017-10-30 22:11:53 0:48:00 0:14:10 0:33:50 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 1792341 2017-10-30 20:58:57 2017-10-30 21:26:27 2017-10-30 21:46:26 0:19:59 0:10:35 0:09:24 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1792344 2017-10-30 20:58:58 2017-10-30 21:28:32 2017-10-30 22:04:32 0:36:00 0:19:35 0:16:25 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1792348 2017-10-30 20:58:59 2017-10-30 21:30:19 2017-10-30 22:38:20 1:08:01 0:23:53 0:44:08 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/damage.yaml whitelist_health.yaml} 4
Failure Reason:

Found coredumps on ubuntu@smithi143.front.sepia.ceph.com

fail 1792350 2017-10-30 20:58:59 2017-10-30 21:30:38 2017-10-30 23:00:39 1:30:01 1:09:10 0:20:51 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-10-30 21:55:01.532332 mon.b mon.0 172.21.15.93:6789/0 140 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 1792353 2017-10-30 20:59:00 2017-10-30 21:32:22 2017-10-30 21:54:21 0:21:59 0:09:40 0:12:19 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1792356 2017-10-30 20:59:01 2017-10-30 21:34:06 2017-10-30 23:22:07 1:48:01 0:28:22 1:19:39 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 1792359 2017-10-30 20:59:01 2017-10-30 21:36:24 2017-10-30 22:02:24 0:26:00 0:15:58 0:10:02 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1792362 2017-10-30 20:59:02 2017-10-30 21:38:30 2017-10-30 22:00:30 0:22:00 0:14:39 0:07:21 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 1792365 2017-10-30 20:59:03 2017-10-30 21:44:10 2017-10-30 22:36:10 0:52:00 0:23:44 0:28:16 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 1792368 2017-10-30 20:59:03 2017-10-30 21:44:44 2017-10-30 22:04:44 0:20:00 0:09:14 0:10:46 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1792371 2017-10-30 20:59:04 2017-10-30 21:46:13 2017-10-30 22:12:13 0:26:00 0:10:12 0:15:48 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792374 2017-10-30 20:59:04 2017-10-30 21:46:38 2017-10-31 00:10:40 2:24:02 0:44:54 1:39:08 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-comp.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
Failure Reason:

"2017-10-30 23:34:05.340223 mon.b mon.0 172.21.15.21:6789/0 117 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

fail 1792377 2017-10-30 20:59:05 2017-10-30 21:48:22 2017-10-30 23:22:23 1:34:01 0:11:06 1:22:55 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)

pass 1792380 2017-10-30 20:59:06 2017-10-30 21:50:12 2017-10-30 22:18:11 0:27:59 0:17:34 0:10:25 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
fail 1792383 2017-10-30 20:59:06 2017-10-30 21:53:09 2017-10-30 23:53:11 2:00:02 0:31:44 1:28:18 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-10-30 23:30:22.732628 mon.a mon.0 172.21.15.31:6789/0 107 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792386 2017-10-30 20:59:07 2017-10-30 21:54:40 2017-10-30 23:18:41 1:24:01 0:17:20 1:06:41 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
fail 1792389 2017-10-30 20:59:08 2017-10-30 21:58:27 2017-10-30 22:42:27 0:44:00 0:37:13 0:06:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-30 22:10:13.055650 mon.a mon.0 172.21.15.56:6789/0 155 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792392 2017-10-30 20:59:08 2017-10-30 22:02:20 2017-10-30 23:12:20 1:10:00 0:18:46 0:51:14 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1792395 2017-10-30 20:59:09 2017-10-30 22:02:42 2017-10-31 00:18:44 2:16:02 0:14:10 2:01:52 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
fail 1792398 2017-10-30 20:59:10 2017-10-30 22:04:33 2017-10-30 23:12:34 1:08:01 0:54:40 0:13:21 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-30 22:32:47.564505 mon.a mon.0 172.21.15.48:6789/0 232 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792401 2017-10-30 20:59:10 2017-10-30 22:06:42 2017-10-30 22:36:42 0:30:00 0:24:12 0:05:48 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1792403 2017-10-30 20:59:11 2017-10-30 22:08:06 2017-10-30 23:34:07 1:26:01 0:08:22 1:17:39 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi162 with status 1: 'sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmpgNeALF'

fail 1792406 2017-10-30 20:59:11 2017-10-30 22:09:03 2017-10-30 23:05:03 0:56:00 0:40:48 0:15:12 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-30 22:47:23.941106 mon.a mon.0 172.21.15.9:6789/0 139 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792409 2017-10-30 20:59:12 2017-10-30 22:12:05 2017-10-30 23:14:05 1:02:00 0:27:04 0:34:56 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1792412 2017-10-30 20:59:13 2017-10-30 22:12:39 2017-10-30 22:48:39 0:36:00 0:19:54 0:16:06 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 1792415 2017-10-30 20:59:13 2017-10-30 22:16:23 2017-10-30 22:28:22 0:11:59 0:09:57 0:02:02 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1792418 2017-10-30 20:59:14 2017-10-30 22:18:21 2017-10-30 23:12:22 0:54:01 0:22:31 0:31:30 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 1792421 2017-10-30 20:59:15 2017-10-30 22:20:24 2017-10-30 22:48:24 0:28:00 0:17:31 0:10:29 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 1792424 2017-10-30 20:59:15 2017-10-30 22:26:35 2017-10-30 22:46:35 0:20:00 0:15:56 0:04:04 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-10-30 22:37:17.892020 mon.a mon.0 172.21.15.24:6789/0 159 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 1792427 2017-10-30 20:59:16 2017-10-30 22:28:26 2017-10-30 22:58:26 0:30:00 0:10:06 0:19:54 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1792430 2017-10-30 20:59:17 2017-10-30 22:28:47 2017-10-30 23:20:47 0:52:00 0:32:54 0:19:06 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/strays.yaml whitelist_health.yaml} 4
pass 1792433 2017-10-30 20:59:17 2017-10-30 22:30:18 2017-10-30 22:54:17 0:23:59 0:14:49 0:09:10 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1792436 2017-10-30 20:59:18 2017-10-30 22:32:26 2017-10-30 23:00:25 0:27:59 0:10:37 0:17:22 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 1792439 2017-10-30 20:59:19 2017-10-30 22:32:26 2017-10-30 22:56:25 0:23:59 0:14:48 0:09:11 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 1792442 2017-10-30 20:59:19 2017-10-30 22:34:53 2017-10-30 22:52:53 0:18:00 0:08:58 0:09:02 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1792445 2017-10-30 20:59:20 2017-10-30 22:36:43 2017-10-30 23:02:43 0:26:00 0:13:47 0:12:13 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792448 2017-10-30 20:59:21 2017-10-30 22:38:26 2017-10-31 02:50:33 4:12:07 0:57:53 3:14:14 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
Failure Reason:

"2017-10-31 02:15:03.055427 mon.a mon.0 172.21.15.3:6789/0 149 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792451 2017-10-30 20:59:21 2017-10-30 22:44:54 2017-10-30 23:06:54 0:22:00 0:11:33 0:10:27 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 1792454 2017-10-30 20:59:22 2017-10-30 22:48:31 2017-10-30 23:20:31 0:32:00 0:22:24 0:09:36 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1792457 2017-10-30 20:59:23 2017-10-30 22:48:31 2017-10-30 23:44:31 0:56:00 0:36:59 0:19:01 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-10-30 23:16:24.418705 mon.a mon.0 172.21.15.1:6789/0 118 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792460 2017-10-30 20:59:23 2017-10-30 22:50:25 2017-10-30 23:12:25 0:22:00 0:11:24 0:10:36 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
fail 1792463 2017-10-30 20:59:24 2017-10-30 22:52:46 2017-10-30 23:48:47 0:56:01 0:32:36 0:23:25 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-30 23:19:58.993382 mon.a mon.0 172.21.15.161:6789/0 113 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792466 2017-10-30 20:59:25 2017-10-30 22:54:24 2017-10-30 23:22:24 0:28:00 0:18:33 0:09:27 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1792469 2017-10-30 20:59:25 2017-10-30 22:56:34 2017-10-30 23:24:34 0:28:00 0:16:43 0:11:17 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
fail 1792472 2017-10-30 20:59:26 2017-10-30 22:58:54 2017-10-31 00:44:55 1:46:01 1:01:42 0:44:19 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-31 00:02:12.173213 mon.a mon.0 172.21.15.108:6789/0 259 : cluster [WRN] MDS health message (mds.0): Behind on trimming (64/30)" in cluster log

pass 1792475 2017-10-30 20:59:27 2017-10-30 23:01:19 2017-10-31 00:27:20 1:26:01 0:17:26 1:08:35 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 1792478 2017-10-30 20:59:27 2017-10-30 23:04:44 2017-10-31 00:20:45 1:16:01 1:13:44 0:02:17 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
fail 1792481 2017-10-30 20:59:28 2017-10-30 23:05:04 2017-10-31 00:25:05 1:20:01 0:36:55 0:43:06 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-31 00:06:26.875948 mon.b mon.0 172.21.15.49:6789/0 144 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792484 2017-10-30 20:59:29 2017-10-30 23:08:18 2017-10-31 01:00:19 1:52:01 1:10:02 0:41:59 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1792487 2017-10-30 20:59:29 2017-10-30 23:10:41 2017-10-30 23:38:41 0:28:00 0:14:38 0:13:22 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 1792491 2017-10-30 20:59:30 2017-10-30 23:12:26 2017-10-31 00:00:26 0:48:00 0:13:40 0:34:20 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1792493 2017-10-30 20:59:31 2017-10-30 23:12:36 2017-10-31 00:02:37 0:50:01 0:16:52 0:33:09 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1792496 2017-10-30 20:59:31 2017-10-30 23:17:41 2017-10-31 01:11:43 1:54:02 0:24:53 1:29:09 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/damage.yaml whitelist_health.yaml} 4
Failure Reason:

"2017-10-31 00:56:37.131774 mon.a mon.0 172.21.15.20:6789/0 386 : cluster [WRN] daemon mds.c is not responding, replacing it as rank 0 with standby daemon mds.a" in cluster log

fail 1792499 2017-10-30 20:59:32 2017-10-30 23:18:50 2017-10-30 23:46:49 0:27:59 0:19:38 0:08:21 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-10-30 23:29:35.028529 mon.b mon.0 172.21.15.77:6789/0 117 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 1792502 2017-10-30 20:59:33 2017-10-30 23:20:21 2017-10-30 23:40:20 0:19:59 0:13:08 0:06:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1792505 2017-10-30 20:59:33 2017-10-30 23:22:18 2017-10-31 00:10:18 0:48:00 0:28:59 0:19:01 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 1792508 2017-10-30 20:59:34 2017-10-30 23:22:24 2017-10-30 23:36:23 0:13:59 0:11:41 0:02:18 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1792511 2017-10-30 20:59:34 2017-10-30 23:22:30 2017-10-30 23:54:30 0:32:00 0:14:11 0:17:49 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 1792514 2017-10-30 20:59:35 2017-10-30 23:24:27 2017-10-31 00:18:27 0:54:00 0:23:02 0:30:58 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 1792517 2017-10-30 20:59:36 2017-10-30 23:24:36 2017-10-30 23:40:35 0:15:59 0:12:20 0:03:39 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1792520 2017-10-30 20:59:36 2017-10-30 23:28:45 2017-10-30 23:44:44 0:15:59 0:13:16 0:02:43 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792523 2017-10-30 20:59:37 2017-10-30 23:30:23 2017-10-31 00:24:23 0:54:00 0:44:19 0:09:41 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
Failure Reason:

"2017-10-30 23:47:35.709568 mon.a mon.0 172.21.15.91:6789/0 133 : cluster [WRN] MDS health message (mds.0): Behind on trimming (64/30)" in cluster log

fail 1792525 2017-10-30 20:59:38 2017-10-30 23:30:32 2017-10-31 00:56:33 1:26:01 0:15:17 1:10:44 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)

pass 1792528 2017-10-30 20:59:38 2017-10-30 23:33:37 2017-10-31 00:05:37 0:32:00 0:26:26 0:05:34 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
fail 1792532 2017-10-30 20:59:39 2017-10-30 23:36:36 2017-10-31 01:18:37 1:42:01 0:35:20 1:06:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-10-31 00:49:19.964630 mon.a mon.0 172.21.15.53:6789/0 141 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792535 2017-10-30 20:59:40 2017-10-30 23:36:54 2017-10-31 02:24:57 2:48:03 0:17:38 2:30:25 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
fail 1792537 2017-10-30 20:59:40 2017-10-30 23:38:44 2017-10-31 00:50:45 1:12:01 0:37:54 0:34:07 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-31 00:21:52.373622 mon.b mon.0 172.21.15.10:6789/0 133 : cluster [WRN] MDS health message (mds.0): Behind on trimming (63/30)" in cluster log

pass 1792540 2017-10-30 20:59:41 2017-10-30 23:40:32 2017-10-31 00:10:32 0:30:00 0:22:42 0:07:18 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1792543 2017-10-30 20:59:42 2017-10-30 23:42:53 2017-10-31 02:18:56 2:36:03 0:10:58 2:25:05 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
fail 1792546 2017-10-30 20:59:42 2017-10-30 23:44:36 2017-10-31 00:54:36 1:10:00 0:48:28 0:21:32 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-31 00:18:56.363174 mon.a mon.0 172.21.15.89:6789/0 251 : cluster [WRN] MDS health message (mds.0): Behind on trimming (63/30)" in cluster log

pass 1792549 2017-10-30 20:59:43 2017-10-30 23:44:49 2017-10-31 00:38:49 0:54:00 0:51:35 0:02:25 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1792552 2017-10-30 20:59:44 2017-10-30 23:46:27 2017-10-31 00:24:27 0:38:00 0:11:02 0:26:58 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi172 with status 1: 'sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmpOlQo6P'

fail 1792555 2017-10-30 20:59:44 2017-10-30 23:48:58 2017-10-31 01:04:59 1:16:01 0:36:16 0:39:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-31 00:46:11.233440 mon.b mon.0 172.21.15.85:6789/0 109 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792558 2017-10-30 20:59:45 2017-10-30 23:53:20 2017-10-31 00:49:20 0:56:00 0:53:29 0:02:31 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1792561 2017-10-30 20:59:46 2017-10-30 23:54:41 2017-10-31 01:30:42 1:36:01 0:15:11 1:20:50 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 1792564 2017-10-30 20:59:47 2017-10-30 23:56:51 2017-10-31 00:50:52 0:54:01 0:10:24 0:43:37 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1792567 2017-10-30 20:59:47 2017-10-30 23:58:50 2017-10-31 00:52:51 0:54:01 0:25:44 0:28:17 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 1792570 2017-10-30 20:59:48 2017-10-31 00:00:36 2017-10-31 02:02:38 2:02:02 0:16:28 1:45:34 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 1792573 2017-10-30 20:59:49 2017-10-31 00:03:12 2017-10-31 00:21:11 0:17:59 0:12:08 0:05:51 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-10-31 00:11:52.781236 mon.b mon.0 172.21.15.63:6789/0 140 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 1792576 2017-10-30 20:59:49 2017-10-31 00:04:41 2017-10-31 00:26:40 0:21:59 0:09:33 0:12:26 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1792579 2017-10-30 20:59:50 2017-10-31 00:04:54 2017-10-31 01:48:56 1:44:02 0:27:34 1:16:28 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/strays.yaml whitelist_health.yaml} 4
pass 1792582 2017-10-30 20:59:51 2017-10-31 00:06:33 2017-10-31 00:58:33 0:52:00 0:21:03 0:30:57 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1792585 2017-10-30 20:59:52 2017-10-31 00:08:56 2017-10-31 00:24:56 0:16:00 0:14:09 0:01:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 1792588 2017-10-30 20:59:52 2017-10-31 00:10:28 2017-10-31 02:28:32 2:18:04 0:19:41 1:58:23 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 1792591 2017-10-30 20:59:53 2017-10-31 00:10:33 2017-10-31 00:26:33 0:16:00 0:09:30 0:06:30 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1792594 2017-10-30 20:59:54 2017-10-31 00:10:46 2017-10-31 00:48:46 0:38:00 0:10:09 0:27:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792597 2017-10-30 20:59:55 2017-10-31 00:14:59 2017-10-31 02:05:00 1:50:01 0:42:44 1:07:17 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/filestore-xfs.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
Failure Reason:

"2017-10-31 01:40:32.764453 mon.b mon.0 172.21.15.54:6789/0 128 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792600 2017-10-30 20:59:55 2017-10-31 00:18:37 2017-10-31 00:36:36 0:17:59 0:15:30 0:02:29 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 1792604 2017-10-30 20:59:56 2017-10-31 00:18:46 2017-10-31 01:28:47 1:10:01 1:06:50 0:03:11 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1792608 2017-10-30 20:59:57 2017-10-31 00:21:12 2017-10-31 00:35:12 0:14:00 0:07:13 0:06:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

{'smithi076.front.sepia.ceph.com': {'msg': 'All items completed', 'failed': True, 'changed': False}}

pass 1792612 2017-10-30 20:59:58 2017-10-31 00:22:37 2017-10-31 00:44:37 0:22:00 0:14:23 0:07:37 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
fail 1792616 2017-10-30 20:59:58 2017-10-31 00:24:42 2017-10-31 01:36:42 1:12:00 0:40:57 0:31:03 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-31 01:02:49.280439 mon.a mon.0 172.21.15.102:6789/0 139 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792620 2017-10-30 20:59:59 2017-10-31 00:26:43 2017-10-31 01:38:44 1:12:01 0:23:02 0:48:59 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1792624 2017-10-30 21:00:00 2017-10-31 00:27:21 2017-10-31 01:57:22 1:30:01 0:16:25 1:13:36 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
fail 1792628 2017-10-30 21:00:00 2017-10-31 00:32:39 2017-10-31 01:44:39 1:12:00 0:53:26 0:18:34 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-31 01:07:58.372071 mon.b mon.0 172.21.15.81:6789/0 211 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792632 2017-10-30 21:00:01 2017-10-31 00:36:28 2017-10-31 01:04:28 0:28:00 0:11:59 0:16:01 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 1792636 2017-10-30 21:00:02 2017-10-31 00:36:42 2017-10-31 03:06:45 2:30:03 1:08:49 1:21:14 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
fail 1792640 2017-10-30 21:00:03 2017-10-31 00:38:50 2017-10-31 01:26:50 0:48:00 0:42:45 0:05:15 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-31 01:06:49.064366 mon.b mon.0 172.21.15.3:6789/0 119 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792644 2017-10-30 21:00:03 2017-10-31 00:41:03 2017-10-31 02:15:04 1:34:01 0:22:26 1:11:35 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1792648 2017-10-30 21:00:04 2017-10-31 00:42:48 2017-10-31 01:24:48 0:42:00 0:11:19 0:30:41 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 1792652 2017-10-30 21:00:05 2017-10-31 00:44:56 2017-10-31 01:10:56 0:26:00 0:13:27 0:12:33 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1792656 2017-10-30 21:00:05 2017-10-31 00:47:57 2017-10-31 02:45:59 1:58:02 0:32:58 1:25:04 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1792660 2017-10-30 21:00:06 2017-10-31 00:49:31 2017-10-31 01:47:31 0:58:00 0:21:00 0:37:00 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/damage.yaml whitelist_health.yaml} 4
Failure Reason:

Found coredumps on ubuntu@smithi055.front.sepia.ceph.com

fail 1792664 2017-10-30 21:00:07 2017-10-31 00:50:46 2017-10-31 01:32:47 0:42:01 0:25:15 0:16:46 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-10-31 01:09:49.179752 mon.a mon.1 172.21.15.156:6789/0 48 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 1792668 2017-10-30 21:00:07 2017-10-31 00:52:39 2017-10-31 01:06:38 0:13:59 0:09:57 0:04:02 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1792672 2017-10-30 21:00:08 2017-10-31 00:56:43 2017-10-31 01:54:43 0:58:00 0:26:11 0:31:49 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 1792675 2017-10-30 21:00:09 2017-10-31 00:56:47 2017-10-31 01:18:46 0:21:59 0:11:56 0:10:03 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1792680 2017-10-30 21:00:09 2017-10-31 00:58:43 2017-10-31 01:22:42 0:23:59 0:14:54 0:09:05 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 1792683 2017-10-30 21:00:10 2017-10-31 01:00:29 2017-10-31 01:22:30 0:22:01 0:19:48 0:02:13 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 1792687 2017-10-30 21:00:11 2017-10-31 01:02:36 2017-10-31 01:14:36 0:12:00 0:09:08 0:02:52 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1792690 2017-10-30 21:00:12 2017-10-31 01:02:46 2017-10-31 01:16:45 0:13:59 0:10:19 0:03:40 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792694 2017-10-30 21:00:12 2017-10-31 01:04:53 2017-10-31 02:28:54 1:24:01 0:42:07 0:41:54 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
Failure Reason:

"2017-10-31 01:51:46.150357 mon.b mon.0 172.21.15.67:6789/0 107 : cluster [WRN] MDS health message (mds.0): Behind on trimming (63/30)" in cluster log

fail 1792698 2017-10-30 21:00:13 2017-10-31 01:06:45 2017-10-31 01:52:45 0:46:00 0:15:16 0:30:44 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)

pass 1792702 2017-10-30 21:00:14 2017-10-31 01:08:42 2017-10-31 01:32:41 0:23:59 0:14:37 0:09:22 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
fail 1792706 2017-10-30 21:00:14 2017-10-31 01:10:53 2017-10-31 02:06:54 0:56:01 0:35:19 0:20:42 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-10-31 01:40:18.144571 mon.a mon.0 172.21.15.20:6789/0 109 : cluster [WRN] MDS health message (mds.0): Behind on trimming (64/30)" in cluster log

pass 1792710 2017-10-30 21:00:15 2017-10-31 01:11:44 2017-10-31 02:37:45 1:26:01 0:22:13 1:03:48 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
fail 1792715 2017-10-30 21:00:16 2017-10-31 01:12:49 2017-10-31 02:06:49 0:54:00 0:35:53 0:18:07 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-31 01:35:57.818072 mon.a mon.0 172.21.15.78:6789/0 152 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792718 2017-10-30 21:00:17 2017-10-31 01:14:37 2017-10-31 01:38:36 0:23:59 0:18:59 0:05:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1792721 2017-10-30 21:00:18 2017-10-31 01:16:54 2017-10-31 03:04:56 1:48:02 0:14:38 1:33:24 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
fail 1792724 2017-10-30 21:00:18 2017-10-31 01:18:53 2017-10-31 02:14:53 0:56:00 0:50:40 0:05:20 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-31 01:42:35.557700 mon.b mon.0 172.21.15.1:6789/0 218 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792727 2017-10-30 21:00:19 2017-10-31 01:20:53 2017-10-31 01:48:52 0:27:59 0:26:06 0:01:53 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1792730 2017-10-30 21:00:20 2017-10-31 01:22:31 2017-10-31 03:20:34 1:58:03 0:11:39 1:46:24 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi037 with status 1: 'sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmpGjFtg_'

fail 1792733 2017-10-30 21:00:20 2017-10-31 01:22:43 2017-10-31 02:08:43 0:46:00 0:39:17 0:06:43 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-31 01:47:49.694752 mon.b mon.0 172.21.15.95:6789/0 144 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792736 2017-10-30 21:00:21 2017-10-31 01:26:48 2017-10-31 02:06:48 0:40:00 0:23:38 0:16:22 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1792739 2017-10-30 21:00:22 2017-10-31 01:28:38 2017-10-31 02:54:40 1:26:02 0:16:34 1:09:28 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 1792742 2017-10-30 21:00:22 2017-10-31 01:29:00 2017-10-31 02:27:01 0:58:01 0:13:19 0:44:42 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1792745 2017-10-30 21:00:23 2017-10-31 01:30:46 2017-10-31 02:16:46 0:46:00 0:19:39 0:26:21 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 1792748 2017-10-30 21:00:24 2017-10-31 01:32:51 2017-10-31 04:40:54 3:08:03 0:12:49 2:55:14 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 1792751 2017-10-30 21:00:24 2017-10-31 01:32:54 2017-10-31 01:52:53 0:19:59 0:11:13 0:08:46 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-10-31 01:44:04.161122 mon.b mon.0 172.21.15.65:6789/0 120 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

pass 1792754 2017-10-30 21:00:25 2017-10-31 01:34:48 2017-10-31 02:00:48 0:26:00 0:09:46 0:16:14 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1792757 2017-10-30 21:00:26 2017-10-31 01:36:52 2017-10-31 03:18:54 1:42:02 0:27:49 1:14:13 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/strays.yaml whitelist_health.yaml} 4
pass 1792760 2017-10-30 21:00:27 2017-10-31 01:38:45 2017-10-31 02:44:45 1:06:00 0:14:16 0:51:44 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1792763 2017-10-30 21:00:27 2017-10-31 01:40:55 2017-10-31 02:00:55 0:20:00 0:14:35 0:05:25 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 1792766 2017-10-30 21:00:28 2017-10-31 01:42:51 2017-10-31 02:36:51 0:54:00 0:16:46 0:37:14 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 1792769 2017-10-30 21:00:29 2017-10-31 01:44:49 2017-10-31 02:14:49 0:30:00 0:12:19 0:17:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1792772 2017-10-30 21:00:29 2017-10-31 01:46:57 2017-10-31 02:56:58 1:10:01 0:11:10 0:58:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792775 2017-10-30 21:00:30 2017-10-31 01:48:46 2017-10-31 02:56:46 1:08:00 0:46:39 0:21:21 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-comp.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
Failure Reason:

"2017-10-31 02:32:32.177076 mon.b mon.0 172.21.15.75:6789/0 106 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792778 2017-10-30 21:00:31 2017-10-31 01:48:54 2017-10-31 02:04:53 0:15:59 0:15:25 0:00:34 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 1792781 2017-10-30 21:00:31 2017-10-31 01:50:54 2017-10-31 02:26:54 0:36:00 0:25:07 0:10:53 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1792784 2017-10-30 21:00:32 2017-10-31 01:52:55 2017-10-31 02:28:55 0:36:00 0:31:35 0:04:25 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-10-31 02:05:31.184045 mon.a mon.0 172.21.15.24:6789/0 143 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792786 2017-10-30 21:00:32 2017-10-31 01:52:55 2017-10-31 02:08:55 0:16:00 0:09:56 0:06:04 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
fail 1792789 2017-10-30 21:00:33 2017-10-31 01:54:53 2017-10-31 03:18:54 1:24:01 0:39:19 0:44:42 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-31 02:47:26.368419 mon.a mon.0 172.21.15.112:6789/0 109 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792792 2017-10-30 21:00:34 2017-10-31 01:57:23 2017-10-31 02:27:23 0:30:00 0:20:05 0:09:55 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1792795 2017-10-30 21:00:34 2017-10-31 01:58:33 2017-10-31 02:44:35 0:46:02 0:16:36 0:29:26 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
fail 1792797 2017-10-30 21:00:35 2017-10-31 02:00:59 2017-10-31 03:06:59 1:06:00 0:58:46 0:07:14 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-31 02:28:03.471978 mon.a mon.0 172.21.15.20:6789/0 229 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792800 2017-10-30 21:00:36 2017-10-31 02:02:57 2017-10-31 02:34:57 0:32:00 0:25:33 0:06:27 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 1792803 2017-10-30 21:00:37 2017-10-31 02:05:02 2017-10-31 03:33:03 1:28:01 0:39:56 0:48:05 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
fail 1792808 2017-10-30 21:00:37 2017-10-31 02:07:00 2017-10-31 03:27:01 1:20:01 0:40:58 0:39:03 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-31 03:06:53.326716 mon.b mon.0 172.21.15.48:6789/0 115 : cluster [WRN] MDS health message (mds.0): Behind on trimming (63/30)" in cluster log

pass 1792810 2017-10-30 21:00:38 2017-10-31 02:08:27 2017-10-31 02:52:27 0:44:00 0:24:35 0:19:25 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1792813 2017-10-30 21:00:39 2017-10-31 02:08:56 2017-10-31 04:30:58 2:22:02 0:14:29 2:07:33 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 1792817 2017-10-30 21:00:39 2017-10-31 02:13:09 2017-10-31 02:27:08 0:13:59 0:10:18 0:03:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1792819 2017-10-30 21:00:40 2017-10-31 02:14:50 2017-10-31 02:56:50 0:42:00 0:25:26 0:16:34 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1792822 2017-10-30 21:00:41 2017-10-31 02:14:57 2017-10-31 02:42:57 0:28:00 0:11:04 0:16:56 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/damage.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi060 with status 32: 'sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3'

fail 1792825 2017-10-30 21:00:41 2017-10-31 02:16:57 2017-10-31 03:50:58 1:34:01 1:27:27 0:06:34 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-10-31 02:28:55.057095 mon.a mon.0 172.21.15.142:6789/0 131 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 1792827 2017-10-30 21:00:42 2017-10-31 02:18:59 2017-10-31 02:30:58 0:11:59 0:10:06 0:01:53 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1792830 2017-10-30 21:00:43 2017-10-31 02:20:53 2017-10-31 02:56:53 0:36:00 0:29:52 0:06:08 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 1792833 2017-10-30 21:00:43 2017-10-31 02:25:03 2017-10-31 03:01:04 0:36:01 0:14:34 0:21:27 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1792836 2017-10-30 21:00:44 2017-10-31 02:26:55 2017-10-31 02:56:55 0:30:00 0:11:43 0:18:17 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 1792839 2017-10-30 21:00:45 2017-10-31 02:27:02 2017-10-31 03:05:02 0:38:00 0:26:55 0:11:05 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 1792842 2017-10-30 21:00:46 2017-10-31 02:28:43 2017-10-31 03:02:43 0:34:00 0:12:59 0:21:01 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1792845 2017-10-30 21:00:47 2017-10-31 02:29:34 2017-10-31 02:55:34 0:26:00 0:11:09 0:14:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792848 2017-10-30 21:00:47 2017-10-31 02:31:08 2017-10-31 03:53:09 1:22:01 0:45:01 0:37:00 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
Failure Reason:

"2017-10-31 03:13:44.081694 mon.a mon.0 172.21.15.89:6789/0 115 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

fail 1792851 2017-10-30 21:00:48 2017-10-31 02:31:27 2017-10-31 07:39:33 5:08:06 0:11:55 4:56:11 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)

pass 1792854 2017-10-30 21:00:49 2017-10-31 02:37:10 2017-10-31 03:12:55 0:35:45 0:18:23 0:17:22 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
fail 1792857 2017-10-30 21:00:50 2017-10-31 02:42:09 2017-10-31 03:58:07 1:15:58 0:34:39 0:41:19 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-10-31 03:27:43.510885 mon.a mon.0 172.21.15.183:6789/0 145 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792860 2017-10-30 21:00:51 2017-10-31 02:44:33 2017-10-31 04:50:30 2:05:57 0:21:42 1:44:15 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
fail 1792863 2017-10-30 21:00:51 2017-10-31 02:44:51 2017-10-31 03:44:48 0:59:57 0:35:58 0:23:59 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-31 03:14:25.927406 mon.b mon.0 172.21.15.52:6789/0 137 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792866 2017-10-30 21:00:52 2017-10-31 02:46:18 2017-10-31 03:22:17 0:35:59 0:19:11 0:16:48 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1792868 2017-10-30 21:00:53 2017-10-31 02:49:11 2017-10-31 05:41:13 2:52:02 0:10:39 2:41:23 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
fail 1792871 2017-10-30 21:00:54 2017-10-31 02:51:00 2017-10-31 04:03:01 1:12:01 0:55:04 0:16:57 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-31 03:21:05.715908 mon.a mon.0 172.21.15.45:6789/0 241 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792875 2017-10-30 21:00:54 2017-10-31 02:52:53 2017-10-31 04:12:54 1:20:01 0:52:30 0:27:31 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1792877 2017-10-30 21:00:55 2017-10-31 02:53:11 2017-10-31 03:11:11 0:18:00 0:10:41 0:07:19 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi092 with status 1: 'sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmppI_lor'

fail 1792881 2017-10-30 21:00:56 2017-10-31 02:57:02 2017-10-31 03:47:01 0:49:59 0:41:12 0:08:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-31 03:27:56.497330 mon.a mon.0 172.21.15.115:6789/0 110 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792884 2017-10-30 21:00:57 2017-10-31 02:57:02 2017-10-31 04:13:02 1:16:00 1:02:05 0:13:55 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1792886 2017-10-30 21:00:57 2017-10-31 02:57:02 2017-10-31 04:09:02 1:12:00 0:11:10 1:00:50 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 1792890 2017-10-30 21:00:58 2017-10-31 03:02:54 2017-10-31 03:42:54 0:40:00 0:10:31 0:29:29 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1792892 2017-10-30 21:00:59 2017-10-31 03:02:54 2017-10-31 04:08:54 1:06:00 0:20:44 0:45:16 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 1792895 2017-10-30 21:01:00 2017-10-31 03:04:59 2017-10-31 03:32:59 0:28:00 0:13:37 0:14:23 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 1792898 2017-10-30 21:01:00 2017-10-31 03:05:04 2017-10-31 03:31:03 0:25:59 0:15:20 0:10:39 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-10-31 03:21:20.957375 mon.a mon.1 172.21.15.143:6789/0 45 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 1792901 2017-10-30 21:01:01 2017-10-31 03:06:47 2017-10-31 04:16:49 1:10:02 0:10:38 0:59:24 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1792903 2017-10-30 21:01:02 2017-10-31 03:06:56 2017-10-31 04:56:58 1:50:02 0:30:49 1:19:13 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/strays.yaml whitelist_health.yaml} 4
pass 1792906 2017-10-30 21:01:03 2017-10-31 03:09:10 2017-10-31 03:53:10 0:44:00 0:15:46 0:28:14 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1792909 2017-10-30 21:01:04 2017-10-31 03:10:57 2017-10-31 04:30:58 1:20:01 0:10:27 1:09:34 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 1792912 2017-10-30 21:01:04 2017-10-31 03:13:07 2017-10-31 04:33:07 1:20:00 0:14:09 1:05:51 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 1792916 2017-10-30 21:01:05 2017-10-31 03:17:10 2017-10-31 04:29:10 1:12:00 0:09:00 1:03:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1792919 2017-10-30 21:01:06 2017-10-31 03:19:07 2017-10-31 04:05:07 0:46:00 0:12:48 0:33:12 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792922 2017-10-30 21:01:07 2017-10-31 03:19:12 2017-10-31 05:25:13 2:06:01 0:46:54 1:19:07 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/bluestore.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
Failure Reason:

"2017-10-31 05:00:24.120162 mon.b mon.0 172.21.15.99:6789/0 127 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792924 2017-10-30 21:01:08 2017-10-31 03:20:49 2017-10-31 05:44:51 2:24:02 0:11:34 2:12:28 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
dead 1792928 2017-10-30 21:01:08 2017-10-31 03:22:28 2017-10-31 15:29:12 12:06:44 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1792930 2017-10-30 21:01:09 2017-10-31 03:22:52 2017-10-31 04:22:53 1:00:01 0:34:18 0:25:43 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-10-31 03:53:23.125789 mon.a mon.0 172.21.15.117:6789/0 160 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792933 2017-10-30 21:01:10 2017-10-31 03:25:23 2017-10-31 03:47:23 0:22:00 0:13:23 0:08:37 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
fail 1792936 2017-10-30 21:01:11 2017-10-31 03:28:54 2017-10-31 04:06:54 0:38:00 0:33:13 0:04:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-31 03:38:08.548670 mon.b mon.0 172.21.15.61:6789/0 134 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792939 2017-10-30 21:01:11 2017-10-31 03:31:04 2017-10-31 03:59:03 0:27:59 0:23:09 0:04:50 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1792942 2017-10-30 21:01:12 2017-10-31 03:33:09 2017-10-31 04:15:09 0:42:00 0:17:13 0:24:47 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
fail 1792945 2017-10-30 21:01:13 2017-10-31 03:35:08 2017-10-31 04:43:08 1:08:00 0:52:38 0:15:22 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-31 04:07:20.035110 mon.b mon.0 172.21.15.10:6789/0 212 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1792948 2017-10-30 21:01:14 2017-10-31 03:39:12 2017-10-31 05:41:14 2:02:02 0:13:07 1:48:55 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp-ec-root.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 1792951 2017-10-30 21:01:15 2017-10-31 03:41:11 2017-10-31 07:09:15 3:28:04 1:09:19 2:18:45 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
fail 1792954 2017-10-30 21:01:15 2017-10-31 03:42:50 2017-10-31 04:48:52 1:06:02 0:36:51 0:29:11 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-31 04:30:41.930490 mon.b mon.0 172.21.15.60:6789/0 145 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

pass 1792957 2017-10-30 21:01:16 2017-10-31 03:44:41 2017-10-31 06:26:44 2:42:03 1:29:27 1:12:36 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1792960 2017-10-30 21:01:17 2017-10-31 03:45:01 2017-10-31 04:03:01 0:18:00 0:14:02 0:03:58 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 1792964 2017-10-30 21:01:18 2017-10-31 03:47:24 2017-10-31 04:11:24 0:24:00 0:14:19 0:09:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1792967 2017-10-30 21:01:19 2017-10-31 03:51:04 2017-10-31 04:35:04 0:44:00 0:31:14 0:12:46 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1792970 2017-10-30 21:01:20 2017-10-31 03:53:19 2017-10-31 04:53:19 1:00:00 0:24:59 0:35:01 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/damage.yaml whitelist_health.yaml} 4
Failure Reason:

"2017-10-31 04:37:08.044447 mon.a mon.0 172.21.15.20:6789/0 387 : cluster [WRN] daemon mds.d is not responding, replacing it as rank 0 with standby daemon mds.a" in cluster log

fail 1792973 2017-10-30 21:01:20 2017-10-31 03:55:16 2017-10-31 04:29:16 0:34:00 0:23:24 0:10:36 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-comp.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-10-31 04:11:54.746828 mon.b mon.0 172.21.15.13:6789/0 140 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 1792976 2017-10-30 21:01:21 2017-10-31 03:56:59 2017-10-31 04:42:59 0:46:00 0:10:18 0:35:42 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1792979 2017-10-30 21:01:22 2017-10-31 03:57:12 2017-10-31 04:33:12 0:36:00 0:29:06 0:06:54 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 1792982 2017-10-30 21:01:23 2017-10-31 03:59:05 2017-10-31 04:29:04 0:29:59 0:22:23 0:07:36 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1792985 2017-10-30 21:01:24 2017-10-31 04:03:12 2017-10-31 04:19:11 0:15:59 0:11:09 0:04:50 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 1792988 2017-10-30 21:01:25 2017-10-31 04:05:10 2017-10-31 05:25:10 1:20:00 0:19:45 1:00:15 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 1792991 2017-10-30 21:01:26 2017-10-31 04:05:12 2017-10-31 04:19:11 0:13:59 0:09:18 0:04:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1792994 2017-10-30 21:01:27 2017-10-31 04:08:36 2017-10-31 04:24:35 0:15:59 0:13:49 0:02:10 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1792997 2017-10-30 21:01:28 2017-10-31 04:09:04 2017-10-31 05:55:05 1:46:01 0:41:52 1:04:09 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore-ec/filestore-xfs.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
Failure Reason:

"2017-10-31 05:21:18.286003 mon.b mon.0 172.21.15.3:6789/0 95 : cluster [WRN] MDS health message (mds.0): Behind on trimming (63/30)" in cluster log

fail 1793000 2017-10-30 21:01:28 2017-10-31 04:11:05 2017-10-31 05:41:06 1:30:01 0:15:50 1:14:11 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub)

pass 1793003 2017-10-30 21:01:29 2017-10-31 04:11:17 2017-10-31 04:25:16 0:13:59 0:12:00 0:01:59 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore-ec-root.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
fail 1793006 2017-10-30 21:01:30 2017-10-31 04:13:03 2017-10-31 05:13:04 1:00:01 0:35:08 0:24:53 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-10-31 04:46:15.654052 mon.b mon.0 172.21.15.37:6789/0 130 : cluster [WRN] MDS health message (mds.0): Behind on trimming (62/30)" in cluster log

fail 1793009 2017-10-30 21:01:31 2017-10-31 04:15:12 2017-10-31 04:49:12 0:34:00 0:07:41 0:26:19 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi168 with status 32: 'sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-5'

fail 1793012 2017-10-30 21:01:31 2017-10-31 04:17:21 2017-10-31 05:27:21 1:10:00 0:39:15 0:30:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-10-31 04:54:13.046443 mon.b mon.0 172.21.15.68:6789/0 111 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1793015 2017-10-30 21:01:32 2017-10-31 04:19:16 2017-10-31 05:05:16 0:46:00 0:22:46 0:23:14 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1793018 2017-10-30 21:01:33 2017-10-31 04:21:19 2017-10-31 05:39:20 1:18:01 0:10:23 1:07:38 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
fail 1793020 2017-10-30 21:01:34 2017-10-31 04:23:05 2017-10-31 05:21:05 0:58:00 0:52:16 0:05:44 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-10-31 04:45:59.923631 mon.a mon.0 172.21.15.28:6789/0 224 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1793023 2017-10-30 21:01:34 2017-10-31 04:24:45 2017-10-31 04:58:45 0:34:00 0:23:59 0:10:01 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/bluestore.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1793027 2017-10-30 21:01:35 2017-10-31 04:25:17 2017-10-31 07:29:20 3:04:03 0:08:12 2:55:51 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi064 with status 1: 'sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmpNC78bl'

fail 1793029 2017-10-30 21:01:36 2017-10-31 04:29:14 2017-10-31 05:13:14 0:44:00 0:39:04 0:04:56 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
Failure Reason:

"2017-10-31 04:52:48.317694 mon.a mon.0 172.21.15.81:6789/0 154 : cluster [WRN] MDS health message (mds.0): Behind on trimming (61/30)" in cluster log

pass 1793032 2017-10-30 21:01:37 2017-10-31 04:31:08 2017-10-31 05:29:08 0:58:00 0:23:32 0:34:28 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1793036 2017-10-30 21:01:37 2017-10-31 04:33:17 2017-10-31 06:13:19 1:40:02 0:11:07 1:28:55 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-comp.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 1793039 2017-10-30 21:01:38 2017-10-31 04:35:09 2017-10-31 05:43:12 1:08:03 0:10:29 0:57:34 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
dead 1793042 2017-10-30 21:01:39 2017-10-31 04:37:13 2017-10-31 16:43:30 12:06:17 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-comp-ec-root.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 1793045 2017-10-30 21:01:40 2017-10-31 04:37:41 2017-10-31 06:25:42 1:48:01 0:14:15 1:33:46 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 1793049 2017-10-30 21:01:41 2017-10-31 04:43:10 2017-10-31 05:15:10 0:32:00 0:23:05 0:08:55 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore-ec/filestore-xfs.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-10-31 04:54:30.621799 mon.a mon.0 172.21.15.46:6789/0 31 : cluster [WRN] Health check update: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

pass 1793051 2017-10-30 21:01:41 2017-10-31 04:43:10 2017-10-31 05:25:10 0:42:00 0:10:35 0:31:25 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore-comp.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1793054 2017-10-30 21:01:42 2017-10-31 04:45:08 2017-10-31 05:27:08 0:42:00 0:30:54 0:11:06 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml tasks/strays.yaml whitelist_health.yaml} 4
pass 1793058 2017-10-30 21:01:43 2017-10-31 04:47:07 2017-10-31 05:01:07 0:14:00 0:11:50 0:02:10 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/bluestore-ec-root.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1793061 2017-10-30 21:01:44 2017-10-31 04:49:03 2017-10-31 05:15:03 0:26:00 0:10:55 0:15:05 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore-ec/bluestore.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 1793064 2017-10-30 21:01:44 2017-10-31 04:53:15 2017-10-31 05:15:14 0:21:59 0:16:05 0:05:54 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
Failure Reason:

Test failure: test_data_isolated (tasks.cephfs.test_volume_client.TestVolumeClient)

pass 1793067 2017-10-30 21:01:45 2017-10-31 04:53:21 2017-10-31 05:23:20 0:29:59 0:12:56 0:17:03 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore-ec/filestore-xfs.yaml tasks/kclient_workunit_trivial_sync.yaml} 3