Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 1401862 2017-07-15 03:25:22 2017-07-15 10:38:25 2017-07-15 11:04:24 0:25:59 0:09:22 0:16:37 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1401865 2017-07-15 03:25:22 2017-07-15 10:38:36 2017-07-15 12:00:36 1:22:00 1:03:00 0:19:00 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore/bluestore.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
Failure Reason:

"2017-07-15 11:19:52.059863 mon.b mon.1 172.21.15.191:6789/0 62 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 1401867 2017-07-15 03:25:23 2017-07-15 10:38:39 2017-07-15 11:46:38 1:07:59 0:15:54 0:52:05 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/auto-repair.yaml} 4
Failure Reason:

"2017-07-15 11:36:12.378159 mon.a mon.0 172.21.15.33:6789/0 207 : cluster [ERR] Health check failed: 4 mds daemons down (MDS_FAILED)" in cluster log

fail 1401870 2017-07-15 03:25:24 2017-07-15 10:40:39 2017-07-15 11:54:39 1:14:00 1:02:28 0:11:32 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/default.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-07-15 11:05:47.812659 mon.a mon.0 172.21.15.91:6789/0 283 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log

fail 1401873 2017-07-15 03:25:24 2017-07-15 10:40:49 2017-07-15 11:24:48 0:43:59 0:33:56 0:10:03 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-07-15 10:56:02.060274 mon.b mon.0 172.21.15.31:6789/0 200 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum b,c" in cluster log

fail 1401876 2017-07-15 03:25:25 2017-07-15 10:41:36 2017-07-15 11:55:35 1:13:59 0:12:53 1:01:06 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/backtrace.yaml} 4
Failure Reason:

"2017-07-15 11:49:06.919693 mon.a mon.0 172.21.15.143:6789/0 186 : cluster [ERR] Health check failed: 2 mds daemons down (MDS_FAILED)" in cluster log

fail 1401879 2017-07-15 03:25:26 2017-07-15 10:42:31 2017-07-15 11:38:31 0:56:00 0:37:15 0:18:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-07-15 11:16:20.480672 mon.b mon.1 172.21.15.90:6789/0 464 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 1401882 2017-07-15 03:25:26 2017-07-15 10:44:28 2017-07-15 11:08:27 0:23:59 0:20:07 0:03:52 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_o_trunc.yaml} 3
fail 1401885 2017-07-15 03:25:27 2017-07-15 10:44:42 2017-07-15 11:22:43 0:38:01 0:11:01 0:27:00 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/client-limits.yaml} 4
Failure Reason:

Test failure: test_client_pin (tasks.cephfs.test_client_limits.TestClientLimits)

dead 1401888 2017-07-15 03:25:28 2017-07-15 10:44:41 2017-07-15 11:42:40 0:57:59 0:33:15 0:24:44 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

SSH connection to smithi168 was lost: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

fail 1401891 2017-07-15 03:25:28 2017-07-15 10:46:55 2017-07-15 11:34:54 0:47:59 0:26:24 0:21:35 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/mds.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-07-15 11:13:10.764468 mon.b mon.0 172.21.15.129:6789/0 299 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED)" in cluster log

fail 1401893 2017-07-15 03:25:29 2017-07-15 10:47:05 2017-07-15 11:11:01 0:23:56 0:14:29 0:09:27 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/client-recovery.yaml} 4
Failure Reason:

Test failure: test_network_death (tasks.cephfs.test_client_recovery.TestClientNetworkRecovery)

pass 1401896 2017-07-15 03:25:30 2017-07-15 10:48:43 2017-07-15 11:50:41 1:01:58 0:52:45 0:09:13 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
fail 1401899 2017-07-15 03:25:31 2017-07-15 10:49:27 2017-07-15 11:47:27 0:58:00 0:45:45 0:12:15 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-07-15 11:09:00.581792 mon.b mon.0 172.21.15.22:6789/0 435 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 1401902 2017-07-15 03:25:31 2017-07-15 10:50:45 2017-07-15 11:14:45 0:24:00 0:13:41 0:10:19 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/config-commands.yaml} 4
Failure Reason:

"2017-07-15 11:04:30.854154 mon.a mon.0 172.21.15.172:6789/0 194 : cluster [ERR] Health check failed: 3 mds daemons down (MDS_FAILED)" in cluster log

pass 1401905 2017-07-15 03:25:32 2017-07-15 10:52:28 2017-07-15 11:16:26 0:23:58 0:13:39 0:10:19 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1401908 2017-07-15 03:25:33 2017-07-15 10:54:32 2017-07-15 11:24:30 0:29:58 0:24:18 0:05:40 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1401911 2017-07-15 03:25:33 2017-07-15 10:54:35 2017-07-15 11:56:35 1:02:00 0:34:38 0:27:22 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/damage.yaml} 4
Failure Reason:

"2017-07-15 11:35:19.609996 mon.a mon.0 172.21.15.172:6789/0 202 : cluster [ERR] Health check failed: 4 mds daemons down (MDS_FAILED)" in cluster log

fail 1401912 2017-07-15 03:25:34 2017-07-15 10:54:38 2017-07-15 11:50:36 0:55:58 0:47:39 0:08:19 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/mon.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-07-15 11:06:40.684725 mon.a mon.1 172.21.15.135:6789/0 147 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 1401913 2017-07-15 03:25:35 2017-07-15 10:54:41 2017-07-15 11:22:40 0:27:59 0:13:15 0:14:44 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
Failure Reason:

SELinux denials found on ubuntu@smithi003.front.sepia.ceph.com: ['type=AVC msg=audit(1500117153.351:3893): avc: denied { getattr } for pid=27552 comm="ceph-osd" path="/dev/nvme0n1p2" dev="devtmpfs" ino=61754 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:nvme_device_t:s0 tclass=blk_file permissive=1']

fail 1401914 2017-07-15 03:25:35 2017-07-15 10:54:41 2017-07-15 11:54:40 0:59:59 0:31:01 0:28:58 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/data-scan.yaml} 4
Failure Reason:

Test failure: test_rebuild_simple_altpool (tasks.cephfs.test_data_scan.TestDataScan)

pass 1401915 2017-07-15 03:25:36 2017-07-15 10:54:43 2017-07-15 11:20:40 0:25:57 0:13:07 0:12:50 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1401916 2017-07-15 03:25:37 2017-07-15 10:56:32 2017-07-15 11:44:31 0:47:59 0:35:12 0:12:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 1401917 2017-07-15 03:25:37 2017-07-15 10:56:32 2017-07-15 11:38:31 0:41:59 0:12:43 0:29:16 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/failover.yaml} 4
Failure Reason:

Test failure: test_simple (tasks.cephfs.test_failover.TestFailover)

pass 1401918 2017-07-15 03:25:38 2017-07-15 10:56:33 2017-07-15 11:20:31 0:23:58 0:10:07 0:13:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1401919 2017-07-15 03:25:39 2017-07-15 10:56:32 2017-07-15 11:50:31 0:53:59 0:11:15 0:42:44 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1401920 2017-07-15 03:25:39 2017-07-15 10:56:38 2017-07-15 12:22:34 1:25:56 0:54:58 0:30:58 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore/filestore-xfs.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
Failure Reason:

"2017-07-15 11:32:19.017854 mon.a mon.1 172.21.15.28:6789/0 2 : cluster [WRN] message from mon.0 was stamped 1.783446s in the future, clocks not synchronized" in cluster log

fail 1401921 2017-07-15 03:25:40 2017-07-15 10:56:40 2017-07-15 12:36:44 1:40:04 0:17:17 1:22:47 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/forward-scrub.yaml} 4
Failure Reason:

"2017-07-15 12:24:30.156004 mon.a mon.0 172.21.15.135:6789/0 198 : cluster [ERR] Health check failed: 3 mds daemons down (MDS_FAILED)" in cluster log

fail 1401922 2017-07-15 03:25:41 2017-07-15 10:58:59 2017-07-15 11:38:57 0:39:58 0:32:58 0:07:00 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/default.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-07-15 11:09:56.488323 mon.b mon.0 172.21.15.57:6789/0 221 : cluster [ERR] Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL)" in cluster log

pass 1401923 2017-07-15 03:25:41 2017-07-15 11:00:39 2017-07-15 11:50:39 0:50:00 0:38:54 0:11:06 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
dead 1401924 2017-07-15 03:25:42 2017-07-15 11:00:39 2017-07-15 23:07:09 12:06:30 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/journal-repair.yaml} 4
fail 1401925 2017-07-15 03:25:43 2017-07-15 11:00:39 2017-07-15 11:38:39 0:38:00 0:30:35 0:07:25 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-07-15 11:11:11.047690 mon.b mon.0 172.21.15.110:6789/0 381 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 1401926 2017-07-15 03:25:43 2017-07-15 11:02:41 2017-07-15 11:38:40 0:35:59 0:24:20 0:11:39 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_o_trunc.yaml} 3
fail 1401927 2017-07-15 03:25:44 2017-07-15 11:02:44 2017-07-15 11:38:40 0:35:56 0:12:34 0:23:22 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/mds-flush.yaml} 4
Failure Reason:

"2017-07-15 11:29:24.786088 mon.a mon.0 172.21.15.60:6789/0 193 : cluster [ERR] Health check failed: 2 mds daemons down (MDS_FAILED)" in cluster log

dead 1401928 2017-07-15 03:25:45 2017-07-15 11:02:42 2017-07-15 11:40:41 0:37:59 0:28:35 0:09:24 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

SSH connection to smithi178 was lost: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

fail 1401929 2017-07-15 03:25:45 2017-07-15 11:02:41 2017-07-15 11:52:41 0:50:00 0:44:16 0:05:44 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/mds.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-07-15 11:15:39.731191 mon.a mon.0 172.21.15.64:6789/0 768 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED)" in cluster log

fail 1401930 2017-07-15 03:25:46 2017-07-15 11:02:48 2017-07-15 11:24:49 0:22:01 0:10:29 0:11:32 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/mds-full.yaml} 4
Failure Reason:

Command failed on smithi020 with status 1: 'sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmpHI6oAw'

pass 1401931 2017-07-15 03:25:47 2017-07-15 11:02:55 2017-07-15 12:12:55 1:10:00 0:52:01 0:17:59 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
pass 1401932 2017-07-15 03:25:47 2017-07-15 11:03:21 2017-07-15 11:39:18 0:35:57 0:26:28 0:09:29 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
fail 1401933 2017-07-15 03:25:48 2017-07-15 11:03:33 2017-07-15 11:27:32 0:23:59 0:12:26 0:11:33 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/pool-perm.yaml} 4
Failure Reason:

"2017-07-15 11:19:33.778656 mon.a mon.0 172.21.15.17:6789/0 191 : cluster [ERR] Health check failed: 2 mds daemons down (MDS_FAILED)" in cluster log

pass 1401934 2017-07-15 03:25:49 2017-07-15 11:04:32 2017-07-15 11:28:31 0:23:59 0:13:16 0:10:43 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1401935 2017-07-15 03:25:49 2017-07-15 11:04:34 2017-07-15 11:32:32 0:27:58 0:16:01 0:11:57 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1401936 2017-07-15 03:25:50 2017-07-15 11:04:32 2017-07-15 12:44:33 1:40:01 0:16:34 1:23:27 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/sessionmap.yaml} 4
Failure Reason:

"2017-07-15 12:32:38.171185 mon.a mon.0 172.21.15.26:6789/0 191 : cluster [ERR] Health check failed: 4 mds daemons down (MDS_FAILED)" in cluster log

fail 1401937 2017-07-15 03:25:51 2017-07-15 11:04:32 2017-07-15 11:52:32 0:48:00 0:26:34 0:21:26 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/mon.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-07-15 11:29:29.148786 mon.a mon.0 172.21.15.45:6789/0 185 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

pass 1401938 2017-07-15 03:25:51 2017-07-15 11:04:34 2017-07-15 11:20:33 0:15:59 0:08:49 0:07:10 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
fail 1401939 2017-07-15 03:25:52 2017-07-15 11:06:50 2017-07-15 11:58:49 0:51:59 0:32:55 0:19:04 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/strays.yaml} 4
Failure Reason:

Test failure: test_replicated_delete_speed (tasks.cephfs.test_strays.TestStrays)

pass 1401940 2017-07-15 03:25:53 2017-07-15 11:08:30 2017-07-15 12:04:28 0:55:58 0:24:05 0:31:53 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1401941 2017-07-15 03:25:53 2017-07-15 11:08:36 2017-07-15 11:50:35 0:41:59 0:29:36 0:12:23 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 1401942 2017-07-15 03:25:54 2017-07-15 11:08:29 2017-07-15 12:22:29 1:14:00 0:23:24 0:50:36 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/volume-client.yaml} 4
Failure Reason:

"2017-07-15 12:04:31.318456 mon.a mon.0 172.21.15.67:6789/0 181 : cluster [ERR] Health check failed: 2 mds daemons down (MDS_FAILED)" in cluster log

pass 1401943 2017-07-15 03:25:55 2017-07-15 11:08:35 2017-07-15 11:32:34 0:23:59 0:09:34 0:14:25 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1401944 2017-07-15 03:25:55 2017-07-15 11:08:37 2017-07-15 11:48:37 0:40:00 0:11:18 0:28:42 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1401945 2017-07-15 03:25:56 2017-07-15 11:10:53 2017-07-15 12:34:52 1:23:59 0:45:52 0:38:07 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore/filestore-xfs.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
Failure Reason:

"2017-07-15 12:10:29.669227 mon.b mon.0 172.21.15.23:6789/0 157 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum b,c" in cluster log

fail 1401946 2017-07-15 03:25:57 2017-07-15 11:11:03 2017-07-15 11:43:03 0:32:00 0:13:15 0:18:45 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/auto-repair.yaml} 4
Failure Reason:

"2017-07-15 11:32:38.364447 mon.a mon.0 172.21.15.200:6789/0 205 : cluster [ERR] Health check failed: 4 mds daemons down (MDS_FAILED)" in cluster log

fail 1401947 2017-07-15 03:25:57 2017-07-15 11:11:11 2017-07-15 11:57:12 0:46:01 0:32:18 0:13:43 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/default.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-07-15 11:24:58.759553 mon.b mon.0 172.21.15.71:6789/0 188 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log

pass 1401948 2017-07-15 03:25:58 2017-07-15 11:14:43 2017-07-15 12:08:38 0:53:55 0:28:23 0:25:32 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
fail 1401949 2017-07-15 03:25:59 2017-07-15 11:14:42 2017-07-15 12:44:42 1:30:00 0:09:33 1:20:27 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/backtrace.yaml} 4
Failure Reason:

"2017-07-15 12:36:40.293157 mon.a mon.0 172.21.15.174:6789/0 190 : cluster [ERR] Health check failed: 2 mds daemons down (MDS_FAILED)" in cluster log

fail 1401950 2017-07-15 03:25:59 2017-07-15 11:15:01 2017-07-15 12:15:03 1:00:02 0:41:40 0:18:22 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-07-15 11:38:31.195177 mon.b mon.0 172.21.15.85:6789/0 212 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum b,c" in cluster log

pass 1401951 2017-07-15 03:26:00 2017-07-15 11:15:49 2017-07-15 11:43:47 0:27:58 0:21:14 0:06:44 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_o_trunc.yaml} 3
fail 1401952 2017-07-15 03:26:01 2017-07-15 11:16:27 2017-07-15 12:08:27 0:52:00 0:13:52 0:38:08 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/client-limits.yaml} 4
Failure Reason:

Test failure: test_client_pin (tasks.cephfs.test_client_limits.TestClientLimits)

dead 1401953 2017-07-15 03:26:01 2017-07-15 11:17:05 2017-07-15 12:01:05 0:44:00 0:29:13 0:14:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

SSH connection to smithi076 was lost: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

fail 1401954 2017-07-15 03:26:02 2017-07-15 11:17:00 2017-07-15 11:40:59 0:23:59 0:12:41 0:11:18 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/mds.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-07-15 11:30:44.228059 mon.b mon.0 172.21.15.165:6789/0 291 : cluster [WRN] Health check failed: 1 filesystem is degraded (FS_DEGRADED)" in cluster log

fail 1401955 2017-07-15 03:26:03 2017-07-15 11:17:59 2017-07-15 11:35:57 0:17:58 0:11:49 0:06:09 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/client-recovery.yaml} 4
Failure Reason:

Test failure: test_network_death (tasks.cephfs.test_client_recovery.TestClientNetworkRecovery)

pass 1401956 2017-07-15 03:26:03 2017-07-15 11:18:36 2017-07-15 12:24:36 1:06:00 0:46:36 0:19:24 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
pass 1401957 2017-07-15 03:26:04 2017-07-15 11:20:32 2017-07-15 12:42:32 1:22:00 0:47:14 0:34:46 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
fail 1401958 2017-07-15 03:26:05 2017-07-15 11:20:39 2017-07-15 11:50:32 0:29:53 0:14:14 0:15:39 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/config-commands.yaml} 4
Failure Reason:

"2017-07-15 11:41:09.638579 mon.a mon.0 172.21.15.179:6789/0 201 : cluster [ERR] Health check failed: 4 mds daemons down (MDS_FAILED)" in cluster log

pass 1401959 2017-07-15 03:26:05 2017-07-15 11:20:36 2017-07-15 11:34:33 0:13:57 0:11:10 0:02:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1401960 2017-07-15 03:26:06 2017-07-15 11:20:35 2017-07-15 12:04:34 0:43:59 0:23:59 0:20:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1401961 2017-07-15 03:26:07 2017-07-15 11:20:44 2017-07-15 13:08:43 1:47:59 0:26:57 1:21:02 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/damage.yaml} 4
Failure Reason:

"2017-07-15 12:46:29.961270 mon.a mon.0 172.21.15.2:6789/0 190 : cluster [ERR] Health check failed: 2 mds daemons down (MDS_FAILED)" in cluster log

fail 1401962 2017-07-15 03:26:07 2017-07-15 11:21:52 2017-07-15 12:13:52 0:52:00 0:29:15 0:22:45 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/mon.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-07-15 11:49:17.317923 mon.b mon.0 172.21.15.36:6789/0 245 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum b,c" in cluster log

pass 1401963 2017-07-15 03:26:08 2017-07-15 11:22:30 2017-07-15 11:40:28 0:17:58 0:10:30 0:07:28 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
fail 1401964 2017-07-15 03:26:09 2017-07-15 11:22:32 2017-07-15 16:10:37 4:48:05 0:28:36 4:19:29 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/data-scan.yaml} 4
Failure Reason:

Test failure: test_rebuild_simple_altpool (tasks.cephfs.test_data_scan.TestDataScan)

pass 1401965 2017-07-15 03:26:09 2017-07-15 11:22:44 2017-07-15 12:02:41 0:39:57 0:11:45 0:28:12 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1401966 2017-07-15 03:26:10 2017-07-15 11:22:45 2017-07-15 12:26:45 1:04:00 0:44:08 0:19:52 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
fail 1401967 2017-07-15 03:26:11 2017-07-15 11:23:16 2017-07-15 14:15:16 2:52:00 0:14:38 2:37:22 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/failover.yaml} 4
Failure Reason:

Test failure: test_simple (tasks.cephfs.test_failover.TestFailover)

pass 1401968 2017-07-15 03:26:11 2017-07-15 11:24:40 2017-07-15 11:46:38 0:21:58 0:10:07 0:11:51 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1401969 2017-07-15 03:26:12 2017-07-15 11:24:42 2017-07-15 11:42:38 0:17:56 0:10:49 0:07:07 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1401970 2017-07-15 03:26:13 2017-07-15 11:24:56 2017-07-15 12:24:50 0:59:54 0:43:51 0:16:03 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore/bluestore.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
Failure Reason:

"2017-07-15 11:47:28.781186 mon.b mon.1 172.21.15.84:6789/0 117 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 1401971 2017-07-15 03:26:13 2017-07-15 11:24:51 2017-07-15 12:10:50 0:45:59 0:16:41 0:29:18 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/forward-scrub.yaml} 4
Failure Reason:

"2017-07-15 11:58:00.403759 mon.a mon.0 172.21.15.17:6789/0 196 : cluster [ERR] Health check failed: 4 mds daemons down (MDS_FAILED)" in cluster log

fail 1401972 2017-07-15 03:26:14 2017-07-15 11:24:52 2017-07-15 11:58:50 0:33:58 0:15:45 0:18:13 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/default.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-07-15 11:47:06.100229 mon.a mon.0 172.21.15.2:6789/0 152 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log

pass 1401973 2017-07-15 03:26:14 2017-07-15 11:27:42 2017-07-15 12:17:42 0:50:00 0:47:10 0:02:50 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
dead 1401974 2017-07-15 03:26:15 2017-07-15 11:28:33 2017-07-15 23:35:28 12:06:55 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/journal-repair.yaml} 4
fail 1401975 2017-07-15 03:26:16 2017-07-15 11:28:33 2017-07-15 12:14:32 0:45:59 0:36:15 0:09:44 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-07-15 11:43:19.023236 mon.b mon.1 172.21.15.48:6789/0 97 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 1401976 2017-07-15 03:26:16 2017-07-15 11:30:37 2017-07-15 12:04:36 0:33:59 0:26:09 0:07:50 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_o_trunc.yaml} 3
fail 1401977 2017-07-15 03:26:17 2017-07-15 11:32:43 2017-07-15 12:02:43 0:30:00 0:09:51 0:20:09 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/mds-flush.yaml} 4
Failure Reason:

"2017-07-15 11:53:37.276383 mon.a mon.0 172.21.15.185:6789/0 184 : cluster [ERR] Health check failed: 2 mds daemons down (MDS_FAILED)" in cluster log

dead 1401978 2017-07-15 03:26:18 2017-07-15 11:32:44 2017-07-15 12:38:44 1:06:00 0:31:18 0:34:42 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

SSH connection to smithi054 was lost: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

dead 1401979 2017-07-15 03:26:18 2017-07-15 11:34:44 2017-07-15 23:41:06 12:06:22 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/mds.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1401980 2017-07-15 03:26:19 2017-07-15 11:34:57 2017-07-15 11:52:55 0:17:58 0:09:50 0:08:08 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/mds-full.yaml} 4
Failure Reason:

Command failed on smithi110 with status 1: 'sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmpAu9HT_'

pass 1401981 2017-07-15 03:26:20 2017-07-15 11:36:10 2017-07-15 12:42:08 1:05:58 0:49:58 0:16:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
dead 1401982 2017-07-15 03:26:20 2017-07-15 11:36:57 2017-07-15 23:43:42 12:06:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
fail 1401983 2017-07-15 03:26:21 2017-07-15 11:38:43 2017-07-15 12:00:42 0:21:59 0:13:08 0:08:51 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/pool-perm.yaml} 4
Failure Reason:

"2017-07-15 11:52:32.804811 mon.a mon.0 172.21.15.33:6789/0 197 : cluster [ERR] Health check failed: 4 mds daemons down (MDS_FAILED)" in cluster log

pass 1401984 2017-07-15 03:26:22 2017-07-15 11:38:44 2017-07-15 11:50:43 0:11:59 0:10:18 0:01:41 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1401985 2017-07-15 03:26:22 2017-07-15 11:38:43 2017-07-15 11:54:41 0:15:58 0:15:43 0:00:15 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1401986 2017-07-15 03:26:23 2017-07-15 11:38:43 2017-07-15 12:14:42 0:35:59 0:15:37 0:20:22 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/sessionmap.yaml} 4
Failure Reason:

"2017-07-15 12:02:53.460998 mon.a mon.0 172.21.15.4:6789/0 205 : cluster [ERR] Health check failed: 4 mds daemons down (MDS_FAILED)" in cluster log

fail 1401987 2017-07-15 03:26:24 2017-07-15 11:38:44 2017-07-15 12:04:41 0:25:57 0:15:22 0:10:35 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/mon.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-07-15 11:53:54.512292 mon.a mon.0 172.21.15.3:6789/0 218 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 1401988 2017-07-15 03:26:25 2017-07-15 11:38:43 2017-07-15 11:54:44 0:16:01 0:10:37 0:05:24 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
fail 1401989 2017-07-15 03:26:25 2017-07-15 11:38:43 2017-07-15 12:24:42 0:45:59 0:24:29 0:21:30 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/strays.yaml} 4
Failure Reason:

Test failure: test_migration_on_shutdown (tasks.cephfs.test_strays.TestStrays)

pass 1401990 2017-07-15 03:26:26 2017-07-15 11:38:58 2017-07-15 12:20:58 0:42:00 0:28:08 0:13:52 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
fail 1401991 2017-07-15 03:26:26 2017-07-15 11:39:21 2017-07-15 12:35:21 0:56:00 0:32:25 0:23:35 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
Failure Reason:

"2017-07-15 12:07:20.080559 mon.a mon.0 172.21.15.2:6789/0 253 : cluster [WRN] Health check failed: Reduced data availability: 8 pgs inactive (PG_AVAILABILITY)" in cluster log

fail 1401992 2017-07-15 03:26:27 2017-07-15 11:40:39 2017-07-15 13:18:39 1:38:00 0:30:21 1:07:39 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/volume-client.yaml} 4
Failure Reason:

"2017-07-15 12:56:08.793623 mon.a mon.0 172.21.15.75:6789/0 191 : cluster [ERR] Health check failed: 2 mds daemons down (MDS_FAILED)" in cluster log

pass 1401993 2017-07-15 03:26:28 2017-07-15 11:40:43 2017-07-15 12:00:41 0:19:58 0:08:02 0:11:56 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_trivial_sync.yaml} 3