Pulpito

pdonnell-2017-09-29_23:20:37-kcephfs-wip-pdonnell-testing-20170929-testing-basic-smithi

User Scheduled Started Updated Runtime Suite Branch Machine Type Revision Pass Fail Dead
pdonnell 2017-09-29 23:20:37 2017-09-30 01:43:07 2017-09-30 15:45:03 14:01:56 kcephfs wip-pdonnell-testing-20170929 smithi a5db74a 84 14 2

Average wait time: 0:32:09

See other runs of suite 'kcephfs' on branch 'wip-pdonnell-testing-20170929'?

See other runs on branch 'wip-pdonnell-testing-20170929'?

See other runs scheduled on 2017-09-29?

See detail view?

Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Description Nodes
pass 1687883 2017-09-29 23:20:40 2017-09-30 01:43:07 2017-09-30 02:05:06 0:21:59 0:11:03 0:10:56 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 1687886 2017-09-29 23:20:40 2017-09-30 01:43:09 2017-09-30 02:47:09 1:04:00 0:46:06 0:17:54 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore/bluestore.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
pass 1687889 2017-09-29 23:20:41 2017-09-30 02:34:41 2017-09-30 03:22:41 0:48:00 0:12:44 0:35:16 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 1687892 2017-09-29 23:20:42 2017-09-30 02:35:09 2017-09-30 03:05:08 0:29:59 0:20:56 0:09:03 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
pass 1687895 2017-09-29 23:20:42 2017-09-30 02:35:13 2017-09-30 03:11:13 0:36:00 0:24:24 0:11:36 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 1687898 2017-09-29 23:20:43 2017-09-30 02:37:37 2017-09-30 02:57:34 0:19:57 0:08:29 0:11:28 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
fail 1687901 2017-09-29 23:20:44 2017-09-30 02:39:17 2017-09-30 03:19:17 0:40:00 0:30:58 0:09:02 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-09-30 02:53:28.809822 mon.b mon.0 172.21.15.59:6789/0 117 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum b,c" in cluster log

pass 1687904 2017-09-29 23:20:44 2017-09-30 02:39:18 2017-09-30 03:03:17 0:23:59 0:17:28 0:06:31 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1687907 2017-09-29 23:20:45 2017-09-30 02:41:16 2017-09-30 05:47:19 3:06:03 0:17:52 2:48:11 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
pass 1687910 2017-09-29 23:20:46 2017-09-30 02:41:17 2017-09-30 03:27:17 0:46:00 0:43:20 0:02:40 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 1687913 2017-09-29 23:20:46 2017-09-30 02:43:15 2017-09-30 03:57:16 1:14:01 0:23:33 0:50:28 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 1687916 2017-09-29 23:20:47 2017-09-30 02:43:28 2017-09-30 04:43:30 2:00:02 0:39:13 1:20:49 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
pass 1687919 2017-09-29 23:20:47 2017-09-30 02:45:12 2017-09-30 03:43:12 0:58:00 0:36:00 0:22:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
pass 1687922 2017-09-29 23:20:48 2017-09-30 02:47:18 2017-09-30 03:15:17 0:27:59 0:19:51 0:08:08 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1687925 2017-09-29 23:20:49 2017-09-30 02:47:18 2017-09-30 03:37:18 0:50:00 0:11:50 0:38:10 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 1687928 2017-09-29 23:20:49 2017-09-30 02:49:01 2017-09-30 03:07:01 0:18:00 0:09:13 0:08:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1687931 2017-09-29 23:20:50 2017-09-30 02:49:17 2017-09-30 03:17:16 0:27:59 0:22:01 0:05:58 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
fail 1687934 2017-09-29 23:20:51 2017-09-30 02:51:13 2017-09-30 04:17:14 1:26:01 0:21:43 1:04:18 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/damage.yaml whitelist_health.yaml} 4
Failure Reason:

"2017-09-30 04:05:03.126164 mon.a mon.0 172.21.15.91:6789/0 1390 : cluster [WRN] Manager daemon x is unresponsive. No standby daemons available." in cluster log

fail 1687937 2017-09-29 23:20:51 2017-09-30 02:55:14 2017-09-30 03:25:14 0:30:00 0:21:44 0:08:16 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-09-30 03:04:49.386721 mon.a mon.0 172.21.15.82:6789/0 137 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 1687940 2017-09-29 23:20:52 2017-09-30 02:55:17 2017-09-30 03:45:17 0:50:00 0:11:50 0:38:10 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1687943 2017-09-29 23:20:52 2017-09-30 02:57:10 2017-09-30 03:23:10 0:26:00 0:22:09 0:03:51 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 1687946 2017-09-29 23:20:53 2017-09-30 02:59:28 2017-09-30 03:25:28 0:26:00 0:09:40 0:16:20 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1687949 2017-09-29 23:20:54 2017-09-30 03:01:21 2017-09-30 03:41:21 0:40:00 0:11:51 0:28:09 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 1687952 2017-09-29 23:20:54 2017-09-30 03:03:15 2017-09-30 04:55:17 1:52:02 0:23:13 1:28:49 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 1687955 2017-09-29 23:20:55 2017-09-30 03:05:00 2017-09-30 03:39:00 0:34:00 0:09:44 0:24:16 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1687958 2017-09-29 23:20:55 2017-09-30 03:05:10 2017-09-30 03:25:10 0:20:00 0:10:25 0:09:35 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 1687961 2017-09-29 23:20:56 2017-09-30 03:05:21 2017-09-30 03:41:21 0:36:00 0:30:48 0:05:12 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore/filestore-xfs.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
pass 1687963 2017-09-29 23:20:57 2017-09-30 03:07:14 2017-09-30 03:25:14 0:18:00 0:14:37 0:03:23 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
pass 1687966 2017-09-29 23:20:57 2017-09-30 03:09:18 2017-09-30 03:43:18 0:34:00 0:25:34 0:08:26 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 1687970 2017-09-29 23:20:58 2017-09-30 03:11:15 2017-09-30 03:47:15 0:36:00 0:26:01 0:09:59 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 1687973 2017-09-29 23:20:58 2017-09-30 03:15:19 2017-09-30 04:53:21 1:38:02 0:18:06 1:19:56 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
pass 1687974 2017-09-29 23:20:59 2017-09-30 03:15:19 2017-09-30 04:07:19 0:52:00 0:36:00 0:16:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_misc.yaml} 3
pass 1687977 2017-09-29 23:21:00 2017-09-30 03:15:29 2017-09-30 04:01:29 0:46:00 0:20:15 0:25:45 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1687980 2017-09-29 23:21:00 2017-09-30 03:17:17 2017-09-30 03:57:17 0:40:00 0:09:02 0:30:58 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
pass 1687984 2017-09-29 23:21:01 2017-09-30 03:19:26 2017-09-30 04:03:26 0:44:00 0:36:57 0:07:03 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 1687987 2017-09-29 23:21:02 2017-09-30 03:21:22 2017-09-30 03:45:21 0:23:59 0:21:17 0:02:42 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1687990 2017-09-29 23:21:02 2017-09-30 03:22:51 2017-09-30 05:14:52 1:52:01 0:06:25 1:45:36 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi177 with status 1: 'sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmpYhhGyV'

pass 1687994 2017-09-29 23:21:03 2017-09-30 03:25:20 2017-09-30 04:21:21 0:56:01 0:39:18 0:16:43 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
pass 1687996 2017-09-29 23:21:03 2017-09-30 03:25:30 2017-09-30 03:59:29 0:33:59 0:23:33 0:10:26 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1687999 2017-09-29 23:21:04 2017-09-30 03:27:15 2017-09-30 04:29:15 1:02:00 0:12:04 0:49:56 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 1688002 2017-09-29 23:21:05 2017-09-30 03:27:18 2017-09-30 03:49:18 0:22:00 0:09:14 0:12:46 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1688004 2017-09-29 23:21:05 2017-09-30 03:29:21 2017-09-30 04:23:21 0:54:00 0:20:00 0:34:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 1688007 2017-09-29 23:21:06 2017-09-30 03:31:23 2017-09-30 05:41:25 2:10:02 0:11:19 1:58:43 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 1688010 2017-09-29 23:21:07 2017-09-30 03:31:46 2017-09-30 04:09:50 0:38:04 0:26:33 0:11:31 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-09-30 03:48:20.406369 mon.a mon.0 172.21.15.10:6789/0 153 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 1688013 2017-09-29 23:21:07 2017-09-30 03:33:24 2017-09-30 03:53:23 0:19:59 0:11:52 0:08:07 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
dead 1688016 2017-09-29 23:21:08 2017-09-30 03:35:29 2017-09-30 04:11:28 0:35:59 0:05:11 0:30:48 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/strays.yaml whitelist_health.yaml} 4
Failure Reason:

'NoneType' object has no attribute 'open_sftp_client'

pass 1688019 2017-09-29 23:21:08 2017-09-30 03:37:19 2017-09-30 04:07:18 0:29:59 0:25:04 0:04:55 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1688022 2017-09-29 23:21:09 2017-09-30 03:37:20 2017-09-30 04:03:19 0:25:59 0:11:44 0:14:15 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
dead 1688025 2017-09-29 23:21:10 2017-09-30 03:39:01 2017-09-30 15:45:03 12:06:02 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
pass 1688028 2017-09-29 23:21:10 2017-09-30 03:39:21 2017-09-30 04:03:20 0:23:59 0:09:43 0:14:16 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1688031 2017-09-29 23:21:11 2017-09-30 03:41:19 2017-09-30 03:53:18 0:11:59 0:08:52 0:03:07 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_direct_io.yaml} 3
pass 1688034 2017-09-29 23:21:11 2017-09-30 03:41:38 2017-09-30 05:23:39 1:42:01 0:42:58 0:59:03 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore/filestore-xfs.yaml tasks/kernel_cfuse_workunits_dbench_iozone.yaml} 4
pass 1688037 2017-09-29 23:21:12 2017-09-30 03:43:18 2017-09-30 04:35:18 0:52:00 0:11:20 0:40:40 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/auto-repair.yaml whitelist_health.yaml} 4
pass 1688040 2017-09-29 23:21:13 2017-09-30 03:43:23 2017-09-30 04:15:22 0:31:59 0:26:11 0:05:48 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1688043 2017-09-29 23:21:13 2017-09-30 03:45:27 2017-09-30 04:21:27 0:36:00 0:25:36 0:10:24 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
Failure Reason:

"2017-09-30 04:09:51.901540 mon.b mon.0 172.21.15.40:6789/0 120 : cluster [WRN] overall HEALTH_WARN 1/3 mons down, quorum b,c" in cluster log

pass 1688046 2017-09-29 23:21:14 2017-09-30 03:45:31 2017-09-30 04:55:31 1:10:00 0:10:57 0:59:03 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/backtrace.yaml whitelist_health.yaml} 4
fail 1688049 2017-09-29 23:21:15 2017-09-30 03:47:30 2017-09-30 04:29:30 0:42:00 0:29:50 0:12:10 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_misc.yaml} 3
Failure Reason:

"2017-09-30 04:06:25.587851 mon.b mon.0 172.21.15.41:6789/0 250 : cluster [WRN] overall HEALTH_WARN Reduced data availability: 1 pg stale; Degraded data redundancy: 12143/69890 objects degraded (17.374%), 6 pgs unclean, 6 pgs degraded, 4 pgs undersized; 1/3 mons down, quorum b,c" in cluster log

fail 1688052 2017-09-29 23:21:15 2017-09-30 03:49:03 2017-09-30 04:19:03 0:30:00 0:21:11 0:08:49 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_o_trunc.yaml} 3
Failure Reason:

"2017-09-30 04:04:31.760820 mon.a mon.0 172.21.15.28:6789/0 147 : cluster [WRN] Health check update: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 1688055 2017-09-29 23:21:16 2017-09-30 03:49:26 2017-09-30 05:13:27 1:24:01 0:18:01 1:06:00 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/client-limits.yaml whitelist_health.yaml} 4
pass 1688058 2017-09-29 23:21:16 2017-09-30 03:51:22 2017-09-30 04:49:23 0:58:01 0:41:25 0:16:36 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_snaps.yaml} 3
pass 1688061 2017-09-29 23:21:17 2017-09-30 03:51:48 2017-09-30 04:03:48 0:12:00 0:10:32 0:01:28 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 1688064 2017-09-29 23:21:18 2017-09-30 03:53:29 2017-09-30 05:09:29 1:16:00 0:39:47 0:36:13 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/client-recovery.yaml whitelist_health.yaml} 4
pass 1688067 2017-09-29 23:21:18 2017-09-30 03:53:29 2017-09-30 05:05:29 1:12:00 0:35:40 0:36:20 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
fail 1688070 2017-09-29 23:21:19 2017-09-30 03:55:17 2017-09-30 04:19:16 0:23:59 0:18:59 0:05:00 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-09-30 04:06:09.048121 mon.b mon.1 172.21.15.204:6789/0 100 : cluster [WRN] Health check update: Degraded data redundancy: 1060/6646 objects degraded (15.949%), 2 pgs unclean, 2 pgs degraded, 1 pg undersized (PG_DEGRADED)" in cluster log

pass 1688073 2017-09-29 23:21:19 2017-09-30 03:55:24 2017-09-30 04:55:24 1:00:00 0:10:46 0:49:14 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/config-commands.yaml whitelist_health.yaml} 4
pass 1688076 2017-09-29 23:21:20 2017-09-30 03:57:19 2017-09-30 04:15:18 0:17:59 0:09:03 0:08:56 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1688079 2017-09-29 23:21:21 2017-09-30 03:59:29 2017-09-30 04:35:29 0:36:00 0:22:23 0:13:37 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 1688082 2017-09-29 23:21:21 2017-09-30 04:01:18 2017-09-30 04:25:18 0:24:00 0:18:51 0:05:09 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/damage.yaml whitelist_health.yaml} 4
fail 1688085 2017-09-29 23:21:22 2017-09-30 04:01:31 2017-09-30 04:53:31 0:52:00 0:24:40 0:27:20 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
Failure Reason:

"2017-09-30 04:29:19.701980 mon.a mon.0 172.21.15.176:6789/0 155 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 1688088 2017-09-29 23:21:22 2017-09-30 04:03:32 2017-09-30 04:43:32 0:40:00 0:08:55 0:31:05 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1688091 2017-09-29 23:21:23 2017-09-30 04:05:33 2017-09-30 04:45:33 0:40:00 0:24:37 0:15:23 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/data-scan.yaml whitelist_health.yaml} 4
pass 1688094 2017-09-29 23:21:24 2017-09-30 04:07:20 2017-09-30 04:31:19 0:23:59 0:09:37 0:14:22 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1688097 2017-09-29 23:21:24 2017-09-30 04:07:31 2017-09-30 04:51:31 0:44:00 0:11:46 0:32:14 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 1688100 2017-09-29 23:21:25 2017-09-30 04:09:51 2017-09-30 05:07:51 0:58:00 0:22:38 0:35:22 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/failover.yaml whitelist_health.yaml} 4
pass 1688103 2017-09-29 23:21:26 2017-09-30 04:11:28 2017-09-30 04:35:28 0:24:00 0:08:18 0:15:42 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_trivial_sync.yaml} 3
pass 1688106 2017-09-29 23:21:26 2017-09-30 04:11:47 2017-09-30 04:23:46 0:11:59 0:08:16 0:03:43 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_direct_io.yaml} 3
fail 1688109 2017-09-29 23:21:27 2017-09-30 04:15:27 2017-09-30 05:31:28 1:16:01 0:34:23 0:41:38 smithi master kcephfs/mixed-clients/{clusters/2-clients.yaml conf.yaml objectstore/bluestore.yaml tasks/kernel_cfuse_workunits_untarbuild_blogbench.yaml} 4
Failure Reason:

"2017-09-30 05:14:42.214099 mon.b mon.1 172.21.15.77:6789/0 96 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 1688112 2017-09-29 23:21:27 2017-09-30 04:17:24 2017-09-30 05:47:25 1:30:01 0:11:43 1:18:18 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/forward-scrub.yaml whitelist_health.yaml} 4
pass 1688115 2017-09-29 23:21:28 2017-09-30 04:19:17 2017-09-30 04:35:17 0:16:00 0:11:20 0:04:40 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/default.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
pass 1688118 2017-09-29 23:21:29 2017-09-30 04:19:37 2017-09-30 04:47:36 0:27:59 0:26:53 0:01:06 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_kernel_untar_build.yaml} 3
pass 1688121 2017-09-29 23:21:29 2017-09-30 04:21:32 2017-09-30 05:15:32 0:54:00 0:19:01 0:34:59 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/journal-repair.yaml whitelist_health.yaml} 4
pass 1688124 2017-09-29 23:21:30 2017-09-30 04:23:30 2017-09-30 04:51:29 0:27:59 0:27:17 0:00:42 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_misc.yaml} 3
pass 1688127 2017-09-29 23:21:30 2017-09-30 04:23:47 2017-09-30 04:45:47 0:22:00 0:18:28 0:03:32 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_o_trunc.yaml} 3
pass 1688130 2017-09-29 23:21:31 2017-09-30 04:25:23 2017-09-30 05:17:24 0:52:01 0:10:20 0:41:41 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/mds-flush.yaml whitelist_health.yaml} 4
fail 1688131 2017-09-29 23:21:32 2017-09-30 04:25:23 2017-09-30 05:49:24 1:24:01 0:40:59 0:43:02 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_snaps.yaml} 3
Failure Reason:

"2017-09-30 05:27:09.030677 mon.a mon.0 172.21.15.54:6789/0 305 : cluster [WRN] daemon mds.a-s is not responding, replacing it as rank 0 with standby daemon mds.a" in cluster log

pass 1688134 2017-09-29 23:21:32 2017-09-30 04:28:44 2017-09-30 05:22:45 0:54:01 0:25:44 0:28:17 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/filestore-xfs.yaml thrashers/mds.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_ffsb.yaml} 3
fail 1688138 2017-09-29 23:21:33 2017-09-30 04:29:28 2017-09-30 06:48:30 2:19:02 0:08:28 2:10:34 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/mds-full.yaml whitelist_health.yaml} 4
Failure Reason:

Command failed on smithi026 with status 1: 'sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmpNL1IRC'

pass 1688141 2017-09-29 23:21:33 2017-09-30 04:29:58 2017-09-30 05:11:58 0:42:00 0:37:07 0:04:53 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_dbench.yaml} 3
pass 1688144 2017-09-29 23:21:34 2017-09-30 04:31:30 2017-09-30 04:59:29 0:27:59 0:23:45 0:04:14 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_ffsb.yaml} 3
pass 1688147 2017-09-29 23:21:35 2017-09-30 04:33:29 2017-09-30 13:19:01 8:45:32 0:09:34 8:35:58 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/pool-perm.yaml whitelist_health.yaml} 4
pass 1688150 2017-09-29 23:21:35 2017-09-30 04:35:28 2017-09-30 05:35:29 1:00:01 0:11:15 0:48:46 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsstress.yaml} 3
pass 1688152 2017-09-29 23:21:36 2017-09-30 04:35:28 2017-09-30 05:01:28 0:26:00 0:16:04 0:09:56 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_fsx.yaml} 3
pass 1688155 2017-09-29 23:21:37 2017-09-30 04:35:30 2017-09-30 05:29:31 0:54:01 0:11:38 0:42:23 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/sessionmap.yaml whitelist_health.yaml} 4
fail 1688159 2017-09-29 23:21:37 2017-09-30 04:41:33 2017-09-30 05:07:33 0:26:00 0:12:17 0:13:43 smithi master kcephfs/thrash/{clusters/fixed-3-cephfs.yaml conf.yaml objectstore/bluestore.yaml thrashers/mon.yaml thrashosds-health.yaml whitelist_health.yaml workloads/kclient_workunit_suites_iozone.yaml} 3
Failure Reason:

"2017-09-30 04:59:06.574878 mon.b mon.1 172.21.15.130:6789/0 32 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 1688162 2017-09-29 23:21:38 2017-09-30 04:43:44 2017-09-30 04:53:42 0:09:58 0:09:19 0:00:39 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_fsync.yaml} 3
pass 1688165 2017-09-29 23:21:38 2017-09-30 04:43:43 2017-09-30 05:27:43 0:44:00 0:35:36 0:08:24 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/filestore-xfs.yaml tasks/strays.yaml whitelist_health.yaml} 4
pass 1688168 2017-09-29 23:21:39 2017-09-30 04:45:44 2017-09-30 05:19:44 0:34:00 0:21:20 0:12:40 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_suites_iozone.yaml} 3
pass 1688171 2017-09-29 23:21:40 2017-09-30 04:45:45 2017-09-30 05:03:44 0:17:59 0:10:12 0:07:47 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/yes.yaml objectstore/bluestore.yaml tasks/kclient_workunit_suites_pjd.yaml} 3
pass 1688174 2017-09-29 23:21:40 2017-09-30 04:47:32 2017-09-30 06:15:33 1:28:01 0:18:55 1:09:06 smithi master kcephfs/recovery/{clusters/4-remote-clients.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore/bluestore.yaml tasks/volume-client.yaml whitelist_health.yaml} 4
pass 1688177 2017-09-29 23:21:41 2017-09-30 04:49:28 2017-09-30 05:23:27 0:33:59 0:10:07 0:23:52 smithi master kcephfs/cephfs/{clusters/fixed-3-cephfs.yaml conf.yaml inline/no.yaml objectstore/filestore-xfs.yaml tasks/kclient_workunit_trivial_sync.yaml} 3