Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 5614396 2020-11-12 00:05:04 2020-11-12 00:06:54 2020-11-12 00:30:53 0:23:59 0:12:20 0:11:39 smithi master ubuntu 16.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_fsstress} 2
pass 5614397 2020-11-12 00:05:04 2020-11-12 00:07:01 2020-11-12 00:41:01 0:34:00 0:19:25 0:14:35 smithi master centos 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/admin} 2
pass 5614398 2020-11-12 00:05:05 2020-11-12 00:09:07 2020-11-12 00:37:07 0:28:00 0:18:46 0:09:14 smithi master ubuntu 16.04 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_fsx} 2
pass 5614399 2020-11-12 00:05:06 2020-11-12 00:09:07 2020-11-12 00:25:07 0:16:00 0:06:54 0:09:06 smithi master fs/bugs/client_trim_caps/{begin clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i22073} 1
pass 5614400 2020-11-12 00:05:07 2020-11-12 00:09:14 2020-11-12 00:59:14 0:50:00 0:17:05 0:32:55 smithi master ubuntu 18.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} 4
pass 5614401 2020-11-12 00:05:08 2020-11-12 00:09:19 2020-11-12 01:01:19 0:52:00 0:35:40 0:16:20 smithi master ubuntu 18.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/failover} 2
pass 5614402 2020-11-12 00:05:08 2020-11-12 00:10:53 2020-11-12 00:34:52 0:23:59 0:08:01 0:15:58 smithi master ubuntu 18.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_misc} 2
pass 5614403 2020-11-12 00:05:09 2020-11-12 00:10:53 2020-11-12 01:44:54 1:34:01 1:18:31 0:15:30 smithi master centos 7.8 fs/snaps/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/snaptests} 2
pass 5614404 2020-11-12 00:05:10 2020-11-12 00:11:00 2020-11-12 04:07:04 3:56:04 3:39:50 0:16:14 smithi master ubuntu 16.04 fs/thrash/{begin ceph-thrash/default clusters/1-mds-1-client-coloc conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_snaptests} 2
pass 5614405 2020-11-12 00:05:11 2020-11-12 00:14:56 2020-11-12 00:48:56 0:34:00 0:17:48 0:16:12 smithi master ubuntu 16.04 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_blogbench traceless/50pc} 2
pass 5614406 2020-11-12 00:05:12 2020-11-12 00:14:56 2020-11-12 00:52:56 0:38:00 0:23:30 0:14:30 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-compat_client/mimic}} 3
pass 5614407 2020-11-12 00:05:12 2020-11-12 00:14:56 2020-11-12 01:22:57 1:08:01 0:51:52 0:16:09 smithi master centos 7.8 fs/verify/{begin centos_latest clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench validater/lockdep} 2
pass 5614408 2020-11-12 00:05:13 2020-11-12 00:17:28 2020-11-12 00:39:28 0:22:00 0:11:30 0:10:30 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/alternate-pool} 2
pass 5614409 2020-11-12 00:05:14 2020-11-12 00:17:28 2020-11-12 00:49:28 0:32:00 0:22:18 0:09:42 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/asok_dump_tree} 2
fail 5614410 2020-11-12 00:05:15 2020-11-12 00:17:28 2020-11-12 00:43:28 0:26:00 0:11:11 0:14:49 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-bitmap overrides/{frag_enable multimds/no whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/no 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 00:38:50.990140 mon.b (mon.0) 88 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614411 2020-11-12 00:05:15 2020-11-12 00:18:20 2020-11-12 00:38:20 0:20:00 0:08:49 0:11:11 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/auto-repair} 2
pass 5614412 2020-11-12 00:05:16 2020-11-12 00:20:09 2020-11-12 00:42:09 0:22:00 0:12:29 0:09:31 smithi master ubuntu 18.04 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_fsync} 2
pass 5614413 2020-11-12 00:05:17 2020-11-12 00:20:55 2020-11-12 00:40:54 0:19:59 0:07:16 0:12:43 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/backtrace} 2
pass 5614414 2020-11-12 00:05:18 2020-11-12 00:20:55 2020-11-12 01:34:56 1:14:01 1:02:00 0:12:01 smithi master ubuntu 18.04 fs/upgrade/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-mimic 1-client 2-upgrade 3-verify} ubuntu_18.04} 2
pass 5614415 2020-11-12 00:05:18 2020-11-12 00:20:56 2020-11-12 00:44:56 0:24:00 0:10:57 0:13:03 smithi master centos 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cap-flush} 2
pass 5614416 2020-11-12 00:05:19 2020-11-12 00:21:39 2020-11-12 00:43:38 0:21:59 0:09:21 0:12:38 smithi master ubuntu 16.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_pjd} 2
pass 5614417 2020-11-12 00:05:20 2020-11-12 00:21:39 2020-11-12 00:47:39 0:26:00 0:11:54 0:14:06 smithi master rhel 7.8 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614418 2020-11-12 00:05:21 2020-11-12 00:22:36 2020-11-12 00:46:36 0:24:00 0:11:38 0:12:22 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cephfs-shell} 2
pass 5614419 2020-11-12 00:05:22 2020-11-12 00:22:51 2020-11-12 01:14:51 0:52:00 0:25:33 0:26:27 smithi master ubuntu 18.04 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/filestore-xfs omap_limit/10000 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_iogen} 2
fail 5614420 2020-11-12 00:05:22 2020-11-12 00:22:55 2020-11-12 00:48:55 0:26:00 0:13:11 0:12:49 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable multimds/yes whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/yes 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 00:43:50.291573 mon.a (mon.0) 94 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614421 2020-11-12 00:05:23 2020-11-12 00:23:04 2020-11-12 00:51:03 0:27:59 0:17:40 0:10:19 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cephfs_scrub_tests} 2
pass 5614422 2020-11-12 00:05:24 2020-11-12 00:24:55 2020-11-12 00:42:55 0:18:00 0:07:46 0:10:14 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_quota} 2
pass 5614423 2020-11-12 00:05:25 2020-11-12 00:25:03 2020-11-12 00:55:03 0:30:00 0:16:33 0:13:27 smithi master fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
pass 5614424 2020-11-12 00:05:25 2020-11-12 00:25:07 2020-11-12 00:53:07 0:28:00 0:20:02 0:07:58 smithi master rhel 7.8 fs/thrash/{begin ceph-thrash/default clusters/1-mds-1-client-coloc conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_suites_fsstress} 2
pass 5614425 2020-11-12 00:05:26 2020-11-12 00:25:08 2020-11-12 01:27:08 1:02:00 0:55:00 0:07:00 smithi master rhel 7.8 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_suites_dbench traceless/50pc} 2
fail 5614426 2020-11-12 00:05:27 2020-11-12 00:25:39 2020-11-12 04:03:43 3:38:04 3:24:04 0:14:00 smithi master centos 7.8 fs/verify/{begin centos_latest clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress validater/valgrind} 2
pass 5614427 2020-11-12 00:05:28 2020-11-12 00:26:41 2020-11-12 01:02:41 0:36:00 0:16:20 0:19:40 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/client-limits} 2
pass 5614428 2020-11-12 00:05:29 2020-11-12 00:26:41 2020-11-12 00:54:41 0:28:00 0:09:24 0:18:36 smithi master ubuntu 16.04 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_iozone} 2
pass 5614429 2020-11-12 00:05:29 2020-11-12 00:47:19 2020-11-12 01:07:18 0:19:59 0:07:42 0:12:17 smithi master ubuntu 18.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} 5
pass 5614430 2020-11-12 00:05:30 2020-11-12 00:47:40 2020-11-12 01:05:39 0:17:59 0:07:38 0:10:21 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/client-readahad} 2
pass 5614431 2020-11-12 00:05:31 2020-11-12 00:48:10 2020-11-12 01:12:10 0:24:00 0:11:27 0:12:33 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-comp overrides/{frag_enable multimds/no whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/no 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
pass 5614432 2020-11-12 00:05:32 2020-11-12 00:49:17 2020-11-12 01:27:17 0:38:00 0:28:13 0:09:47 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/client-recovery} 2
pass 5614433 2020-11-12 00:05:33 2020-11-12 00:49:17 2020-11-12 01:13:17 0:24:00 0:13:11 0:10:49 smithi master ubuntu 18.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_fsstress} 2
pass 5614434 2020-11-12 00:05:33 2020-11-12 00:49:17 2020-11-12 01:37:17 0:48:00 0:37:33 0:10:27 smithi master ubuntu 16.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/failover} 2
pass 5614435 2020-11-12 00:05:34 2020-11-12 00:49:17 2020-11-12 01:09:17 0:20:00 0:11:25 0:08:35 smithi master centos 7.8 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_misc} 2
pass 5614436 2020-11-12 00:05:35 2020-11-12 00:49:17 2020-11-12 02:13:18 1:24:01 1:12:59 0:11:02 smithi master ubuntu 16.04 fs/snaps/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/snaptests} 2
pass 5614437 2020-11-12 00:05:36 2020-11-12 00:49:17 2020-11-12 01:17:17 0:28:00 0:16:48 0:11:12 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/damage} 2
pass 5614438 2020-11-12 00:05:36 2020-11-12 00:49:29 2020-11-12 01:11:29 0:22:00 0:12:09 0:09:51 smithi master centos 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614439 2020-11-12 00:05:37 2020-11-12 00:49:32 2020-11-12 01:33:32 0:44:00 0:31:11 0:12:49 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-compat_client/no}} 3
pass 5614440 2020-11-12 00:05:38 2020-11-12 00:51:31 2020-11-12 01:25:31 0:34:00 0:27:21 0:06:39 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/data-scan} 2
pass 5614441 2020-11-12 00:05:39 2020-11-12 00:51:32 2020-11-12 01:13:31 0:21:59 0:14:22 0:07:37 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/forward-scrub} 2
fail 5614442 2020-11-12 00:05:39 2020-11-12 00:51:32 2020-11-12 01:17:31 0:25:59 0:13:51 0:12:08 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-ec-root overrides/{frag_enable multimds/yes whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/yes 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 01:12:14.197948 mon.a (mon.0) 87 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614443 2020-11-12 00:05:40 2020-11-12 00:51:31 2020-11-12 01:13:31 0:22:00 0:10:55 0:11:05 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/fragment} 2
pass 5614444 2020-11-12 00:05:41 2020-11-12 00:51:32 2020-11-12 01:11:31 0:19:59 0:11:20 0:08:39 smithi master rhel 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_suites_truncate_delay} 2
pass 5614445 2020-11-12 00:05:42 2020-11-12 00:51:32 2020-11-12 01:21:32 0:30:00 0:18:21 0:11:39 smithi master centos 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/journal-repair} 2
pass 5614446 2020-11-12 00:05:43 2020-11-12 00:51:58 2020-11-12 01:27:58 0:36:00 0:23:11 0:12:49 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-compat_client/mimic}} 3
pass 5614447 2020-11-12 00:05:43 2020-11-12 00:53:35 2020-11-12 01:09:35 0:16:00 0:10:42 0:05:18 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/libcephfs_python} 2
pass 5614448 2020-11-12 00:05:44 2020-11-12 00:53:35 2020-11-12 01:11:35 0:18:00 0:08:40 0:09:20 smithi master ubuntu 18.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614449 2020-11-12 00:05:45 2020-11-12 00:53:35 2020-11-12 01:13:35 0:20:00 0:09:35 0:10:25 smithi master ubuntu 16.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_pjd} 2
pass 5614450 2020-11-12 00:05:46 2020-11-12 00:54:16 2020-11-12 01:18:16 0:24:00 0:13:06 0:10:54 smithi master centos 7.8 fs/thrash/{begin ceph-thrash/default clusters/1-mds-1-client-coloc conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614451 2020-11-12 00:05:46 2020-11-12 00:55:13 2020-11-12 01:35:13 0:40:00 0:31:10 0:08:50 smithi master ubuntu 16.04 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_ffsb traceless/50pc} 2
pass 5614452 2020-11-12 00:05:47 2020-11-12 00:55:13 2020-11-12 02:09:14 1:14:01 0:51:31 0:22:30 smithi master centos 7.8 fs/verify/{begin centos_latest clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench validater/lockdep} 2
pass 5614453 2020-11-12 00:05:48 2020-11-12 00:55:13 2020-11-12 01:23:13 0:28:00 0:07:09 0:20:51 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/mds-flush} 2
pass 5614454 2020-11-12 00:05:49 2020-11-12 00:55:13 2020-11-12 01:17:13 0:22:00 0:10:16 0:11:44 smithi master centos 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_trivial_sync} 2
pass 5614455 2020-11-12 00:05:50 2020-11-12 00:55:13 2020-11-12 01:41:13 0:46:00 0:22:35 0:23:25 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/mds-full} 2
fail 5614456 2020-11-12 00:05:50 2020-11-12 00:55:13 2020-11-12 01:19:13 0:24:00 0:11:16 0:12:44 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/filestore-xfs overrides/{frag_enable multimds/no whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/no 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 01:14:33.124898 mon.b (mon.0) 91 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614457 2020-11-12 00:05:51 2020-11-12 00:55:29 2020-11-12 01:19:28 0:23:59 0:09:55 0:14:04 smithi master ubuntu 18.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} 4
pass 5614458 2020-11-12 00:05:52 2020-11-12 00:55:30 2020-11-12 01:15:29 0:19:59 0:07:43 0:12:16 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/mds_creation_retry} 2
pass 5614459 2020-11-12 00:05:53 2020-11-12 00:55:40 2020-11-12 01:13:39 0:17:59 0:08:15 0:09:44 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/openfiletable} 2
pass 5614460 2020-11-12 00:05:54 2020-11-12 00:55:50 2020-11-12 01:25:50 0:30:00 0:17:31 0:12:29 smithi master fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
pass 5614461 2020-11-12 00:05:54 2020-11-12 00:56:58 2020-11-12 01:20:57 0:23:59 0:13:01 0:10:58 smithi master centos 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/libcephfs_interface_tests} 2
pass 5614462 2020-11-12 00:05:55 2020-11-12 00:56:58 2020-11-12 01:20:58 0:24:00 0:12:05 0:11:55 smithi master centos 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/pool-perm} 2
pass 5614463 2020-11-12 00:05:56 2020-11-12 00:57:30 2020-11-12 01:17:30 0:20:00 0:08:39 0:11:21 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/quota} 2
pass 5614464 2020-11-12 00:05:57 2020-11-12 00:57:31 2020-11-12 01:19:31 0:22:00 0:11:11 0:10:49 smithi master ubuntu 16.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_fsstress} 2
pass 5614465 2020-11-12 00:05:58 2020-11-12 00:57:43 2020-11-12 01:55:43 0:58:00 0:45:15 0:12:45 smithi master centos 7.8 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/failover} 2
pass 5614466 2020-11-12 00:05:58 2020-11-12 00:57:48 2020-11-12 01:17:48 0:20:00 0:07:41 0:12:19 smithi master ubuntu 18.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_misc} 2
pass 5614467 2020-11-12 00:05:59 2020-11-12 00:59:53 2020-11-12 02:15:53 1:16:00 1:00:40 0:15:20 smithi master ubuntu 16.04 fs/snaps/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/snaptests} 2
fail 5614468 2020-11-12 00:06:00 2020-11-12 00:59:53 2020-11-12 01:23:52 0:23:59 0:12:53 0:11:06 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-bitmap overrides/{frag_enable multimds/yes whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/yes 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 01:19:27.271078 mon.b (mon.0) 98 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614469 2020-11-12 00:06:01 2020-11-12 00:59:53 2020-11-12 01:29:53 0:30:00 0:10:43 0:19:17 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/sessionmap/{sessionmap}} 2
pass 5614470 2020-11-12 00:06:01 2020-11-12 01:00:11 2020-11-12 02:36:12 1:36:01 1:27:26 0:08:35 smithi master rhel 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_kernel_untar_build} 2
pass 5614471 2020-11-12 00:06:02 2020-11-12 01:00:25 2020-11-12 01:32:25 0:32:00 0:21:09 0:10:51 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/strays} 2
pass 5614472 2020-11-12 00:06:03 2020-11-12 01:01:34 2020-11-12 01:55:34 0:54:00 0:32:45 0:21:15 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-compat_client/no}} 3
pass 5614473 2020-11-12 00:06:04 2020-11-12 01:01:34 2020-11-12 01:39:34 0:38:00 0:14:58 0:23:02 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/test_journal_migration} 2
pass 5614474 2020-11-12 00:06:05 2020-11-12 01:01:34 2020-11-12 01:29:34 0:28:00 0:07:55 0:20:05 smithi master ubuntu 16.04 fs/thrash/{begin ceph-thrash/default clusters/1-mds-1-client-coloc conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_trivial_sync} 2
pass 5614475 2020-11-12 00:06:05 2020-11-12 01:01:34 2020-11-12 01:39:34 0:38:00 0:14:39 0:23:21 smithi master ubuntu 18.04 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_fsstress traceless/50pc} 2
pass 5614476 2020-11-12 00:06:06 2020-11-12 01:01:34 2020-11-12 04:33:38 3:32:04 3:09:58 0:22:06 smithi master centos 7.8 fs/verify/{begin centos_latest clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress validater/valgrind} 2
pass 5614477 2020-11-12 00:06:07 2020-11-12 01:01:40 2020-11-12 01:31:40 0:30:00 0:19:31 0:10:29 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/volume-client/{task/test/{test}}} 2
fail 5614478 2020-11-12 00:06:08 2020-11-12 01:06:10 2020-11-12 01:32:10 0:26:00 0:11:35 0:14:25 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable multimds/no whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/no 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 01:28:41.923240 mon.a (mon.0) 95 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614479 2020-11-12 00:06:09 2020-11-12 01:07:05 2020-11-12 02:01:05 0:54:00 0:38:56 0:15:04 smithi master ubuntu 16.04 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_misc} 2
dead 5614480 2020-11-12 00:06:09 2020-11-12 01:07:57 2020-11-12 13:10:27 12:02:30 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/volumes} 2
pass 5614481 2020-11-12 00:06:10 2020-11-12 01:09:56 2020-11-12 01:33:55 0:23:59 0:11:37 0:12:22 smithi master centos 7.8 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614482 2020-11-12 00:06:11 2020-11-12 01:09:56 2020-11-12 01:35:55 0:25:59 0:14:02 0:11:57 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/admin} 2
pass 5614483 2020-11-12 00:06:12 2020-11-12 01:09:56 2020-11-12 01:55:56 0:46:00 0:16:57 0:29:03 smithi master ubuntu 18.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} 5
pass 5614484 2020-11-12 00:06:13 2020-11-12 01:10:22 2020-11-12 01:32:21 0:21:59 0:12:02 0:09:57 smithi master centos 7.8 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614485 2020-11-12 00:06:13 2020-11-12 01:12:08 2020-11-12 01:56:08 0:44:00 0:21:35 0:22:25 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-compat_client/mimic}} 3
pass 5614486 2020-11-12 00:06:14 2020-11-12 01:12:08 2020-11-12 01:36:08 0:24:00 0:08:42 0:15:18 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/alternate-pool} 2
pass 5614487 2020-11-12 00:06:15 2020-11-12 01:12:08 2020-11-12 01:44:08 0:32:00 0:22:11 0:09:49 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/asok_dump_tree} 2
pass 5614488 2020-11-12 00:06:16 2020-11-12 01:12:11 2020-11-12 01:44:11 0:32:00 0:20:34 0:11:26 smithi master centos 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_misc_test_o_trunc} 2
pass 5614489 2020-11-12 00:06:16 2020-11-12 01:13:56 2020-11-12 02:29:57 1:16:01 1:04:24 0:11:37 smithi master ubuntu 18.04 fs/upgrade/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-mimic 1-client 2-upgrade 3-verify} ubuntu_18.04} 2
pass 5614490 2020-11-12 00:06:17 2020-11-12 01:13:56 2020-11-12 01:31:55 0:17:59 0:08:29 0:09:30 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/auto-repair} 2
pass 5614491 2020-11-12 00:06:18 2020-11-12 01:13:56 2020-11-12 01:33:55 0:19:59 0:07:03 0:12:56 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/backtrace} 2
fail 5614492 2020-11-12 00:06:19 2020-11-12 01:13:56 2020-11-12 01:37:56 0:24:00 0:13:17 0:10:43 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-comp overrides/{frag_enable multimds/yes whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/yes 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 01:33:29.336540 mon.a (mon.0) 92 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614493 2020-11-12 00:06:20 2020-11-12 01:13:56 2020-11-12 01:35:56 0:22:00 0:07:16 0:14:44 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cap-flush} 2
pass 5614494 2020-11-12 00:06:21 2020-11-12 01:14:24 2020-11-12 01:38:24 0:24:00 0:15:33 0:08:27 smithi master rhel 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_norstats} 2
pass 5614495 2020-11-12 00:06:21 2020-11-12 01:14:52 2020-11-12 01:40:52 0:26:00 0:14:40 0:11:20 smithi master ubuntu 18.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_fsstress} 2
pass 5614496 2020-11-12 00:06:22 2020-11-12 01:16:08 2020-11-12 02:04:08 0:48:00 0:35:39 0:12:21 smithi master ubuntu 18.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/failover} 2
pass 5614497 2020-11-12 00:06:23 2020-11-12 01:17:51 2020-11-12 01:43:51 0:26:00 0:11:34 0:14:26 smithi master rhel 7.8 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_misc} 2
pass 5614498 2020-11-12 00:06:24 2020-11-12 01:17:51 2020-11-12 02:45:52 1:28:01 1:11:08 0:16:53 smithi master ubuntu 16.04 fs/snaps/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/snaptests} 2
pass 5614499 2020-11-12 00:06:25 2020-11-12 01:17:51 2020-11-12 05:15:56 3:58:05 3:47:59 0:10:06 smithi master rhel 7.8 fs/thrash/{begin ceph-thrash/default clusters/1-mds-1-client-coloc conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_snaptests} 2
pass 5614500 2020-11-12 00:06:25 2020-11-12 01:17:51 2020-11-12 01:45:51 0:28:00 0:21:04 0:06:56 smithi master rhel 7.8 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_suites_blogbench traceless/50pc} 2
pass 5614501 2020-11-12 00:06:26 2020-11-12 01:17:51 2020-11-12 02:21:52 1:04:01 0:50:32 0:13:29 smithi master centos 7.8 fs/verify/{begin centos_latest clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench validater/lockdep} 2
pass 5614502 2020-11-12 00:06:27 2020-11-12 01:18:17 2020-11-12 01:40:17 0:22:00 0:11:14 0:10:46 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cephfs-shell} 2
pass 5614503 2020-11-12 00:06:28 2020-11-12 01:19:32 2020-11-12 01:53:31 0:33:59 0:18:20 0:15:39 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cephfs_scrub_tests} 2
pass 5614504 2020-11-12 00:06:29 2020-11-12 01:19:32 2020-11-12 01:47:31 0:27:59 0:16:27 0:11:32 smithi master fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
pass 5614505 2020-11-12 00:06:29 2020-11-12 01:19:32 2020-11-12 01:39:31 0:19:59 0:11:54 0:08:05 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_quota} 2
pass 5614506 2020-11-12 00:06:30 2020-11-12 01:19:32 2020-11-12 01:51:32 0:32:00 0:18:10 0:13:50 smithi master ubuntu 16.04 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/filestore-xfs omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_blogbench} 2
pass 5614507 2020-11-12 00:06:31 2020-11-12 01:21:37 2020-11-12 01:47:37 0:26:00 0:19:30 0:06:30 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/client-limits} 2
fail 5614508 2020-11-12 00:06:32 2020-11-12 01:21:37 2020-11-12 01:43:36 0:21:59 0:11:11 0:10:48 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-ec-root overrides/{frag_enable multimds/no whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/no 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 01:39:49.224935 mon.a (mon.0) 84 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614509 2020-11-12 00:06:33 2020-11-12 01:38:33 2020-11-12 01:58:33 0:20:00 0:07:50 0:12:10 smithi master ubuntu 18.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} 4
pass 5614510 2020-11-12 00:06:34 2020-11-12 01:40:11 2020-11-12 01:58:10 0:17:59 0:08:16 0:09:43 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/client-readahad} 2
pass 5614511 2020-11-12 00:06:34 2020-11-12 01:40:11 2020-11-12 02:22:11 0:42:00 0:29:28 0:12:32 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/client-recovery} 2
pass 5614512 2020-11-12 00:06:35 2020-11-12 01:40:11 2020-11-12 02:26:11 0:46:00 0:31:58 0:14:02 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-compat_client/no}} 3
pass 5614513 2020-11-12 00:06:36 2020-11-12 01:40:12 2020-11-12 02:42:12 1:02:00 0:53:03 0:08:57 smithi master rhel 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_suites_dbench} 2
pass 5614514 2020-11-12 00:06:37 2020-11-12 01:40:11 2020-11-12 02:02:11 0:22:00 0:09:17 0:12:43 smithi master ubuntu 18.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614515 2020-11-12 00:06:38 2020-11-12 01:40:11 2020-11-12 02:00:10 0:19:59 0:09:21 0:10:38 smithi master ubuntu 16.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_pjd} 2
pass 5614516 2020-11-12 00:06:39 2020-11-12 01:40:11 2020-11-12 02:10:11 0:30:00 0:17:30 0:12:30 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/damage} 2
pass 5614517 2020-11-12 00:06:39 2020-11-12 01:40:18 2020-11-12 02:12:18 0:32:00 0:21:56 0:10:04 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/data-scan} 2
fail 5614518 2020-11-12 00:06:40 2020-11-12 01:40:53 2020-11-12 02:06:53 0:26:00 0:12:54 0:13:06 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/filestore-xfs overrides/{frag_enable multimds/yes whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/yes 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 02:02:13.989564 mon.b (mon.0) 103 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614519 2020-11-12 00:06:41 2020-11-12 01:41:54 2020-11-12 02:01:54 0:20:00 0:14:03 0:05:57 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/forward-scrub} 2
pass 5614520 2020-11-12 00:06:42 2020-11-12 01:43:47 2020-11-12 02:29:47 0:46:00 0:36:39 0:09:21 smithi master ubuntu 16.04 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_ffsb} 2
pass 5614521 2020-11-12 00:06:43 2020-11-12 01:43:48 2020-11-12 02:11:47 0:27:59 0:17:09 0:10:50 smithi master centos 7.8 fs/thrash/{begin ceph-thrash/default clusters/1-mds-1-client-coloc conf/{client mds mon osd} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_suites_fsstress} 2
pass 5614522 2020-11-12 00:06:44 2020-11-12 01:43:52 2020-11-12 02:41:52 0:58:00 0:51:22 0:06:38 smithi master rhel 7.8 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_suites_dbench traceless/50pc} 2
fail 5614523 2020-11-12 00:06:44 2020-11-12 01:44:03 2020-11-12 05:08:07 3:24:04 3:11:56 0:12:08 smithi master centos 7.8 fs/verify/{begin centos_latest clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress validater/valgrind} 2
pass 5614524 2020-11-12 00:06:45 2020-11-12 01:44:09 2020-11-12 02:06:09 0:22:00 0:10:53 0:11:07 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/fragment} 2
pass 5614525 2020-11-12 00:06:46 2020-11-12 01:44:12 2020-11-12 02:20:11 0:35:59 0:22:57 0:13:02 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-compat_client/mimic}} 3
pass 5614526 2020-11-12 00:06:47 2020-11-12 01:44:13 2020-11-12 02:10:13 0:26:00 0:14:06 0:11:54 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/journal-repair} 2
pass 5614527 2020-11-12 00:06:48 2020-11-12 01:44:13 2020-11-12 02:06:13 0:22:00 0:10:33 0:11:27 smithi master centos 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/libcephfs_python} 2
pass 5614528 2020-11-12 00:06:48 2020-11-12 01:45:34 2020-11-12 02:07:34 0:22:00 0:12:45 0:09:15 smithi master ubuntu 16.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_fsstress} 2
pass 5614529 2020-11-12 00:06:49 2020-11-12 01:45:34 2020-11-12 02:43:35 0:58:01 0:46:23 0:11:38 smithi master centos 7.8 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/failover} 2
pass 5614530 2020-11-12 00:06:50 2020-11-12 01:45:34 2020-11-12 02:05:34 0:20:00 0:11:40 0:08:20 smithi master rhel 7.8 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_misc} 2
pass 5614531 2020-11-12 00:06:51 2020-11-12 01:45:52 2020-11-12 03:03:53 1:18:01 1:11:47 0:06:14 smithi master rhel 7.8 fs/snaps/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/snaptests} 2
pass 5614532 2020-11-12 00:06:52 2020-11-12 01:46:20 2020-11-12 02:10:19 0:23:59 0:12:55 0:11:04 smithi master ubuntu 18.04 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_fsstress} 2
fail 5614533 2020-11-12 00:06:53 2020-11-12 01:46:29 2020-11-12 02:10:29 0:24:00 0:12:23 0:11:37 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-bitmap overrides/{frag_enable multimds/yes whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/no 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 02:06:34.988283 mon.a (mon.0) 96 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614534 2020-11-12 00:06:53 2020-11-12 01:48:03 2020-11-12 02:08:03 0:20:00 0:07:36 0:12:24 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/mds-flush} 2
pass 5614535 2020-11-12 00:06:54 2020-11-12 01:48:03 2020-11-12 02:22:03 0:34:00 0:26:45 0:07:15 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/mds-full} 2
pass 5614536 2020-11-12 00:06:55 2020-11-12 01:48:04 2020-11-12 02:52:04 1:04:00 0:11:20 0:52:40 smithi master ubuntu 18.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} 5
fail 5614537 2020-11-12 00:06:56 2020-11-12 01:48:04 2020-11-12 02:28:04 0:40:00 0:12:27 0:27:33 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable multimds/no whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/yes 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 02:22:38.227582 mon.a (mon.0) 103 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614538 2020-11-12 00:06:57 2020-11-12 01:48:04 2020-11-12 02:14:03 0:25:59 0:10:27 0:15:32 smithi master centos 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/mds_creation_retry} 2
pass 5614539 2020-11-12 00:06:57 2020-11-12 01:48:04 2020-11-12 02:32:04 0:44:00 0:24:28 0:19:32 smithi master ubuntu 16.04 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/cfuse_workunit_suites_fsx} 2
pass 5614540 2020-11-12 00:06:58 2020-11-12 01:48:15 2020-11-12 02:18:14 0:29:59 0:07:26 0:22:33 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/openfiletable} 2
pass 5614541 2020-11-12 00:06:59 2020-11-12 01:48:15 2020-11-12 02:20:15 0:32:00 0:17:11 0:14:49 smithi master fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
pass 5614542 2020-11-12 00:07:00 2020-11-12 01:48:18 2020-11-12 02:08:18 0:20:00 0:11:25 0:08:35 smithi master rhel 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/pool-perm} 2
pass 5614543 2020-11-12 00:07:01 2020-11-12 01:48:19 2020-11-12 02:08:18 0:19:59 0:08:10 0:11:49 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/quota} 2
pass 5614544 2020-11-12 00:07:02 2020-11-12 01:48:27 2020-11-12 02:10:27 0:22:00 0:11:35 0:10:25 smithi master centos 7.8 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614545 2020-11-12 00:07:02 2020-11-12 01:48:57 2020-11-12 02:06:57 0:18:00 0:08:31 0:09:29 smithi master ubuntu 18.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614546 2020-11-12 00:07:03 2020-11-12 01:49:53 2020-11-12 02:11:53 0:22:00 0:10:32 0:11:28 smithi master ubuntu 18.04 fs/thrash/{begin ceph-thrash/default clusters/1-mds-1-client-coloc conf/{client mds mon osd} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cfuse_workunit_suites_pjd} 2
pass 5614547 2020-11-12 00:07:04 2020-11-12 01:49:53 2020-11-12 02:37:53 0:48:00 0:41:48 0:06:12 smithi master rhel 7.8 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_suites_ffsb traceless/50pc} 2
pass 5614548 2020-11-12 00:07:05 2020-11-12 01:49:56 2020-11-12 03:05:57 1:16:01 0:55:20 0:20:41 smithi master centos 7.8 fs/verify/{begin centos_latest clusters/fixed-2-ucephfs conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench validater/lockdep} 2
pass 5614549 2020-11-12 00:07:06 2020-11-12 01:50:09 2020-11-12 02:14:08 0:23:59 0:12:31 0:11:28 smithi master rhel 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/filestore-xfs omap_limit/10000 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{rhel_latest} tasks/cfuse_workunit_suites_fsync} 2
pass 5614550 2020-11-12 00:07:06 2020-11-12 01:50:09 2020-11-12 02:24:09 0:34:00 0:16:44 0:17:16 smithi master centos 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/sessionmap/{sessionmap}} 2
fail 5614551 2020-11-12 00:07:07 2020-11-12 01:50:10 2020-11-12 02:20:09 0:29:59 0:11:45 0:18:14 smithi master fs/upgrade/snaps/{clusters/3-mds conf/{client mds mon osd} objectstore-ec/bluestore-comp overrides/{frag_enable multimds/yes whitelist_health whitelist_rstat whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-sanity 4-client-upgrade/no 5-client-sanity 6-snap-upgrade 7-client-sanity}} 3
Failure Reason:

"2020-11-12 02:16:57.110605 mon.b (mon.0) 105 : cluster [ERR] Health check failed: 3 mgr modules have failed (MGR_MODULE_ERROR)" in cluster log

pass 5614552 2020-11-12 00:07:08 2020-11-12 01:52:12 2020-11-12 02:24:11 0:31:59 0:20:25 0:11:34 smithi master ubuntu 16.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_16.04} tasks/strays} 2
pass 5614553 2020-11-12 00:07:09 2020-11-12 01:52:12 2020-11-12 02:24:12 0:32:00 0:15:35 0:16:25 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/test_journal_migration} 2
pass 5614554 2020-11-12 00:07:10 2020-11-12 01:52:12 2020-11-12 02:38:12 0:46:00 0:32:55 0:13:05 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no whitelist_health whitelist_wrongly_marked_down} tasks/{0-luminous 1-client 2-upgrade 3-compat_client/no}} 3
pass 5614555 2020-11-12 00:07:10 2020-11-12 01:52:12 2020-11-12 02:40:12 0:48:00 0:31:59 0:16:01 smithi master centos 7.8 fs/basic_workload/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} inline/yes mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/cfuse_workunit_suites_iogen} 2
pass 5614556 2020-11-12 00:07:11 2020-11-12 01:52:15 2020-11-12 02:38:15 0:46:00 0:28:02 0:17:58 smithi master centos 7.8 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{centos_latest} tasks/volume-client/{task/test/{test}}} 2