Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 6418135 2021-10-02 19:36:33 2021-10-02 19:38:30 2021-10-02 19:58:04 0:19:34 0:08:39 0:10:55 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} 3
Failure Reason:

"2021-10-02T19:55:51.658448+0000 mon.a (mon.0) 251 : cluster [WRN] Health check failed: Degraded data redundancy: 2/48 objects degraded (4.167%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6418136 2021-10-02 19:36:34 2021-10-02 19:38:31 2021-10-02 19:58:35 0:20:04 0:08:27 0:11:37 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

managers

pass 6418137 2021-10-02 19:36:35 2021-10-02 19:38:31 2021-10-02 20:06:57 0:28:26 0:19:49 0:08:37 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
fail 6418138 2021-10-02 19:36:36 2021-10-02 19:38:31 2021-10-02 19:58:29 0:19:58 0:11:32 0:08:26 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota} 2
Failure Reason:

managers

fail 6418139 2021-10-02 19:36:38 2021-10-02 19:39:02 2021-10-02 19:58:12 0:19:10 0:12:25 0:06:45 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
Failure Reason:

managers

pass 6418140 2021-10-02 19:36:39 2021-10-02 19:39:12 2021-10-02 20:05:19 0:26:07 0:18:58 0:07:09 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} 2
pass 6418141 2021-10-02 19:36:40 2021-10-02 19:40:13 2021-10-02 20:11:53 0:31:40 0:18:32 0:13:08 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
pass 6418142 2021-10-02 19:36:41 2021-10-02 19:40:23 2021-10-02 19:59:26 0:19:03 0:13:17 0:05:46 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs} 2
fail 6418143 2021-10-02 19:36:42 2021-10-02 19:40:33 2021-10-02 19:59:48 0:19:15 0:11:39 0:07:36 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
Failure Reason:

managers

fail 6418144 2021-10-02 19:36:43 2021-10-02 19:40:44 2021-10-02 20:00:21 0:19:37 0:09:13 0:10:24 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
Failure Reason:

"2021-10-02T19:58:34.381171+0000 mon.a (mon.0) 255 : cluster [WRN] Health check failed: Degraded data redundancy: 4/48 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED)" in cluster log

pass 6418145 2021-10-02 19:36:44 2021-10-02 19:40:44 2021-10-02 20:03:55 0:23:11 0:17:33 0:05:38 smithi master rhel 8.4 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
pass 6418146 2021-10-02 19:36:45 2021-10-02 19:40:54 2021-10-02 20:01:35 0:20:41 0:11:13 0:09:28 smithi master centos 8.3 fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i24137} 1
pass 6418147 2021-10-02 19:36:46 2021-10-02 19:40:54 2021-10-02 20:36:30 0:55:36 0:45:02 0:10:34 smithi master centos 8.3 fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest} 2
pass 6418148 2021-10-02 19:36:47 2021-10-02 19:40:55 2021-10-02 19:58:24 0:17:29 0:11:51 0:05:38 smithi master ubuntu 20.04 fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full} 1
pass 6418149 2021-10-02 19:36:48 2021-10-02 19:40:55 2021-10-02 20:02:23 0:21:28 0:11:41 0:09:47 smithi master centos 8.stream fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8.stream} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client} 2
pass 6418150 2021-10-02 19:36:50 2021-10-02 19:41:25 2021-10-02 21:41:39 2:00:14 1:53:55 0:06:19 smithi master ubuntu 20.04 fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} 1
pass 6418151 2021-10-02 19:36:51 2021-10-02 19:41:56 2021-10-02 20:21:56 0:40:00 0:32:44 0:07:16 smithi master rhel 8.4 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
pass 6418152 2021-10-02 19:36:52 2021-10-02 19:42:47 2021-10-02 20:11:22 0:28:35 0:19:52 0:08:43 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} 5
fail 6418153 2021-10-02 19:36:53 2021-10-02 19:43:38 2021-10-02 20:32:42 0:49:04 0:29:20 0:19:44 smithi master centos 8.3 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} 2
Failure Reason:

Test failure: test_simple (tasks.cephfs.test_failover.TestFailover)

pass 6418154 2021-10-02 19:36:54 2021-10-02 19:44:58 2021-10-02 20:09:16 0:24:18 0:12:59 0:11:19 smithi master centos 8.3 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
pass 6418155 2021-10-02 19:36:55 2021-10-02 19:46:19 2021-10-02 20:27:19 0:41:00 0:33:28 0:07:32 smithi master ubuntu 20.04 fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell} 2
pass 6418156 2021-10-02 19:36:56 2021-10-02 19:46:49 2021-10-02 20:10:43 0:23:54 0:12:07 0:11:47 smithi master ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} 2
pass 6418157 2021-10-02 19:36:57 2021-10-02 19:47:09 2021-10-02 20:03:08 0:15:59 0:08:05 0:07:54 smithi master ubuntu 20.04 fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} 1
fail 6418158 2021-10-02 19:36:58 2021-10-02 19:47:40 2021-10-02 20:05:15 0:17:35 0:06:15 0:11:20 smithi master centos 8.3 fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

Command failed on smithi111 with status 1: 'sudo yum -y install ceph-volume'

pass 6418159 2021-10-02 19:36:59 2021-10-02 19:47:50 2021-10-02 22:00:53 2:13:03 2:03:23 0:09:40 smithi master centos 8.3 fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} 1
pass 6418160 2021-10-02 19:37:00 2021-10-02 19:47:51 2021-10-02 20:57:00 1:09:09 0:55:51 0:13:18 smithi master rhel 8.4 fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/valgrind} 2
fail 6418161 2021-10-02 19:37:01 2021-10-02 19:47:51 2021-10-02 20:08:03 0:20:12 0:12:07 0:08:05 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
Failure Reason:

managers

fail 6418162 2021-10-02 19:37:02 2021-10-02 19:48:31 2021-10-02 20:15:00 0:26:29 0:10:24 0:16:05 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} 2
Failure Reason:

managers

pass 6418163 2021-10-02 19:37:03 2021-10-02 19:49:42 2021-10-02 20:29:38 0:39:56 0:30:19 0:09:37 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
pass 6418164 2021-10-02 19:37:04 2021-10-02 19:51:43 2021-10-02 20:49:11 0:57:28 0:47:19 0:10:09 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
fail 6418165 2021-10-02 19:37:05 2021-10-02 19:52:13 2021-10-02 20:14:55 0:22:42 0:10:18 0:12:24 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
Failure Reason:

managers

pass 6418166 2021-10-02 19:37:06 2021-10-02 19:53:54 2021-10-02 20:19:39 0:25:45 0:11:50 0:13:55 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap} 2
fail 6418167 2021-10-02 19:37:07 2021-10-02 19:55:44 2021-10-02 20:14:43 0:18:59 0:07:17 0:11:42 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} 2
Failure Reason:

managers

fail 6418168 2021-10-02 19:37:08 2021-10-02 19:56:55 2021-10-02 20:25:39 0:28:44 0:18:17 0:10:27 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
Failure Reason:

"2021-10-02T20:14:01.294628+0000 mon.a (mon.0) 163 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6418169 2021-10-02 19:37:09 2021-10-02 19:57:25 2021-10-02 20:31:09 0:33:44 0:27:07 0:06:37 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
fail 6418170 2021-10-02 19:37:10 2021-10-02 19:57:26 2021-10-02 20:11:29 0:14:03 0:06:33 0:07:30 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} 3
Failure Reason:

managers

fail 6418171 2021-10-02 19:37:13 2021-10-02 19:58:06 2021-10-02 20:17:03 0:18:57 0:09:40 0:09:17 smithi master centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} 2
Failure Reason:

managers

pass 6418172 2021-10-02 19:37:14 2021-10-02 19:58:16 2021-10-02 21:31:23 1:33:07 1:26:28 0:06:39 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} 3
fail 6418173 2021-10-02 19:37:15 2021-10-02 19:58:27 2021-10-02 20:27:47 0:29:20 0:22:17 0:07:03 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} 2
Failure Reason:

reached maximum tries (90) after waiting for 540 seconds

dead 6418174 2021-10-02 19:37:16 2021-10-02 19:58:37 2021-10-02 20:16:37 0:18:00 0:06:00 0:12:00 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} 3
Failure Reason:

{'Failure object was': {'smithi106.front.sepia.ceph.com': {'ansible_facts': {}, 'msg': 'The following modules failed to execute: setup\\n', 'failed_modules': {'setup': {'failed': True, 'module_stdout': '', 'module_stderr': '', 'msg': 'MODULE FAILURE\\nSee stdout/stderr for the exact error', 'rc': -13}}, '_ansible_verbose_override': True, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_f08b273eeec75318a83a4e8d7fb51841e4d71be0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'failed')"}

pass 6418175 2021-10-02 19:37:17 2021-10-02 19:59:27 2021-10-02 20:28:53 0:29:26 0:23:08 0:06:18 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} 2
pass 6418176 2021-10-02 19:37:18 2021-10-02 19:59:58 2021-10-02 20:29:00 0:29:02 0:19:41 0:09:21 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} 2
fail 6418177 2021-10-02 19:37:19 2021-10-02 20:00:08 2021-10-02 20:23:54 0:23:46 0:14:21 0:09:25 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
Failure Reason:

"2021-10-02T20:16:27.214995+0000 mon.a (mon.0) 179 : cluster [WRN] Health check failed: Degraded data redundancy: 1/4 objects degraded (25.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6418178 2021-10-02 19:37:21 2021-10-02 20:00:28 2021-10-02 20:22:40 0:22:12 0:09:44 0:12:28 smithi master centos 8.3 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

managers

pass 6418179 2021-10-02 19:37:22 2021-10-02 20:01:39 2021-10-02 21:05:39 1:04:00 0:53:45 0:10:15 smithi master centos 8.stream fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} 2
pass 6418180 2021-10-02 19:37:23 2021-10-02 20:02:29 2021-10-02 20:49:55 0:47:26 0:37:30 0:09:56 smithi master centos 8.3 fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 6418181 2021-10-02 19:37:24 2021-10-02 20:03:00 2021-10-02 20:57:56 0:54:56 0:42:21 0:12:35 smithi master centos 8.3 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} 2
Failure Reason:

"2021-10-02T20:41:59.079802+0000 mon.a (mon.0) 1968 : cluster [WRN] Health check failed: Degraded data redundancy: 1/184 objects degraded (0.543%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6418182 2021-10-02 19:37:25 2021-10-02 20:03:30 2021-10-02 20:21:33 0:18:03 0:07:14 0:10:49 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} 2
Failure Reason:

managers

fail 6418183 2021-10-02 19:37:26 2021-10-02 20:04:01 2021-10-02 20:22:57 0:18:56 0:11:57 0:06:59 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
Failure Reason:

managers

fail 6418184 2021-10-02 19:37:27 2021-10-02 20:04:01 2021-10-02 20:28:32 0:24:31 0:10:19 0:14:12 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} 2
Failure Reason:

managers

pass 6418185 2021-10-02 19:37:28 2021-10-02 20:05:22 2021-10-02 20:26:44 0:21:22 0:14:51 0:06:31 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 6418186 2021-10-02 19:37:29 2021-10-02 20:05:22 2021-10-02 21:18:23 1:13:01 1:04:10 0:08:51 smithi master ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}} 2
fail 6418187 2021-10-02 19:37:30 2021-10-02 20:06:53 2021-10-02 20:28:56 0:22:03 0:10:11 0:11:52 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
Failure Reason:

managers

fail 6418188 2021-10-02 19:37:32 2021-10-02 20:07:03 2021-10-02 20:21:55 0:14:52 0:06:01 0:08:51 smithi master ubuntu 20.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

managers

fail 6418189 2021-10-02 19:37:32 2021-10-02 20:08:03 2021-10-02 20:27:41 0:19:38 0:12:00 0:07:38 smithi master rhel 8.4 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

managers

fail 6418190 2021-10-02 19:37:34 2021-10-02 20:08:14 2021-10-03 00:04:59 3:56:45 3:45:49 0:10:56 smithi master centos 8.stream fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi038 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ab8f2f64360119e72bc635c23e114f82b2e1e802 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'

pass 6418191 2021-10-02 19:37:35 2021-10-02 20:09:24 2021-10-02 20:35:00 0:25:36 0:15:05 0:10:31 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration} 2
fail 6418192 2021-10-02 19:37:36 2021-10-02 20:09:25 2021-10-02 20:58:15 0:48:50 0:37:19 0:11:31 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
Failure Reason:

"2021-10-02T20:27:59.285703+0000 mon.a (mon.0) 157 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6418193 2021-10-02 19:37:37 2021-10-02 20:10:45 2021-10-02 20:30:38 0:19:53 0:11:53 0:08:00 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} 2
Failure Reason:

managers

fail 6418194 2021-10-02 19:37:38 2021-10-02 20:11:25 2021-10-02 20:30:21 0:18:56 0:12:10 0:06:46 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
Failure Reason:

managers

fail 6418195 2021-10-02 19:37:39 2021-10-02 20:11:26 2021-10-02 20:30:21 0:18:55 0:08:13 0:10:42 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay} 2
Failure Reason:

managers

pass 6418196 2021-10-02 19:37:40 2021-10-02 20:11:26 2021-10-02 20:32:49 0:21:23 0:15:44 0:05:39 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
pass 6418197 2021-10-02 19:37:41 2021-10-02 20:11:37 2021-10-02 20:38:33 0:26:56 0:16:11 0:10:45 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
fail 6418198 2021-10-02 19:37:42 2021-10-02 20:11:37 2021-10-02 20:34:16 0:22:39 0:09:57 0:12:42 smithi master centos 8.3 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} 2
Failure Reason:

managers

fail 6418199 2021-10-02 19:37:43 2021-10-02 20:11:38 2021-10-02 20:25:58 0:14:20 0:06:27 0:07:53 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} 4
Failure Reason:

managers

fail 6418200 2021-10-02 19:37:44 2021-10-02 20:12:09 2021-10-02 20:31:39 0:19:30 0:07:50 0:11:40 smithi master ubuntu 20.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} 2
Failure Reason:

"2021-10-02T20:29:22.236399+0000 mon.a (mon.0) 271 : cluster [WRN] Health check failed: Degraded data redundancy: 9/46 objects degraded (19.565%), 4 pgs degraded (PG_DEGRADED)" in cluster log

fail 6418201 2021-10-02 19:37:45 2021-10-02 20:12:09 2021-10-02 20:25:58 0:13:49 0:07:06 0:06:43 smithi master ubuntu 20.04 fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/lockdep} 2
Failure Reason:

"2021-10-02T20:24:30.245889+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log

fail 6418202 2021-10-02 19:37:46 2021-10-02 20:12:19 2021-10-02 20:35:49 0:23:30 0:10:51 0:12:39 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
Failure Reason:

managers

fail 6418203 2021-10-02 19:37:47 2021-10-02 20:14:50 2021-10-02 20:42:21 0:27:31 0:17:42 0:09:49 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} 2
Failure Reason:

reached maximum tries (90) after waiting for 540 seconds

pass 6418204 2021-10-02 19:37:49 2021-10-02 20:15:01 2021-10-02 20:36:11 0:21:10 0:14:51 0:06:19 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota} 2
pass 6418205 2021-10-02 19:37:50 2021-10-02 20:15:02 2021-10-02 21:10:24 0:55:22 0:44:21 0:11:01 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
fail 6418206 2021-10-02 19:37:51 2021-10-02 20:16:42 2021-10-02 20:31:32 0:14:50 0:06:31 0:08:19 smithi master centos 8.3 fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} 1
Failure Reason:

Command failed on smithi096 with status 1: 'sudo yum -y install ceph-volume'

pass 6418207 2021-10-02 19:37:53 2021-10-02 20:16:42 2021-10-02 20:34:29 0:17:47 0:10:34 0:07:13 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} 2
fail 6418208 2021-10-02 19:37:54 2021-10-02 20:17:13 2021-10-02 20:31:34 0:14:21 0:06:53 0:07:28 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
Failure Reason:

"2021-10-02T20:30:31.562830+0000 mon.a (mon.0) 395 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs down (PG_AVAILABILITY)" in cluster log

fail 6418209 2021-10-02 19:37:55 2021-10-02 20:18:13 2021-10-02 20:43:03 0:24:50 0:09:51 0:14:59 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} 2
Failure Reason:

managers

pass 6418210 2021-10-02 19:37:56 2021-10-02 20:19:44 2021-10-02 20:53:33 0:33:49 0:26:42 0:07:07 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
fail 6418211 2021-10-02 19:37:57 2021-10-02 20:20:24 2021-10-02 20:39:33 0:19:09 0:11:32 0:07:37 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
Failure Reason:

managers

fail 6418212 2021-10-02 19:37:58 2021-10-02 20:21:35 2021-10-02 20:43:09 0:21:34 0:10:15 0:11:19 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
Failure Reason:

managers

pass 6418213 2021-10-02 19:37:59 2021-10-02 20:22:05 2021-10-02 20:45:11 0:23:06 0:16:40 0:06:26 smithi master rhel 8.4 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}} 2
pass 6418214 2021-10-02 19:38:00 2021-10-02 20:22:06 2021-10-02 21:22:29 1:00:23 0:47:39 0:12:44 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} 2
pass 6418215 2021-10-02 19:38:02 2021-10-02 20:22:46 2021-10-02 22:00:37 1:37:51 1:27:47 0:10:04 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
pass 6418216 2021-10-02 19:38:03 2021-10-02 20:23:07 2021-10-02 21:22:39 0:59:32 0:48:09 0:11:23 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
fail 6418217 2021-10-02 19:38:04 2021-10-02 20:23:27 2021-10-02 20:42:48 0:19:21 0:08:41 0:10:40 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} 3
Failure Reason:

managers

fail 6418218 2021-10-02 19:38:05 2021-10-02 20:23:57 2021-10-02 20:39:28 0:15:31 0:06:10 0:09:21 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} 2
Failure Reason:

managers

pass 6418219 2021-10-02 19:38:07 2021-10-02 20:25:48 2021-10-02 20:55:15 0:29:27 0:17:04 0:12:23 smithi master centos 8.stream fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
pass 6418220 2021-10-02 19:38:08 2021-10-02 20:26:08 2021-10-02 20:41:45 0:15:37 0:08:33 0:07:04 smithi master ubuntu 20.04 fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/ino_release_cb} 2
pass 6418221 2021-10-02 19:38:09 2021-10-02 20:26:09 2021-10-02 20:49:16 0:23:07 0:13:37 0:09:30 smithi master centos 8.stream fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
pass 6418222 2021-10-02 19:38:10 2021-10-02 20:26:09 2021-10-02 20:49:50 0:23:41 0:16:16 0:07:25 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
fail 6418223 2021-10-02 19:38:11 2021-10-02 20:26:19 2021-10-02 20:44:53 0:18:34 0:06:55 0:11:39 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} 2
Failure Reason:

managers

fail 6418224 2021-10-02 19:38:13 2021-10-02 20:26:50 2021-10-02 20:49:07 0:22:17 0:10:38 0:11:39 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
Failure Reason:

managers

pass 6418225 2021-10-02 19:38:14 2021-10-02 20:27:20 2021-10-02 20:53:02 0:25:42 0:18:17 0:07:25 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
pass 6418226 2021-10-02 19:38:15 2021-10-02 20:27:51 2021-10-02 20:57:30 0:29:39 0:23:20 0:06:19 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} 2
fail 6418227 2021-10-02 19:38:16 2021-10-02 20:27:51 2021-10-02 21:20:20 0:52:29 0:40:45 0:11:44 smithi master ubuntu 20.04 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

"2021-10-02T20:44:26.324083+0000 mon.a (mon.0) 118 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6418228 2021-10-02 19:38:17 2021-10-02 20:28:41 2021-10-02 20:48:28 0:19:47 0:09:55 0:09:52 smithi master centos 8.stream fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} 2
Failure Reason:

managers

fail 6418229 2021-10-02 19:38:18 2021-10-02 20:29:02 2021-10-02 20:46:37 0:17:35 0:06:23 0:11:12 smithi master centos 8.3 fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

Command failed on smithi157 with status 1: 'sudo yum -y install ceph-volume'

pass 6418230 2021-10-02 19:38:19 2021-10-02 20:29:02 2021-10-02 20:54:35 0:25:33 0:14:51 0:10:42 smithi master ubuntu 20.04 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} 2
pass 6418231 2021-10-02 19:38:20 2021-10-02 20:29:03 2021-10-02 20:52:07 0:23:04 0:11:27 0:11:37 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} 3
fail 6418232 2021-10-02 19:38:21 2021-10-02 20:29:43 2021-10-02 20:48:55 0:19:12 0:12:21 0:06:51 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

managers

fail 6418233 2021-10-02 19:38:23 2021-10-02 20:30:23 2021-10-02 20:49:19 0:18:56 0:08:30 0:10:26 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
Failure Reason:

managers

fail 6418234 2021-10-02 19:38:24 2021-10-02 20:30:24 2021-10-02 20:53:45 0:23:21 0:09:57 0:13:24 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} 2
Failure Reason:

managers

pass 6418235 2021-10-02 19:38:25 2021-10-02 20:30:44 2021-10-02 21:05:01 0:34:17 0:27:04 0:07:13 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
pass 6418236 2021-10-02 19:38:26 2021-10-02 20:31:14 2021-10-02 21:06:15 0:35:01 0:24:13 0:10:48 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} 2
fail 6418237 2021-10-02 19:38:27 2021-10-02 20:31:36 2021-10-02 20:52:27 0:20:51 0:10:29 0:10:22 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
Failure Reason:

managers

fail 6418238 2021-10-02 19:38:28 2021-10-02 20:31:46 2021-10-02 20:50:57 0:19:11 0:07:14 0:11:57 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
Failure Reason:

managers

pass 6418239 2021-10-02 19:38:29 2021-10-02 20:32:47 2021-10-02 20:52:36 0:19:49 0:09:26 0:10:23 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} 2
pass 6418240 2021-10-02 19:38:30 2021-10-02 20:32:57 2021-10-02 20:57:09 0:24:12 0:13:05 0:11:07 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} 3
pass 6418241 2021-10-02 19:38:31 2021-10-02 20:33:37 2021-10-02 21:13:41 0:40:04 0:28:44 0:11:20 smithi master ubuntu 20.04 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
pass 6418242 2021-10-02 19:38:32 2021-10-02 20:34:18 2021-10-02 20:58:43 0:24:25 0:16:15 0:08:10 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} 5
fail 6418243 2021-10-02 19:38:33 2021-10-02 20:35:08 2021-10-02 21:13:56 0:38:48 0:25:57 0:12:51 smithi master ubuntu 20.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} 2
Failure Reason:

Test failure: test_simple (tasks.cephfs.test_failover.TestFailover)

pass 6418244 2021-10-02 19:38:34 2021-10-02 20:35:59 2021-10-02 21:39:42 1:03:43 0:49:12 0:14:31 smithi master centos 8.3 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} 2
pass 6418245 2021-10-02 19:38:35 2021-10-02 20:36:19 2021-10-02 21:42:10 1:05:51 0:54:29 0:11:22 smithi master rhel 8.4 fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/valgrind} 2
pass 6418246 2021-10-02 19:38:37 2021-10-02 20:36:40 2021-10-02 21:00:14 0:23:34 0:11:22 0:12:12 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
fail 6418247 2021-10-02 19:38:38 2021-10-02 20:38:30 2021-10-02 20:53:13 0:14:43 0:07:01 0:07:42 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
Failure Reason:

"2021-10-02T20:52:09.713575+0000 mon.a (mon.0) 363 : cluster [WRN] Health check failed: Reduced data availability: 7 pgs down (PG_AVAILABILITY)" in cluster log

fail 6418248 2021-10-02 19:38:39 2021-10-02 20:39:31 2021-10-02 20:58:35 0:19:04 0:11:33 0:07:31 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush} 2
Failure Reason:

managers

fail 6418249 2021-10-02 19:38:40 2021-10-02 20:39:41 2021-10-02 21:01:05 0:21:24 0:12:17 0:09:07 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
Failure Reason:

managers

pass 6418250 2021-10-02 19:38:41 2021-10-02 20:41:52 2021-10-02 21:19:32 0:37:40 0:30:00 0:07:40 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
pass 6418251 2021-10-02 19:38:42 2021-10-02 20:42:22 2021-10-02 21:28:11 0:45:49 0:35:01 0:10:48 smithi master centos 8.3 fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 6418252 2021-10-02 19:38:43 2021-10-02 20:42:32 2021-10-02 21:59:47 1:17:15 1:10:31 0:06:44 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} 2
fail 6418253 2021-10-02 19:38:44 2021-10-02 20:42:54 2021-10-02 21:04:16 0:21:22 0:10:26 0:10:56 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
Failure Reason:

managers

pass 6418254 2021-10-02 19:38:45 2021-10-02 20:43:04 2021-10-02 21:10:01 0:26:57 0:20:05 0:06:52 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} 2
fail 6418255 2021-10-02 19:38:46 2021-10-02 20:43:15 2021-10-02 21:04:11 0:20:56 0:11:53 0:09:03 smithi master rhel 8.4 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

managers

fail 6418256 2021-10-02 19:38:47 2021-10-02 20:44:55 2021-10-02 20:59:07 0:14:12 0:06:07 0:08:05 smithi master ubuntu 20.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

"2021-10-02T20:57:14.645343+0000 mon.a (mon.0) 277 : cluster [WRN] Health check failed: Degraded data redundancy: 5/46 objects degraded (10.870%), 3 pgs degraded (PG_DEGRADED)" in cluster log

fail 6418257 2021-10-02 19:38:48 2021-10-02 20:45:16 2021-10-02 21:09:29 0:24:13 0:10:21 0:13:52 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
Failure Reason:

managers

fail 6418258 2021-10-02 19:38:49 2021-10-02 20:48:36 2021-10-02 21:42:21 0:53:45 0:44:40 0:09:05 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} 3
Failure Reason:

"2021-10-02T21:29:30.215238+0000 mds.d (mds.1) 216 : cluster [WRN] Scrub error on inode 0x10000000f73 (/client.0/tmp/clients/client7/~dmtmp/WORD/BASEMACH.DOC) see mds.d log and `damage ls` output for details" in cluster log

fail 6418259 2021-10-02 19:38:50 2021-10-02 20:48:57 2021-10-02 21:10:20 0:21:23 0:09:04 0:12:19 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
Failure Reason:

managers

fail 6418260 2021-10-02 19:38:51 2021-10-02 20:49:17 2021-10-02 21:12:01 0:22:44 0:10:03 0:12:41 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} 2
Failure Reason:

managers

pass 6418261 2021-10-02 19:38:52 2021-10-02 20:49:18 2021-10-02 21:11:20 0:22:02 0:10:46 0:11:16 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} 2
pass 6418262 2021-10-02 19:38:53 2021-10-02 20:49:18 2021-10-02 21:12:12 0:22:54 0:16:30 0:06:24 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} 3
fail 6418263 2021-10-02 19:38:54 2021-10-02 20:49:28 2021-10-02 21:08:53 0:19:25 0:08:23 0:11:02 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} 2
Failure Reason:

managers

fail 6418264 2021-10-02 19:38:55 2021-10-02 20:49:59 2021-10-02 21:11:33 0:21:34 0:10:01 0:11:33 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
Failure Reason:

managers

pass 6418265 2021-10-02 19:38:56 2021-10-02 20:49:59 2021-10-02 21:31:11 0:41:12 0:26:55 0:14:17 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} 2
pass 6418266 2021-10-02 19:38:57 2021-10-02 20:50:59 2021-10-02 21:15:54 0:24:55 0:15:54 0:09:01 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} 2
pass 6418267 2021-10-02 19:38:58 2021-10-02 20:52:10 2021-10-02 21:21:51 0:29:41 0:19:39 0:10:02 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
fail 6418268 2021-10-02 19:38:59 2021-10-02 20:52:30 2021-10-02 21:06:53 0:14:23 0:06:38 0:07:45 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
Failure Reason:

managers

fail 6418269 2021-10-02 19:39:00 2021-10-02 20:52:41 2021-10-02 21:12:42 0:20:01 0:09:25 0:10:36 smithi master centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} 2
Failure Reason:

managers

pass 6418270 2021-10-02 19:39:01 2021-10-02 20:53:11 2021-10-02 21:27:03 0:33:52 0:23:10 0:10:42 smithi master ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} 2
pass 6418271 2021-10-02 19:39:02 2021-10-02 20:53:22 2021-10-02 21:24:52 0:31:30 0:25:26 0:06:04 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} 3
fail 6418272 2021-10-02 19:39:03 2021-10-02 20:53:42 2021-10-02 21:12:49 0:19:07 0:08:28 0:10:39 smithi master centos 8.3 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

managers

pass 6418273 2021-10-02 19:39:04 2021-10-02 20:53:52 2021-10-02 21:17:34 0:23:42 0:17:16 0:06:26 smithi master rhel 8.4 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} 2
fail 6418274 2021-10-02 19:39:05 2021-10-02 20:54:43 2021-10-02 21:12:36 0:17:53 0:06:38 0:11:15 smithi master centos 8.3 fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} 3
Failure Reason:

Command failed on smithi191 with status 1: 'sudo yum -y install ceph-volume'

pass 6418275 2021-10-02 19:39:06 2021-10-02 20:55:23 2021-10-02 21:26:10 0:30:47 0:19:56 0:10:51 smithi master centos 8.3 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} 2
fail 6418276 2021-10-02 19:39:07 2021-10-02 20:57:04 2021-10-02 21:18:20 0:21:16 0:09:48 0:11:28 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

managers

fail 6418277 2021-10-02 19:39:08 2021-10-02 20:57:14 2021-10-02 21:19:26 0:22:12 0:10:23 0:11:49 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
Failure Reason:

managers

pass 6418278 2021-10-02 19:39:09 2021-10-02 20:57:35 2021-10-02 21:32:27 0:34:52 0:27:45 0:07:07 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} 2
pass 6418279 2021-10-02 19:39:10 2021-10-02 20:58:05 2021-10-02 22:05:47 1:07:42 0:57:30 0:10:12 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} 2
pass 6418280 2021-10-02 19:39:11 2021-10-02 20:58:25 2021-10-02 21:22:18 0:23:53 0:12:42 0:11:11 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
fail 6418281 2021-10-02 19:39:12 2021-10-02 20:58:46 2021-10-02 21:12:08 0:13:22 0:06:41 0:06:41 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} 3
Failure Reason:

"2021-10-02T21:10:33.741609+0000 mon.a (mon.0) 157 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6418282 2021-10-02 19:39:13 2021-10-02 20:58:46 2021-10-02 21:21:29 0:22:43 0:10:30 0:12:13 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} 2
Failure Reason:

managers

fail 6418283 2021-10-02 19:39:14 2021-10-02 20:58:46 2021-10-02 21:18:27 0:19:41 0:09:19 0:10:22 smithi master centos 8.stream fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
Failure Reason:

managers

pass 6418284 2021-10-02 19:39:15 2021-10-02 20:59:17 2021-10-02 21:23:37 0:24:20 0:13:22 0:10:58 smithi master centos 8.stream fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 6418285 2021-10-02 19:39:16 2021-10-02 21:00:17 2021-10-02 21:25:18 0:25:01 0:14:27 0:10:34 smithi master centos 8.3 fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs} 2
fail 6418286 2021-10-02 19:39:17 2021-10-02 21:00:18 2021-10-02 21:19:28 0:19:10 0:11:39 0:07:31 smithi master rhel 8.4 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} 2
Failure Reason:

managers

fail 6418287 2021-10-02 19:39:18 2021-10-02 21:00:18 2021-10-02 21:18:01 0:17:43 0:06:49 0:10:54 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} 4
Failure Reason:

managers

fail 6418288 2021-10-02 19:39:19 2021-10-02 21:04:20 2021-10-02 21:23:45 0:19:25 0:12:04 0:07:21 smithi master rhel 8.4 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} 2
Failure Reason:

managers

pass 6418289 2021-10-02 19:39:20 2021-10-02 21:04:21 2021-10-02 21:27:30 0:23:09 0:13:29 0:09:40 smithi master centos 8.stream fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
fail 6418290 2021-10-02 19:39:21 2021-10-02 21:04:21 2021-10-02 21:19:19 0:14:58 0:07:00 0:07:58 smithi master ubuntu 20.04 fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/lockdep} 2
Failure Reason:

"2021-10-02T21:17:36.451675+0000 mon.a (mon.0) 181 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log

pass 6418291 2021-10-02 19:39:22 2021-10-02 21:05:12 2021-10-02 21:28:33 0:23:21 0:15:53 0:07:28 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
pass 6418292 2021-10-02 19:39:23 2021-10-02 21:05:42 2021-10-02 21:27:14 0:21:32 0:14:43 0:06:49 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} 2
fail 6418293 2021-10-02 19:39:24 2021-10-02 21:06:23 2021-10-02 21:28:15 0:21:52 0:10:24 0:11:28 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

managers

pass 6418294 2021-10-02 19:39:25 2021-10-02 21:07:04 2021-10-02 21:31:34 0:24:30 0:13:13 0:11:17 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} 2
fail 6418295 2021-10-02 19:39:26 2021-10-02 21:07:54 2021-10-02 21:29:36 0:21:42 0:10:02 0:11:40 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} 2
Failure Reason:

managers

pass 6418296 2021-10-02 19:39:27 2021-10-02 21:08:55 2021-10-02 21:51:55 0:43:00 0:31:19 0:11:41 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
fail 6418297 2021-10-02 19:39:28 2021-10-02 21:10:05 2021-10-02 21:24:25 0:14:20 0:06:56 0:07:24 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
Failure Reason:

"2021-10-02T21:23:09.225167+0000 mon.a (mon.0) 361 : cluster [WRN] Health check failed: Reduced data availability: 4 pgs down (PG_AVAILABILITY)" in cluster log

fail 6418298 2021-10-02 19:39:29 2021-10-02 21:10:26 2021-10-02 21:29:21 0:18:55 0:09:13 0:09:42 smithi master centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment} 2
Failure Reason:

managers

pass 6418299 2021-10-02 19:39:31 2021-10-02 21:10:28 2021-10-02 21:34:05 0:23:37 0:13:19 0:10:18 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
fail 6418300 2021-10-02 19:39:32 2021-10-02 21:10:29 2021-10-02 21:28:58 0:18:29 0:06:35 0:11:54 smithi master centos 8.3 fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

Command failed on smithi129 with status 1: 'sudo yum -y install ceph-volume'

pass 6418301 2021-10-02 19:39:33 2021-10-02 21:11:39 2021-10-02 22:42:58 1:31:19 1:24:18 0:07:01 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 6418302 2021-10-02 19:39:34 2021-10-02 21:12:10 2021-10-02 21:31:32 0:19:22 0:12:08 0:07:14 smithi master ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} 2
fail 6418303 2021-10-02 19:39:36 2021-10-02 21:12:13 2021-10-02 21:33:08 0:20:55 0:10:18 0:10:37 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
Failure Reason:

managers

pass 6418304 2021-10-02 19:39:37 2021-10-02 21:12:14 2021-10-02 21:38:20 0:26:06 0:19:36 0:06:30 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair} 2
fail 6418305 2021-10-02 19:39:38 2021-10-02 21:12:14 2021-10-02 21:31:54 0:19:40 0:11:34 0:08:06 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
Failure Reason:

managers

pass 6418306 2021-10-02 19:39:39 2021-10-02 21:12:45 2021-10-02 21:36:23 0:23:38 0:12:54 0:10:44 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
fail 6418307 2021-10-02 19:39:40 2021-10-02 21:12:45 2021-10-02 21:34:55 0:22:10 0:09:04 0:13:06 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
Failure Reason:

managers

fail 6418308 2021-10-02 19:39:41 2021-10-02 21:13:45 2021-10-02 21:42:40 0:28:55 0:22:08 0:06:47 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} 2
Failure Reason:

reached maximum tries (90) after waiting for 540 seconds

fail 6418309 2021-10-02 19:39:42 2021-10-02 21:14:06 2021-10-02 21:39:47 0:25:41 0:10:03 0:15:38 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} 2
Failure Reason:

managers

pass 6418310 2021-10-02 19:39:42 2021-10-02 21:15:56 2021-10-02 21:44:37 0:28:41 0:21:05 0:07:36 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
fail 6418311 2021-10-02 19:39:43 2021-10-02 21:17:37 2021-10-02 21:39:05 0:21:28 0:09:59 0:11:29 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
Failure Reason:

managers

fail 6418312 2021-10-02 19:39:45 2021-10-02 21:18:07 2021-10-02 21:39:28 0:21:21 0:10:02 0:11:19 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} 3
Failure Reason:

managers

pass 6418313 2021-10-02 19:39:46 2021-10-02 21:18:28 2021-10-02 21:49:57 0:31:29 0:20:05 0:11:24 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} 2
pass 6418314 2021-10-02 19:39:47 2021-10-02 21:18:28 2021-10-02 21:40:01 0:21:33 0:11:27 0:10:06 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
dead 6418315 2021-10-02 19:39:48 2021-10-02 21:18:29 2021-10-02 22:25:47 1:07:18 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} 3
Failure Reason:

Error reimaging machines: Timeout opening channel.

fail 6418316 2021-10-02 19:39:49 2021-10-02 21:19:30 2021-10-02 21:38:27 0:18:57 0:09:34 0:09:23 smithi master centos 8.stream fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
Failure Reason:

managers

fail 6418317 2021-10-02 19:39:50 2021-10-02 21:19:30 2021-10-02 21:39:00 0:19:30 0:08:34 0:10:56 smithi master centos 8.3 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
Failure Reason:

managers

fail 6418318 2021-10-02 19:39:51 2021-10-02 21:19:30 2021-10-02 21:33:16 0:13:46 0:06:43 0:07:03 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
Failure Reason:

managers

fail 6418319 2021-10-02 19:39:52 2021-10-02 21:19:41 2021-10-02 21:39:19 0:19:38 0:08:17 0:11:21 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} 2
Failure Reason:

managers

fail 6418320 2021-10-02 19:39:53 2021-10-02 21:20:21 2021-10-02 21:40:12 0:19:51 0:11:52 0:07:59 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry} 2
Failure Reason:

managers

pass 6418321 2021-10-02 19:39:54 2021-10-02 21:21:32 2021-10-02 22:14:59 0:53:27 0:42:27 0:11:00 smithi master centos 8.3 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
pass 6418322 2021-10-02 19:39:55 2021-10-02 21:21:52 2021-10-02 21:45:37 0:23:45 0:17:00 0:06:45 smithi master ubuntu 20.04 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} 2
pass 6418323 2021-10-02 19:39:56 2021-10-02 21:21:53 2021-10-02 22:03:18 0:41:25 0:32:03 0:09:22 smithi master centos 8.3 fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 6418324 2021-10-02 19:39:57 2021-10-02 21:22:23 2021-10-02 22:19:58 0:57:35 0:50:36 0:06:59 smithi master rhel 8.4 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} 2
pass 6418325 2021-10-02 19:39:58 2021-10-02 21:22:34 2021-10-02 21:47:41 0:25:07 0:18:51 0:06:16 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
pass 6418326 2021-10-02 19:39:59 2021-10-02 21:22:44 2021-10-02 21:47:03 0:24:19 0:15:54 0:08:25 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
fail 6418327 2021-10-02 19:40:00 2021-10-02 21:23:45 2021-10-02 21:45:45 0:22:00 0:10:02 0:11:58 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
Failure Reason:

managers

fail 6418328 2021-10-02 19:40:01 2021-10-02 21:24:35 2021-10-02 21:53:48 0:29:13 0:22:45 0:06:28 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} 2
Failure Reason:

"2021-10-02T21:43:40.103283+0000 mon.a (mon.0) 357 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

pass 6418329 2021-10-02 19:40:02 2021-10-02 21:24:35 2021-10-02 22:10:03 0:45:28 0:33:24 0:12:04 smithi master centos 8.3 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
pass 6418330 2021-10-02 19:40:04 2021-10-02 21:24:56 2021-10-02 21:41:58 0:17:02 0:09:23 0:07:39 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} 5
fail 6418331 2021-10-02 19:40:05 2021-10-02 21:26:16 2021-10-02 22:02:24 0:36:08 0:28:15 0:07:53 smithi master rhel 8.4 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} 2
Failure Reason:

Test failure: test_simple (tasks.cephfs.test_failover.TestFailover)

pass 6418332 2021-10-02 19:40:06 2021-10-02 21:27:07 2021-10-02 21:48:06 0:20:59 0:15:01 0:05:58 smithi master rhel 8.4 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} 2
pass 6418333 2021-10-02 19:40:07 2021-10-02 21:27:17 2021-10-02 22:31:28 1:04:11 0:53:59 0:10:12 smithi master ubuntu 20.04 fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/valgrind} 2
fail 6418334 2021-10-02 19:40:08 2021-10-02 21:27:38 2021-10-02 21:46:18 0:18:40 0:08:02 0:10:38 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
Failure Reason:

"2021-10-02T21:45:04.485962+0000 mon.a (mon.0) 350 : cluster [WRN] Health check failed: Degraded data redundancy: 3/48 objects degraded (6.250%), 2 pgs degraded (PG_DEGRADED)" in cluster log

pass 6418335 2021-10-02 19:40:09 2021-10-02 21:28:18 2021-10-02 22:19:48 0:51:30 0:39:24 0:12:06 smithi master centos 8.stream fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
fail 6418336 2021-10-02 19:40:10 2021-10-02 21:28:38 2021-10-02 21:50:30 0:21:52 0:10:09 0:11:43 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
Failure Reason:

managers

fail 6418337 2021-10-02 19:40:11 2021-10-02 21:28:39 2021-10-02 22:00:04 0:31:25 0:10:16 0:21:09 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} 2
Failure Reason:

managers

fail 6418338 2021-10-02 19:40:12 2021-10-02 21:29:09 2021-10-02 21:48:30 0:19:21 0:12:10 0:07:11 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
Failure Reason:

managers

fail 6418339 2021-10-02 19:40:13 2021-10-02 21:29:09 2021-10-02 21:56:50 0:27:41 0:17:51 0:09:50 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} 2
Failure Reason:

reached maximum tries (90) after waiting for 540 seconds

dead 6418340 2021-10-02 19:40:14 2021-10-02 21:29:30 2021-10-03 09:47:14 12:17:44 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
Failure Reason:

hit max job timeout

fail 6418341 2021-10-02 19:40:15 2021-10-02 21:30:00 2021-10-02 21:53:38 0:23:38 0:11:17 0:12:21 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} 2
Failure Reason:

Test failure: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable)

fail 6418342 2021-10-02 19:40:16 2021-10-02 21:31:21 2021-10-02 21:50:38 0:19:17 0:09:28 0:09:49 smithi master centos 8.stream fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
Failure Reason:

managers

fail 6418343 2021-10-02 19:40:17 2021-10-02 21:31:31 2021-10-02 21:49:22 0:17:51 0:07:49 0:10:02 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
Failure Reason:

"2021-10-02T21:48:15.220234+0000 mon.a (mon.0) 254 : cluster [WRN] Health check failed: Degraded data redundancy: 3/48 objects degraded (6.250%), 2 pgs degraded (PG_DEGRADED)" in cluster log

pass 6418344 2021-10-02 19:40:18 2021-10-02 21:31:41 2021-10-02 22:33:54 1:02:13 0:55:35 0:06:38 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} 3
fail 6418345 2021-10-02 19:40:19 2021-10-02 21:32:02 2021-10-02 21:49:02 0:17:00 0:06:35 0:10:25 smithi master centos 8.3 fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

Command failed on smithi007 with status 1: 'sudo yum -y install ceph-volume'

pass 6418346 2021-10-02 19:40:20 2021-10-02 21:32:32 2021-10-02 21:53:57 0:21:25 0:14:53 0:06:32 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} 2
fail 6418347 2021-10-02 19:40:21 2021-10-02 21:33:13 2021-10-02 21:46:49 0:13:36 0:06:35 0:07:01 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} 2
Failure Reason:

managers

fail 6418348 2021-10-02 19:40:22 2021-10-02 21:33:23 2021-10-02 21:53:23 0:20:00 0:12:33 0:07:27 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
Failure Reason:

managers

fail 6418349 2021-10-02 19:40:23 2021-10-02 21:34:14 2021-10-02 21:57:50 0:23:36 0:09:49 0:13:47 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} 2
Failure Reason:

managers

pass 6418350 2021-10-02 19:40:24 2021-10-02 21:35:04 2021-10-02 21:58:10 0:23:06 0:13:35 0:09:31 smithi master centos 8.stream fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 6418351 2021-10-02 19:40:25 2021-10-02 21:35:05 2021-10-02 21:59:38 0:24:33 0:12:22 0:12:11 smithi master centos 8.stream fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8.stream} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python} 2
pass 6418352 2021-10-02 19:40:25 2021-10-02 21:36:25 2021-10-02 21:55:18 0:18:53 0:10:42 0:08:11 smithi master ubuntu 20.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 6418353 2021-10-02 19:40:26 2021-10-02 21:37:56 2021-10-02 22:05:45 0:27:49 0:16:08 0:11:41 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
pass 6418354 2021-10-02 19:40:27 2021-10-02 21:38:36 2021-10-02 22:00:07 0:21:31 0:13:51 0:07:40 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota} 2
fail 6418355 2021-10-02 19:40:28 2021-10-02 21:39:07 2021-10-02 22:07:08 0:28:01 0:17:37 0:10:24 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} 2
Failure Reason:

reached maximum tries (90) after waiting for 540 seconds

fail 6418356 2021-10-02 19:40:30 2021-10-02 21:39:07 2021-10-02 21:57:43 0:18:36 0:07:41 0:10:55 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
Failure Reason:

"2021-10-02T21:57:01.823721+0000 mon.a (mon.0) 332 : cluster [WRN] Health check failed: Degraded data redundancy: 4/48 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED)" in cluster log

pass 6418357 2021-10-02 19:40:31 2021-10-02 21:39:28 2021-10-02 21:56:58 0:17:30 0:11:11 0:06:19 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} 3
pass 6418358 2021-10-02 19:40:32 2021-10-02 21:39:38 2021-10-02 23:39:39 2:00:01 1:43:18 0:16:43 smithi master centos 8.3 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}} 2
fail 6418359 2021-10-02 19:40:33 2021-10-02 21:39:49 2021-10-02 22:03:08 0:23:19 0:10:09 0:13:10 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs} 2
Failure Reason:

managers

fail 6418360 2021-10-02 19:40:34 2021-10-02 21:39:49 2021-10-02 21:59:32 0:19:43 0:12:42 0:07:01 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} 3
Failure Reason:

managers

fail 6418361 2021-10-02 19:40:35 2021-10-02 21:40:19 2021-10-02 22:00:43 0:20:24 0:11:54 0:08:30 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

managers

pass 6418362 2021-10-02 19:40:36 2021-10-02 21:41:40 2021-10-02 22:37:11 0:55:31 0:42:11 0:13:20 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
fail 6418363 2021-10-02 19:40:37 2021-10-02 21:42:00 2021-10-02 22:09:41 0:27:41 0:16:30 0:11:11 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} 2
Failure Reason:

Test failure: test_scrub_task_status_on_mds_failover (tasks.cephfs.test_scrub_checks.TestScrubControls)

pass 6418364 2021-10-02 19:40:38 2021-10-02 21:42:01 2021-10-02 22:51:46 1:09:45 1:01:57 0:07:48 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} 2
fail 6418365 2021-10-02 19:40:39 2021-10-02 21:42:11 2021-10-02 21:59:54 0:17:43 0:07:50 0:09:53 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
Failure Reason:

"2021-10-02T21:58:39.549749+0000 mon.a (mon.0) 264 : cluster [WRN] Health check failed: Degraded data redundancy: 4/48 objects degraded (8.333%), 3 pgs degraded (PG_DEGRADED)" in cluster log

fail 6418366 2021-10-02 19:40:40 2021-10-02 21:42:31 2021-10-02 22:01:27 0:18:56 0:12:07 0:06:49 smithi master rhel 8.4 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

managers

fail 6418367 2021-10-02 19:40:41 2021-10-02 21:42:42 2021-10-02 22:03:51 0:21:09 0:08:08 0:13:01 smithi master centos 8.3 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} 2
Failure Reason:

managers

pass 6418368 2021-10-02 19:40:42 2021-10-02 21:44:42 2021-10-02 22:31:02 0:46:20 0:36:06 0:10:14 smithi master centos 8.3 fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.3_container_tools_3.0 conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 6418369 2021-10-02 19:40:43 2021-10-02 21:45:43 2021-10-02 22:31:15 0:45:32 0:38:22 0:07:10 smithi master rhel 8.4 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} 2
pass 6418370 2021-10-02 19:40:44 2021-10-02 21:45:53 2021-10-02 22:04:23 0:18:30 0:10:34 0:07:56 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
fail 6418371 2021-10-02 19:40:45 2021-10-02 21:46:23 2021-10-02 22:07:17 0:20:54 0:09:59 0:10:55 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
Failure Reason:

managers

fail 6418372 2021-10-02 19:40:46 2021-10-02 21:46:24 2021-10-02 22:03:48 0:17:24 0:11:27 0:05:57 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap} 2
Failure Reason:

managers

fail 6418373 2021-10-02 19:40:47 2021-10-02 21:46:54 2021-10-02 22:06:50 0:19:56 0:07:42 0:12:14 smithi master ubuntu 20.04 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} 2
Failure Reason:

"2021-10-02T22:04:12.595711+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: Degraded data redundancy: 9/48 objects degraded (18.750%), 4 pgs degraded (PG_DEGRADED)" in cluster log

fail 6418374 2021-10-02 19:40:48 2021-10-02 21:47:04 2021-10-02 22:01:18 0:14:14 0:06:21 0:07:53 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} 4
Failure Reason:

managers

fail 6418375 2021-10-02 19:40:49 2021-10-02 21:47:45 2021-10-02 22:11:04 0:23:19 0:09:58 0:13:21 smithi master centos 8.3 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} 2
Failure Reason:

managers

fail 6418376 2021-10-02 19:40:50 2021-10-02 21:48:15 2021-10-02 22:08:00 0:19:45 0:11:32 0:08:13 smithi master rhel 8.4 fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/lockdep} 2
Failure Reason:

managers

fail 6418377 2021-10-02 19:40:51 2021-10-02 21:48:36 2021-10-02 22:08:21 0:19:45 0:12:37 0:07:08 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
Failure Reason:

managers

pass 6418378 2021-10-02 19:40:52 2021-10-02 21:49:27 2021-10-02 22:11:45 0:22:18 0:10:58 0:11:20 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} 2
fail 6418379 2021-10-02 19:40:53 2021-10-02 21:50:08 2021-10-02 22:38:25 0:48:17 0:35:34 0:12:43 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

"2021-10-02T22:10:38.488933+0000 mon.a (mon.0) 153 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6418380 2021-10-02 19:40:54 2021-10-02 21:50:38 2021-10-02 22:17:34 0:26:56 0:19:59 0:06:57 smithi master rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} 2
fail 6418381 2021-10-02 19:40:59 2021-10-02 21:50:48 2021-10-02 22:09:04 0:18:16 0:11:25 0:06:51 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} 2
Failure Reason:

managers

fail 6418382 2021-10-02 19:41:00 2021-10-02 21:51:59 2021-10-02 22:11:02 0:19:03 0:07:56 0:11:07 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
Failure Reason:

"2021-10-02T22:09:41.708517+0000 mon.a (mon.0) 260 : cluster [WRN] Health check failed: Degraded data redundancy: 2/48 objects degraded (4.167%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6418383 2021-10-02 19:41:01 2021-10-02 21:53:29 2021-10-02 22:19:48 0:26:19 0:19:30 0:06:49 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
pass 6418384 2021-10-02 19:41:02 2021-10-02 21:53:40 2021-10-02 22:17:05 0:23:25 0:16:35 0:06:50 smithi master rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
fail 6418385 2021-10-02 19:41:03 2021-10-02 21:53:50 2021-10-02 22:11:07 0:17:17 0:11:18 0:05:59 smithi master rhel 8.4 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
Failure Reason:

managers

fail 6418386 2021-10-02 19:41:04 2021-10-02 21:54:00 2021-10-02 22:08:49 0:14:49 0:06:09 0:08:40 smithi master ubuntu 20.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
Failure Reason:

managers

fail 6418387 2021-10-02 19:41:05 2021-10-02 21:55:21 2021-10-02 22:19:34 0:24:13 0:10:11 0:14:02 smithi master centos 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} 2
Failure Reason:

managers

fail 6418388 2021-10-02 19:41:06 2021-10-02 21:56:52 2021-10-02 22:16:01 0:19:09 0:12:09 0:07:00 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} 3
Failure Reason:

managers

pass 6418389 2021-10-02 19:41:07 2021-10-02 21:57:02 2021-10-02 22:23:28 0:26:26 0:15:20 0:11:06 smithi master ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}} 2
fail 6418390 2021-10-02 19:41:08 2021-10-02 21:57:22 2021-10-03 01:45:03 3:47:41 3:35:58 0:11:43 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
Failure Reason:

Command failed (workunit test suites/ffsb.sh) on smithi079 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ab8f2f64360119e72bc635c23e114f82b2e1e802 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

fail 6418391 2021-10-02 19:41:09 2021-10-02 21:57:53 2021-10-02 22:19:56 0:22:03 0:10:27 0:11:36 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} 2
Failure Reason:

Test failure: test_dirfrag_limit (tasks.cephfs.test_strays.TestStrays)

fail 6418392 2021-10-02 19:41:10 2021-10-02 21:57:53 2021-10-02 22:19:26 0:21:33 0:09:40 0:11:53 smithi master centos 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
Failure Reason:

managers

fail 6418393 2021-10-02 19:41:11 2021-10-02 21:58:13 2021-10-02 22:17:19 0:19:06 0:08:20 0:10:46 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
Failure Reason:

"2021-10-02T22:16:22.690201+0000 mon.a (mon.0) 253 : cluster [WRN] Health check failed: Degraded data redundancy: 5/48 objects degraded (10.417%), 4 pgs degraded (PG_DEGRADED)" in cluster log

fail 6418394 2021-10-02 19:41:12 2021-10-02 21:59:34 2021-10-02 22:16:41 0:17:07 0:06:41 0:10:26 smithi master centos 8.3 fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} 3
Failure Reason:

Command failed on smithi136 with status 1: 'sudo yum -y install ceph-volume'

pass 6418395 2021-10-02 19:41:13 2021-10-02 21:59:56 2021-10-02 22:23:03 0:23:07 0:17:01 0:06:06 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
pass 6418396 2021-10-02 19:41:14 2021-10-02 22:00:06 2021-10-02 23:05:42 1:05:36 0:56:27 0:09:09 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} 2
fail 6418397 2021-10-02 19:41:15 2021-10-02 22:00:07 2021-10-02 22:14:02 0:13:55 0:05:51 0:08:04 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration} 2
Failure Reason:

managers

fail 6418398 2021-10-02 19:41:16 2021-10-02 22:00:17 2021-10-02 22:19:42 0:19:25 0:12:40 0:06:45 smithi master rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
Failure Reason:

managers

fail 6418399 2021-10-02 19:41:17 2021-10-02 22:00:37 2021-10-02 22:14:37 0:14:00 0:05:56 0:08:04 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
Failure Reason:

managers

pass 6418400 2021-10-02 19:41:18 2021-10-02 22:00:48 2021-10-02 22:26:03 0:25:15 0:14:18 0:10:57 smithi master centos 8.3 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3