Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7429063 2023-10-16 12:48:52 2023-10-18 15:42:44 2023-10-18 16:16:19 0:33:35 0:22:22 0:11:13 smithi main centos 8.stream fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} 2
Failure Reason:

Command failed on smithi029 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

dead 7429064 2023-10-16 12:48:53 2023-10-18 15:43:04 2023-10-18 22:35:41 6:52:37 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}} 3
fail 7429065 2023-10-16 12:48:54 2023-10-18 15:43:55 2023-10-18 16:10:04 0:26:09 0:14:08 0:12:01 smithi main rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
Failure Reason:

"2023-10-18T16:07:15.278845+0000 mon.a (mon.0) 191 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.j" in cluster log

fail 7429066 2023-10-16 12:48:55 2023-10-18 15:46:55 2023-10-18 16:13:45 0:26:50 0:14:11 0:12:39 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} 2
Failure Reason:

Command failed on smithi139 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429067 2023-10-16 12:48:55 2023-10-18 15:48:36 2023-10-18 16:18:06 0:29:30 0:20:32 0:08:58 smithi main rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
Failure Reason:

Command failed on smithi088 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

fail 7429068 2023-10-16 12:48:56 2023-10-18 15:48:36 2023-10-18 16:18:12 0:29:36 0:16:35 0:13:01 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} 2
Failure Reason:

"2023-10-18T16:13:04.775319+0000 mon.a (mon.0) 169 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.c" in cluster log

fail 7429069 2023-10-16 12:48:57 2023-10-18 15:49:37 2023-10-18 16:20:18 0:30:41 0:19:51 0:10:50 smithi main ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} 2
Failure Reason:

"2023-10-18T16:12:51.909741+0000 mon.a (mon.0) 209 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429070 2023-10-16 12:48:58 2023-10-18 15:50:17 2023-10-18 16:19:51 0:29:34 0:16:55 0:12:39 smithi main ubuntu 20.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} 2
Failure Reason:

"2023-10-18T16:15:07.636687+0000 mon.a (mon.0) 170 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.a" in cluster log

fail 7429071 2023-10-16 12:48:58 2023-10-18 15:51:28 2023-10-18 16:25:39 0:34:11 0:18:30 0:15:41 smithi main centos 8.stream fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

"2023-10-18T16:20:19.057781+0000 mon.a (mon.0) 218 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.b" in cluster log

fail 7429072 2023-10-16 12:48:59 2023-10-18 15:54:49 2023-10-18 16:24:13 0:29:24 0:18:35 0:10:49 smithi main rhel 8.4 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} 2
Failure Reason:

"2023-10-18T16:19:44.859159+0000 mon.a (mon.0) 167 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429073 2023-10-16 12:49:00 2023-10-18 15:54:49 2023-10-18 16:25:54 0:31:05 0:18:22 0:12:43 smithi main centos 8.stream fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client} 2
Failure Reason:

"2023-10-18T16:21:19.711265+0000 mon.a (mon.0) 229 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.c" in cluster log

fail 7429074 2023-10-16 12:49:01 2023-10-18 15:56:40 2023-10-18 16:27:56 0:31:16 0:14:51 0:16:25 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} 2
Failure Reason:

Command failed on smithi049 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'

fail 7429075 2023-10-16 12:49:02 2023-10-18 15:59:20 2023-10-18 16:30:10 0:30:50 0:17:25 0:13:25 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

Command failed on smithi178 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0e6a7fb8-6dd3-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json'

fail 7429076 2023-10-16 12:49:02 2023-10-18 16:00:31 2023-10-18 16:28:01 0:27:30 0:15:07 0:12:23 smithi main centos 8.stream fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
Failure Reason:

Command failed on smithi012 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'

fail 7429077 2023-10-16 12:49:03 2023-10-18 16:03:02 2023-10-18 16:28:20 0:25:18 0:14:31 0:10:47 smithi main centos 8.stream fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} 2
Failure Reason:

Command failed on smithi026 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e'

fail 7429078 2023-10-16 12:49:04 2023-10-18 16:03:22 2023-10-18 16:28:33 0:25:11 0:15:59 0:09:12 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} 2
Failure Reason:

"2023-10-18T16:23:14.543671+0000 mon.a (mon.0) 232 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.d" in cluster log

fail 7429079 2023-10-16 12:49:05 2023-10-18 16:03:22 2023-10-18 16:34:21 0:30:59 0:19:00 0:11:59 smithi main centos 8.stream fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} 2
Failure Reason:

Command failed on smithi135 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f'

fail 7429080 2023-10-16 12:49:05 2023-10-18 16:05:13 2023-10-18 16:35:01 0:29:48 0:17:16 0:12:32 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} 2
Failure Reason:

Command failed on smithi167 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429081 2023-10-16 12:49:06 2023-10-18 16:05:34 2023-10-18 16:36:10 0:30:36 0:16:19 0:14:17 smithi main ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
Failure Reason:

"2023-10-18T16:31:15.915985+0000 mon.a (mon.0) 169 : cluster [WRN] Replacing daemon mds.e as rank 0 with standby daemon mds.c" in cluster log

fail 7429082 2023-10-16 12:49:07 2023-10-18 16:08:04 2023-10-18 16:35:20 0:27:16 0:16:54 0:10:22 smithi main rhel 8.4 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
Failure Reason:

"2023-10-18T16:33:22.650564+0000 mon.a (mon.0) 169 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429083 2023-10-16 12:49:08 2023-10-18 16:08:05 2023-10-18 16:41:05 0:33:00 0:21:41 0:11:19 smithi main centos 8.stream fs/bugs/client_trim_caps/{begin centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} 1
Failure Reason:

"2023-10-18T16:32:30.246944+0000 mon.a (mon.0) 143 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429084 2023-10-16 12:49:09 2023-10-18 16:08:05 2023-10-18 16:53:46 0:45:41 0:32:59 0:12:42 smithi main centos 8.stream fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest} 2
Failure Reason:

Timed out waiting for MDS daemons to become healthy

fail 7429085 2023-10-16 12:49:09 2023-10-18 16:08:05 2023-10-18 16:40:44 0:32:39 0:21:38 0:11:01 smithi main rhel 8.4 fs/full/{begin clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} 1
Failure Reason:

Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 7429086 2023-10-16 12:49:10 2023-10-18 16:10:06 2023-10-18 16:37:58 0:27:52 0:17:37 0:10:15 smithi main ubuntu 20.04 fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} 2
Failure Reason:

"2023-10-18T16:33:01.128309+0000 mon.a (mon.0) 220 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429087 2023-10-16 12:49:11 2023-10-18 16:10:06 2023-10-18 16:43:46 0:33:40 0:21:15 0:12:25 smithi main rhel 8.4 fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} 1
Failure Reason:

"2023-10-18T16:34:39.261075+0000 mon.a (mon.0) 145 : cluster [WRN] Replacing daemon mds.e as rank 0 with standby daemon mds.d" in cluster log

fail 7429088 2023-10-16 12:49:12 2023-10-18 16:10:07 2023-10-18 16:45:13 0:35:06 0:24:02 0:11:04 smithi main rhel 8.4 fs/mirror-ha/{begin cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} 1
Failure Reason:

"2023-10-18T16:36:21.846487+0000 mon.a (mon.0) 145 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.e" in cluster log

fail 7429089 2023-10-16 12:49:13 2023-10-18 16:10:07 2023-10-18 16:38:13 0:28:06 0:15:50 0:12:16 smithi main rhel 8.4 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} 2
Failure Reason:

"2023-10-18T16:32:40.164776+0000 mon.a (mon.0) 170 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429090 2023-10-16 12:49:13 2023-10-18 16:13:18 2023-10-18 16:41:40 0:28:22 0:16:41 0:11:41 smithi main ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} 4
Failure Reason:

"2023-10-18T16:36:48.147628+0000 mon.a (mon.0) 175 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.c" in cluster log

fail 7429091 2023-10-16 12:49:14 2023-10-18 16:13:48 2023-10-18 16:45:50 0:32:02 0:19:16 0:12:46 smithi main rhel 8.4 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
Failure Reason:

"2023-10-18T16:41:53.767155+0000 mon.a (mon.0) 165 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429092 2023-10-16 12:49:15 2023-10-18 16:16:29 2023-10-18 16:45:54 0:29:25 0:16:50 0:12:35 smithi main ubuntu 20.04 fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} 2
Failure Reason:

"2023-10-18T16:41:17.739943+0000 mon.a (mon.0) 173 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429093 2023-10-16 12:49:16 2023-10-18 16:17:39 2023-10-18 16:49:28 0:31:49 0:19:10 0:12:39 smithi main ubuntu 20.04 fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} 1
Failure Reason:

"2023-10-18T16:40:07.047893+0000 mon.a (mon.0) 145 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

dead 7429094 2023-10-16 12:49:17 2023-10-18 16:17:40 2023-10-18 22:34:11 6:16:31 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7429095 2023-10-16 12:49:18 2023-10-18 16:18:20 2023-10-18 17:01:36 0:43:16 0:32:04 0:11:12 smithi main centos 8.stream fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} 1
Failure Reason:

"2023-10-18T16:53:59.919084+0000 mon.a (mon.0) 142 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.e" in cluster log

fail 7429096 2023-10-16 12:49:18 2023-10-18 16:18:21 2023-10-18 16:46:14 0:27:53 0:15:13 0:12:40 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} 2
Failure Reason:

"2023-10-18T16:43:17.254122+0000 mon.a (mon.0) 165 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429097 2023-10-16 12:49:19 2023-10-18 16:20:01 2023-10-18 16:57:55 0:37:54 0:21:44 0:16:10 smithi main ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
Failure Reason:

"2023-10-18T16:46:31.894368+0000 mon.a (mon.0) 213 : cluster [WRN] Replacing daemon mds.l as rank 0 with standby daemon mds.g" in cluster log

fail 7429098 2023-10-16 12:49:20 2023-10-18 16:24:02 2023-10-18 16:56:56 0:32:54 0:22:36 0:10:18 smithi main rhel 8.4 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} 2
Failure Reason:

Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 7429099 2023-10-16 12:49:21 2023-10-18 16:24:02 2023-10-18 16:49:34 0:25:32 0:15:24 0:10:08 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} 2
Failure Reason:

Command failed on smithi046 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c'

fail 7429100 2023-10-16 12:49:21 2023-10-18 16:24:23 2023-10-18 16:50:38 0:26:15 0:15:13 0:11:02 smithi main rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} 2
Failure Reason:

Command failed on smithi045 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

dead 7429101 2023-10-16 12:49:22 2023-10-18 16:25:43 2023-10-18 22:35:34 6:09:51 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7429102 2023-10-16 12:49:23 2023-10-18 16:26:04 2023-10-18 16:53:02 0:26:58 0:15:27 0:11:31 smithi main rhel 8.4 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} 2
Failure Reason:

"2023-10-18T16:47:27.804986+0000 mon.a (mon.0) 173 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.d" in cluster log

fail 7429103 2023-10-16 12:49:24 2023-10-18 16:28:05 2023-10-18 16:50:58 0:22:53 0:13:45 0:09:08 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} 2
Failure Reason:

"2023-10-18T16:47:49.868173+0000 mon.a (mon.0) 171 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.b" in cluster log

fail 7429104 2023-10-16 12:49:24 2023-10-18 16:28:05 2023-10-18 16:53:19 0:25:14 0:16:52 0:08:22 smithi main rhel 8.4 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} 2
Failure Reason:

"2023-10-18T16:48:18.009594+0000 mon.a (mon.0) 213 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.d" in cluster log

fail 7429105 2023-10-16 12:49:25 2023-10-18 16:28:26 2023-10-18 17:01:10 0:32:44 0:15:42 0:17:02 smithi main centos 8.stream fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
Failure Reason:

Command failed on smithi179 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i l'

fail 7429106 2023-10-16 12:49:26 2023-10-18 16:28:36 2023-10-18 16:59:01 0:30:25 0:14:50 0:15:35 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} 2
Failure Reason:

Command failed on smithi155 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

fail 7429107 2023-10-16 12:49:27 2023-10-18 16:29:37 2023-10-18 16:59:36 0:29:59 0:20:59 0:09:00 smithi main rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} 2
Failure Reason:

"2023-10-18T16:50:27.960046+0000 mon.a (mon.0) 236 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.e" in cluster log

fail 7429108 2023-10-16 12:49:27 2023-10-18 16:30:17 2023-10-18 17:00:30 0:30:13 0:14:02 0:16:11 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} 2
Failure Reason:

"2023-10-18T16:57:03.634898+0000 mon.a (mon.0) 233 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.b" in cluster log

pass 7429109 2023-10-16 12:49:28 2023-10-18 16:34:28 2023-10-18 17:05:19 0:30:51 0:17:52 0:12:59 smithi main centos 8.stream fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} 1
fail 7429110 2023-10-16 12:49:29 2023-10-18 16:34:29 2023-10-18 16:56:41 0:22:12 0:12:31 0:09:41 smithi main rhel 8.4 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} 2
Failure Reason:

Command failed on smithi072 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c'

fail 7429111 2023-10-16 12:49:30 2023-10-18 16:35:09 2023-10-18 16:56:44 0:21:35 0:11:34 0:10:01 smithi main rhel 8.4 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

Command failed on smithi148 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

fail 7429112 2023-10-16 12:49:31 2023-10-18 16:35:30 2023-10-18 17:07:31 0:32:01 0:17:26 0:14:35 smithi main centos 8.stream fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} 2
Failure Reason:

Command failed on smithi186 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429113 2023-10-16 12:49:31 2023-10-18 16:36:20 2023-10-18 17:06:54 0:30:34 0:15:04 0:15:30 smithi main centos 8.stream fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

Command failed on smithi062 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429114 2023-10-16 12:49:32 2023-10-18 16:38:01 2023-10-18 17:01:58 0:23:57 0:12:58 0:10:59 smithi main ubuntu 20.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

"2023-10-18T17:00:04.307408+0000 mon.a (mon.0) 278 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs down (PG_AVAILABILITY)" in cluster log

fail 7429115 2023-10-16 12:49:33 2023-10-18 16:38:21 2023-10-18 17:20:04 0:41:43 0:20:40 0:21:03 smithi main centos 8.stream fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} 2
Failure Reason:

"2023-10-18T17:14:25.303947+0000 mon.a (mon.0) 261 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.d" in cluster log

fail 7429116 2023-10-16 12:49:34 2023-10-18 16:41:12 2023-10-18 17:10:35 0:29:23 0:19:24 0:09:59 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} 2
Failure Reason:

Command failed on smithi139 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429117 2023-10-16 12:49:35 2023-10-18 16:41:42 2023-10-18 17:07:25 0:25:43 0:13:46 0:11:57 smithi main rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
Failure Reason:

"2023-10-18T17:03:46.949582+0000 mon.a (mon.0) 192 : cluster [WRN] Replacing daemon mds.f as rank 0 with standby daemon mds.i" in cluster log

fail 7429118 2023-10-16 12:49:35 2023-10-18 16:43:53 2023-10-18 17:12:45 0:28:52 0:14:48 0:14:04 smithi main centos 8.stream fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} 2
Failure Reason:

Command failed on smithi105 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e'

fail 7429119 2023-10-16 12:49:36 2023-10-18 16:45:54 2023-10-18 17:09:06 0:23:12 0:15:18 0:07:54 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} 2
Failure Reason:

"2023-10-18T17:05:07.090888+0000 mon.a (mon.0) 225 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.c" in cluster log

fail 7429120 2023-10-16 12:49:37 2023-10-18 16:46:04 2023-10-18 17:17:47 0:31:43 0:16:07 0:15:36 smithi main centos 8.stream fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
Failure Reason:

Command failed on smithi154 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429121 2023-10-16 12:49:38 2023-10-18 16:46:24 2023-10-18 17:16:52 0:30:28 0:13:27 0:17:01 smithi main ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} 5
Failure Reason:

"2023-10-18T17:15:02.187202+0000 mon.a (mon.0) 249 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs down (PG_AVAILABILITY)" in cluster log

pass 7429122 2023-10-16 12:49:38 2023-10-18 16:51:46 2023-10-18 17:29:59 0:38:13 0:18:35 0:19:38 smithi main centos 8.stream fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} 2
fail 7429123 2023-10-16 12:49:39 2023-10-18 16:53:07 2023-10-18 17:24:58 0:31:51 0:18:02 0:13:49 smithi main ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} 2
Failure Reason:

Command failed on smithi191 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429124 2023-10-16 12:49:40 2023-10-18 16:53:27 2023-10-18 17:20:09 0:26:42 0:14:46 0:11:56 smithi main ubuntu 20.04 fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} 2
Failure Reason:

"2023-10-18T17:16:39.398184+0000 mon.a (mon.0) 171 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429125 2023-10-16 12:49:41 2023-10-18 16:53:48 2023-10-18 17:32:22 0:38:34 0:18:58 0:19:36 smithi main centos 8.stream fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} 2
Failure Reason:

"2023-10-18T17:26:54.867228+0000 mon.a (mon.0) 170 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.c" in cluster log

fail 7429126 2023-10-16 12:49:41 2023-10-18 16:56:49 2023-10-18 17:27:52 0:31:03 0:15:13 0:15:50 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} 2
Failure Reason:

Command failed on smithi148 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

fail 7429127 2023-10-16 12:49:42 2023-10-18 16:56:49 2023-10-18 17:26:07 0:29:18 0:15:17 0:14:01 smithi main centos 8.stream fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} 3
Failure Reason:

Command failed on smithi033 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

fail 7429128 2023-10-16 12:49:43 2023-10-18 16:56:59 2023-10-18 17:34:40 0:37:41 0:21:16 0:16:25 smithi main ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

"2023-10-18T17:24:50.516443+0000 mon.a (mon.0) 179 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.d" in cluster log

fail 7429129 2023-10-16 12:49:44 2023-10-18 16:57:50 2023-10-18 17:28:02 0:30:12 0:16:29 0:13:43 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} 2
Failure Reason:

"2023-10-18T17:22:09.937651+0000 mon.a (mon.0) 167 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.c" in cluster log

fail 7429130 2023-10-16 12:49:45 2023-10-18 16:58:00 2023-10-18 18:24:47 1:26:47 1:10:51 0:15:56 smithi main ubuntu 18.04 fs/upgrade/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04} 2
Failure Reason:

Command failed on smithi050 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.14-100-gb4ddd52c-1bionic cephadm=16.2.14-100-gb4ddd52c-1bionic ceph-mds=16.2.14-100-gb4ddd52c-1bionic ceph-mgr=16.2.14-100-gb4ddd52c-1bionic ceph-common=16.2.14-100-gb4ddd52c-1bionic ceph-fuse=16.2.14-100-gb4ddd52c-1bionic ceph-test=16.2.14-100-gb4ddd52c-1bionic radosgw=16.2.14-100-gb4ddd52c-1bionic python3-rados=16.2.14-100-gb4ddd52c-1bionic python3-rgw=16.2.14-100-gb4ddd52c-1bionic python3-cephfs=16.2.14-100-gb4ddd52c-1bionic python3-rbd=16.2.14-100-gb4ddd52c-1bionic libcephfs2=16.2.14-100-gb4ddd52c-1bionic libcephfs-dev=16.2.14-100-gb4ddd52c-1bionic librados2=16.2.14-100-gb4ddd52c-1bionic librbd1=16.2.14-100-gb4ddd52c-1bionic rbd-fuse=16.2.14-100-gb4ddd52c-1bionic'

fail 7429131 2023-10-16 12:49:45 2023-10-18 16:59:11 2023-10-18 17:27:59 0:28:48 0:12:51 0:15:57 smithi main ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} 2
Failure Reason:

Command failed on smithi124 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f'

fail 7429132 2023-10-16 12:49:46 2023-10-18 16:59:41 2023-10-18 17:29:53 0:30:12 0:14:25 0:15:47 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} 2
Failure Reason:

Command failed on smithi070 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c'

fail 7429133 2023-10-16 12:49:47 2023-10-18 17:00:32 2023-10-18 17:30:18 0:29:46 0:18:58 0:10:48 smithi main rhel 8.4 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
Failure Reason:

"2023-10-18T17:26:13.700350+0000 mon.a (mon.0) 169 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429134 2023-10-16 12:49:48 2023-10-18 17:01:12 2023-10-18 17:34:50 0:33:38 0:17:17 0:16:21 smithi main ubuntu 20.04 fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} 2
Failure Reason:

"2023-10-18T17:28:49.683426+0000 mon.a (mon.0) 166 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429135 2023-10-16 12:49:48 2023-10-18 17:01:13 2023-10-18 17:29:41 0:28:28 0:14:01 0:14:27 smithi main ubuntu 20.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
Failure Reason:

"2023-10-18T17:27:14.937923+0000 mon.a (mon.0) 212 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429136 2023-10-16 12:49:49 2023-10-18 17:02:03 2023-10-18 17:30:45 0:28:42 0:15:46 0:12:56 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} 2
Failure Reason:

"2023-10-18T17:25:25.543888+0000 mon.a (mon.0) 222 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.a" in cluster log

fail 7429137 2023-10-16 12:49:50 2023-10-18 17:05:24 2023-10-18 17:40:02 0:34:38 0:22:35 0:12:03 smithi main rhel 8.4 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} 2
Failure Reason:

"2023-10-18T17:31:36.643951+0000 mon.a (mon.0) 263 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.c" in cluster log

fail 7429138 2023-10-16 12:49:51 2023-10-18 17:07:04 2023-10-18 17:42:06 0:35:02 0:21:48 0:13:14 smithi main ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
Failure Reason:

"2023-10-18T17:31:31.263183+0000 mon.a (mon.0) 217 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.i" in cluster log

fail 7429139 2023-10-16 12:49:51 2023-10-18 17:07:35 2023-10-18 17:36:36 0:29:01 0:14:58 0:14:03 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} 2
Failure Reason:

Command failed on smithi170 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c'

fail 7429140 2023-10-16 12:49:52 2023-10-18 17:07:35 2023-10-18 17:30:22 0:22:47 0:13:09 0:09:38 smithi main rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
Failure Reason:

"2023-10-18T17:28:32.757915+0000 mon.a (mon.0) 176 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.e" in cluster log

dead 7429141 2023-10-16 12:49:53 2023-10-18 17:09:16 2023-10-18 22:35:58 5:26:42 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7429142 2023-10-16 12:49:54 2023-10-18 17:10:36 2023-10-18 17:43:41 0:33:05 0:16:38 0:16:27 smithi main centos 8.stream fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} 2
Failure Reason:

"2023-10-18T17:40:40.091393+0000 mon.a (mon.0) 227 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.d" in cluster log

fail 7429143 2023-10-16 12:49:54 2023-10-18 17:12:47 2023-10-18 17:44:02 0:31:15 0:16:21 0:14:54 smithi main centos 8.stream fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

"2023-10-18T17:40:50.053131+0000 mon.a (mon.0) 167 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429144 2023-10-16 12:49:55 2023-10-18 17:12:57 2023-10-18 17:42:41 0:29:44 0:15:12 0:14:32 smithi main ubuntu 20.04 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} 2
Failure Reason:

"2023-10-18T17:39:57.178709+0000 mon.a (mon.0) 227 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429145 2023-10-16 12:49:56 2023-10-18 17:16:58 2023-10-18 17:44:56 0:27:58 0:16:31 0:11:27 smithi main ubuntu 20.04 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} 2
Failure Reason:

"2023-10-18T17:40:01.451316+0000 mon.a (mon.0) 173 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.a" in cluster log

fail 7429146 2023-10-16 12:49:57 2023-10-18 17:16:59 2023-10-18 17:47:28 0:30:29 0:17:16 0:13:13 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} 2
Failure Reason:

"2023-10-18T17:42:56.587959+0000 mon.a (mon.0) 170 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.a" in cluster log

fail 7429147 2023-10-16 12:49:57 2023-10-18 17:17:49 2023-10-18 17:45:01 0:27:12 0:17:08 0:10:04 smithi main rhel 8.4 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} 2
Failure Reason:

Command failed on smithi007 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c'

fail 7429148 2023-10-16 12:49:58 2023-10-18 17:20:10 2023-10-18 17:45:55 0:25:45 0:14:20 0:11:25 smithi main ubuntu 20.04 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} 2
Failure Reason:

"2023-10-18T17:42:20.643534+0000 mon.a (mon.0) 225 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.b" in cluster log

fail 7429149 2023-10-16 12:49:59 2023-10-18 17:20:10 2023-10-18 17:52:09 0:31:59 0:14:35 0:17:24 smithi main ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} 4
Failure Reason:

"2023-10-18T17:49:05.986029+0000 mon.a (mon.0) 230 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429150 2023-10-16 12:50:00 2023-10-18 17:26:11 2023-10-18 17:59:27 0:33:16 0:15:33 0:17:43 smithi main centos 8.stream fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
Failure Reason:

Command failed on smithi157 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i l'

fail 7429151 2023-10-16 12:50:00 2023-10-18 17:28:02 2023-10-18 17:53:56 0:25:54 0:12:50 0:13:04 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} 2
Failure Reason:

"2023-10-18T17:51:58.618247+0000 mon.a (mon.0) 237 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs down (PG_AVAILABILITY)" in cluster log

fail 7429152 2023-10-16 12:50:01 2023-10-18 17:28:03 2023-10-18 17:57:15 0:29:12 0:14:45 0:14:27 smithi main centos 8.stream fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} 2
Failure Reason:

Command failed on smithi005 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c'

dead 7429153 2023-10-16 12:50:02 2023-10-18 17:28:03 2023-10-18 22:35:21 5:07:18 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
fail 7429154 2023-10-16 12:50:03 2023-10-18 17:29:44 2023-10-18 17:54:46 0:25:02 0:15:31 0:09:31 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} 2
Failure Reason:

"2023-10-18T17:49:37.010735+0000 mon.a (mon.0) 171 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.c" in cluster log

fail 7429155 2023-10-16 12:50:03 2023-10-18 17:29:44 2023-10-18 18:09:10 0:39:26 0:22:23 0:17:03 smithi main centos 8.stream fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} 2
Failure Reason:

"2023-10-18T18:00:46.759997+0000 mon.a (mon.0) 261 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.c" in cluster log

fail 7429156 2023-10-16 12:50:04 2023-10-18 17:29:54 2023-10-18 17:53:59 0:24:05 0:12:39 0:11:26 smithi main ubuntu 20.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

"2023-10-18T17:52:02.599063+0000 mon.a (mon.0) 197 : cluster [WRN] Health check failed: Reduced data availability: 1 pg down (PG_AVAILABILITY)" in cluster log

fail 7429157 2023-10-16 12:50:05 2023-10-18 17:30:05 2023-10-18 18:01:23 0:31:18 0:15:17 0:16:01 smithi main centos 8.stream fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

Command failed on smithi192 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429158 2023-10-16 12:50:06 2023-10-18 17:30:25 2023-10-18 18:15:24 0:44:59 0:29:43 0:15:16 smithi main centos 8.stream fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} 2
Failure Reason:

"2023-10-18T18:13:26.296221+0000 mon.a (mon.0) 172 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs down (PG_AVAILABILITY)" in cluster log

fail 7429159 2023-10-16 12:50:06 2023-10-18 17:30:26 2023-10-18 18:02:03 0:31:37 0:15:22 0:16:15 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} 2
Failure Reason:

Command failed on smithi088 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

fail 7429160 2023-10-16 12:50:07 2023-10-18 17:30:46 2023-10-18 17:58:27 0:27:41 0:13:53 0:13:48 smithi main rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
Failure Reason:

"2023-10-18T17:55:21.291011+0000 mon.a (mon.0) 195 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.f" in cluster log

fail 7429161 2023-10-16 12:50:08 2023-10-18 17:34:47 2023-10-18 18:08:41 0:33:54 0:21:24 0:12:30 smithi main ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} 2
Failure Reason:

Command failed on smithi072 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

dead 7429162 2023-10-16 12:50:09 2023-10-18 17:34:58 2023-10-18 22:34:12 4:59:14 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7429163 2023-10-16 12:50:09 2023-10-18 17:36:38 2023-10-18 18:09:07 0:32:29 0:17:02 0:15:27 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} 2
Failure Reason:

"2023-10-18T18:03:58.971628+0000 mon.a (mon.0) 167 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.a" in cluster log

fail 7429164 2023-10-16 12:50:10 2023-10-18 17:40:09 2023-10-18 18:12:47 0:32:38 0:17:45 0:14:53 smithi main ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} 2
Failure Reason:

Command failed on smithi118 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'

fail 7429165 2023-10-16 12:50:11 2023-10-18 17:42:10 2023-10-18 18:16:48 0:34:38 0:18:52 0:15:46 smithi main centos 8.stream fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} 2
Failure Reason:

"2023-10-18T18:12:17.222809+0000 mon.a (mon.0) 171 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.d" in cluster log

fail 7429166 2023-10-16 12:50:12 2023-10-18 17:42:10 2023-10-18 18:10:05 0:27:55 0:12:49 0:15:06 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} 2
Failure Reason:

"2023-10-18T18:07:37.355596+0000 mon.a (mon.0) 314 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs down (PG_AVAILABILITY)" in cluster log

fail 7429167 2023-10-16 12:50:12 2023-10-18 17:42:51 2023-10-18 18:08:46 0:25:55 0:14:50 0:11:05 smithi main rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
Failure Reason:

Command failed on smithi144 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f'

fail 7429168 2023-10-16 12:50:13 2023-10-18 17:43:51 2023-10-18 18:12:09 0:28:18 0:15:02 0:13:16 smithi main centos 8.stream fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
Failure Reason:

Command failed on smithi007 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i g'

fail 7429169 2023-10-16 12:50:14 2023-10-18 17:45:02 2023-10-18 18:09:58 0:24:56 0:15:51 0:09:05 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} 2
Failure Reason:

"2023-10-18T18:04:34.059899+0000 mon.a (mon.0) 171 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.c" in cluster log

dead 7429170 2023-10-16 12:50:15 2023-10-18 17:45:02 2023-10-18 22:35:27 4:50:25 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}} 3
fail 7429171 2023-10-16 12:50:16 2023-10-18 17:46:03 2023-10-18 18:18:34 0:32:31 0:20:46 0:11:45 smithi main rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
Failure Reason:

"2023-10-18T18:09:21.423705+0000 mon.a (mon.0) 181 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.f" in cluster log

fail 7429172 2023-10-16 12:50:16 2023-10-18 17:47:33 2023-10-18 18:17:08 0:29:35 0:14:30 0:15:05 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} 2
Failure Reason:

Command failed on smithi136 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

fail 7429173 2023-10-16 12:50:17 2023-10-18 17:52:14 2023-10-18 18:18:08 0:25:54 0:12:41 0:13:13 smithi main ubuntu 20.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} 2
Failure Reason:

"2023-10-18T18:16:42.835189+0000 mon.a (mon.0) 306 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs down (PG_AVAILABILITY)" in cluster log

fail 7429174 2023-10-16 12:50:18 2023-10-18 17:52:15 2023-10-18 18:17:58 0:25:43 0:12:41 0:13:02 smithi main ubuntu 20.04 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

"2023-10-18T18:17:07.716753+0000 mon.a (mon.0) 254 : cluster [WRN] Health check failed: Reduced data availability: 6 pgs down (PG_AVAILABILITY)" in cluster log

fail 7429175 2023-10-16 12:50:19 2023-10-18 17:54:05 2023-10-18 18:19:08 0:25:03 0:14:42 0:10:21 smithi main rhel 8.4 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} 2
Failure Reason:

Command failed on smithi177 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429176 2023-10-16 12:50:19 2023-10-18 17:54:06 2023-10-18 18:21:58 0:27:52 0:16:30 0:11:22 smithi main centos 8.stream fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

"2023-10-18T18:19:46.609163+0000 mon.a (mon.0) 175 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429177 2023-10-16 12:50:20 2023-10-18 17:54:56 2023-10-18 18:26:28 0:31:32 0:18:52 0:12:40 smithi main rhel 8.4 fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} 2
Failure Reason:

"2023-10-18T18:21:57.902027+0000 mon.a (mon.0) 171 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429178 2023-10-16 12:50:21 2023-10-18 17:57:17 2023-10-18 18:18:19 0:21:02 0:11:14 0:09:48 smithi main rhel 8.4 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
Failure Reason:

Command failed on smithi183 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429179 2023-10-16 12:50:22 2023-10-18 17:58:37 2023-10-18 18:27:42 0:29:05 0:13:05 0:16:00 smithi main ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} 5
Failure Reason:

"2023-10-18T18:25:28.435994+0000 mon.a (mon.0) 278 : cluster [WRN] Health check failed: Reduced data availability: 1 pg down (PG_AVAILABILITY)" in cluster log

fail 7429180 2023-10-16 12:50:23 2023-10-18 18:01:28 2023-10-18 18:30:24 0:28:56 0:16:50 0:12:06 smithi main ubuntu 20.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

"2023-10-18T18:26:12.705672+0000 mon.a (mon.0) 223 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429181 2023-10-16 12:50:23 2023-10-18 18:02:09 2023-10-18 18:38:27 0:36:18 0:16:08 0:20:10 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} 2
Failure Reason:

"2023-10-18T18:33:11.056709+0000 mon.a (mon.0) 217 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429182 2023-10-16 12:50:24 2023-10-18 18:08:50 2023-10-18 18:38:34 0:29:44 0:17:50 0:11:54 smithi main ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} 2
Failure Reason:

Command failed on smithi179 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429183 2023-10-16 12:50:25 2023-10-18 18:08:50 2023-10-18 18:37:33 0:28:43 0:14:46 0:13:57 smithi main ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} 3
Failure Reason:

"2023-10-18T18:33:44.062447+0000 mon.a (mon.0) 191 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.l" in cluster log

dead 7429184 2023-10-16 12:50:26 2023-10-18 18:09:11 2023-10-18 22:35:01 4:25:50 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7429185 2023-10-16 12:50:27 2023-10-18 18:09:11 2023-10-18 18:34:59 0:25:48 0:15:00 0:10:48 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} 2
Failure Reason:

Command failed on smithi008 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'

fail 7429186 2023-10-16 12:50:27 2023-10-18 18:10:02 2023-10-18 18:36:55 0:26:53 0:14:35 0:12:18 smithi main centos 8.stream fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} 2
Failure Reason:

Command failed on smithi045 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429187 2023-10-16 12:50:28 2023-10-18 18:10:12 2023-10-18 18:36:57 0:26:45 0:15:35 0:11:10 smithi main rhel 8.4 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} 2
Failure Reason:

"2023-10-18T18:31:54.250299+0000 mon.a (mon.0) 227 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.d" in cluster log

fail 7429188 2023-10-16 12:50:29 2023-10-18 18:12:13 2023-10-18 18:38:27 0:26:14 0:15:20 0:10:54 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} 2
Failure Reason:

"2023-10-18T18:33:13.472136+0000 mon.a (mon.0) 233 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429189 2023-10-16 12:50:30 2023-10-18 18:12:53 2023-10-18 18:47:04 0:34:11 0:22:38 0:11:33 smithi main rhel 8.4 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} 2
Failure Reason:

"2023-10-18T18:38:52.036123+0000 mon.a (mon.0) 213 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.c" in cluster log

fail 7429190 2023-10-16 12:50:31 2023-10-18 18:13:54 2023-10-18 18:48:41 0:34:47 0:23:42 0:11:05 smithi main rhel 8.4 fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} 2
Failure Reason:

"2023-10-18T18:39:15.738740+0000 mon.a (mon.0) 172 : cluster [WRN] Replacing daemon mds.e as rank 0 with standby daemon mds.a" in cluster log

fail 7429191 2023-10-16 12:50:31 2023-10-18 18:15:34 2023-10-18 18:42:01 0:26:27 0:14:15 0:12:12 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} 2
Failure Reason:

Command failed on smithi195 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

fail 7429192 2023-10-16 12:50:32 2023-10-18 18:16:55 2023-10-18 18:44:59 0:28:04 0:13:38 0:14:26 smithi main ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
Failure Reason:

"2023-10-18T18:44:10.595524+0000 mon.a (mon.0) 425 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs down (PG_AVAILABILITY)" in cluster log

dead 7429193 2023-10-16 12:50:33 2023-10-18 18:17:15 2023-10-18 22:35:31 4:18:16 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7429194 2023-10-16 12:50:34 2023-10-18 18:18:16 2023-10-18 18:46:10 0:27:54 0:14:04 0:13:50 smithi main ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
Failure Reason:

Command failed on smithi183 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429195 2023-10-16 12:50:35 2023-10-18 18:18:26 2023-10-18 18:46:20 0:27:54 0:16:18 0:11:36 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} 2
Failure Reason:

"2023-10-18T18:41:30.567070+0000 mon.a (mon.0) 216 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.d" in cluster log

fail 7429196 2023-10-16 12:50:35 2023-10-18 18:18:27 2023-10-18 20:26:09 2:07:42 1:57:11 0:10:31 smithi main rhel 8.4 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} 2
Failure Reason:

Command failed on smithi029 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'

fail 7429197 2023-10-16 12:50:36 2023-10-18 18:18:37 2023-10-18 18:44:11 0:25:34 0:14:47 0:10:47 smithi main rhel 8.4 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
Failure Reason:

Command failed on smithi177 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429198 2023-10-16 12:50:37 2023-10-18 18:19:17 2023-10-18 18:48:56 0:29:39 0:15:20 0:14:19 smithi main rhel 8.4 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
Failure Reason:

Command failed on smithi053 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'

fail 7429199 2023-10-16 12:50:38 2023-10-18 18:20:38 2023-10-18 18:47:15 0:26:37 0:15:35 0:11:02 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} 2
Failure Reason:

Command failed on smithi161 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

dead 7429200 2023-10-16 12:50:38 2023-10-18 18:22:09 2023-10-18 22:34:30 4:12:21 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7429201 2023-10-16 12:50:39 2023-10-18 18:23:09 2023-10-18 18:50:43 0:27:34 0:12:34 0:15:00 smithi main ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

Command failed on smithi059 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e'

fail 7429202 2023-10-16 12:50:40 2023-10-18 18:24:40 2023-10-18 18:48:08 0:23:28 0:14:16 0:09:12 smithi main rhel 8.4 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
Failure Reason:

"2023-10-18T18:45:27.895593+0000 mon.a (mon.0) 198 : cluster [WRN] Replacing daemon mds.l as rank 0 with standby daemon mds.c" in cluster log

fail 7429203 2023-10-16 12:50:41 2023-10-18 18:24:50 2023-10-18 18:51:22 0:26:32 0:15:15 0:11:17 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} 2
Failure Reason:

"2023-10-18T18:45:59.408892+0000 mon.a (mon.0) 226 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.a" in cluster log

fail 7429204 2023-10-16 12:50:41 2023-10-18 18:26:21 2023-10-18 18:57:38 0:31:17 0:18:38 0:12:39 smithi main centos 8.stream fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} 2
Failure Reason:

"2023-10-18T18:54:08.682743+0000 mon.a (mon.0) 224 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.c" in cluster log

fail 7429205 2023-10-16 12:50:42 2023-10-18 18:26:31 2023-10-18 18:58:14 0:31:43 0:17:08 0:14:35 smithi main ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} 4
Failure Reason:

"2023-10-18T18:51:50.901266+0000 mon.a (mon.0) 225 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.c" in cluster log

fail 7429206 2023-10-16 12:50:43 2023-10-18 18:28:37 2023-10-18 19:09:41 0:41:04 0:29:56 0:11:08 smithi main centos 8.stream fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} 2
Failure Reason:

"2023-10-18T19:01:12.910728+0000 mon.a (mon.0) 258 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.d" in cluster log

fail 7429207 2023-10-16 12:50:44 2023-10-18 18:28:37 2023-10-18 18:58:25 0:29:48 0:16:30 0:13:18 smithi main ubuntu 20.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} 2
Failure Reason:

"2023-10-18T18:53:34.954760+0000 mon.a (mon.0) 221 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.c" in cluster log

fail 7429208 2023-10-16 12:50:44 2023-10-18 18:30:28 2023-10-18 19:03:01 0:32:33 0:16:44 0:15:49 smithi main ubuntu 20.04 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

"2023-10-18T18:57:17.520253+0000 mon.a (mon.0) 153 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.b" in cluster log

fail 7429209 2023-10-16 12:50:45 2023-10-18 18:35:09 2023-10-18 19:04:27 0:29:18 0:15:57 0:13:21 smithi main centos 8.stream fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} 2
Failure Reason:

Command failed on smithi007 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'

fail 7429210 2023-10-16 12:50:46 2023-10-18 18:37:00 2023-10-18 19:05:38 0:28:38 0:18:37 0:10:01 smithi main centos 8.stream fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volume-client} 2
Failure Reason:

"2023-10-18T19:03:42.398588+0000 mon.a (mon.0) 219 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.a" in cluster log

fail 7429211 2023-10-16 12:50:47 2023-10-18 18:37:00 2023-10-18 19:06:28 0:29:28 0:14:43 0:14:45 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} 2
Failure Reason:

Command failed on smithi062 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

fail 7429212 2023-10-16 12:50:47 2023-10-18 18:37:40 2023-10-18 19:07:02 0:29:22 0:20:24 0:08:58 smithi main rhel 8.4 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} 2
Failure Reason:

"2023-10-18T18:57:42.928086+0000 mon.a (mon.0) 181 : cluster [WRN] Replacing daemon mds.e as rank 0 with standby daemon mds.c" in cluster log

dead 7429213 2023-10-16 12:50:48 2023-10-18 18:37:41 2023-10-18 22:34:21 3:56:40 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7429214 2023-10-16 12:50:49 2023-10-18 18:38:31 2023-10-18 19:07:05 0:28:34 0:16:38 0:11:56 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} 2
Failure Reason:

Command failed on smithi118 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429215 2023-10-16 12:50:50 2023-10-18 18:38:32 2023-10-18 19:06:56 0:28:24 0:14:43 0:13:41 smithi main centos 8.stream fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
Failure Reason:

Command failed on smithi079 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'

fail 7429216 2023-10-16 12:50:50 2023-10-18 18:42:02 2023-10-18 19:09:10 0:27:08 0:14:50 0:12:18 smithi main centos 8.stream fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
Failure Reason:

Command failed on smithi069 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c'

fail 7429217 2023-10-16 12:50:51 2023-10-18 19:04:35 2023-10-18 19:32:19 0:27:44 0:14:56 0:12:48 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} 2
Failure Reason:

Command failed on smithi045 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b'

fail 7429218 2023-10-16 12:50:52 2023-10-18 19:05:45 2023-10-18 19:36:22 0:30:37 0:17:10 0:13:27 smithi main ubuntu 20.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

"2023-10-18T19:31:11.554059+0000 mon.a (mon.0) 224 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429219 2023-10-16 12:50:53 2023-10-18 19:06:36 2023-10-18 19:35:44 0:29:08 0:16:40 0:12:28 smithi main ubuntu 20.04 fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} 2
Failure Reason:

"2023-10-18T19:31:17.660579+0000 mon.a (mon.0) 218 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429220 2023-10-16 12:50:53 2023-10-18 19:07:06 2023-10-18 19:34:53 0:27:47 0:16:49 0:10:58 smithi main ubuntu 20.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

"2023-10-18T19:30:27.877572+0000 mon.a (mon.0) 225 : cluster [WRN] Replacing daemon mds.a as rank 0 with standby daemon mds.b" in cluster log

fail 7429221 2023-10-16 12:50:54 2023-10-18 19:07:07 2023-10-18 19:38:51 0:31:44 0:19:41 0:12:03 smithi main ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} 2
Failure Reason:

Command failed on smithi105 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 7429222 2023-10-16 12:50:55 2023-10-18 19:07:07 2023-10-18 19:34:11 0:27:04 0:15:28 0:11:36 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} 2
Failure Reason:

"2023-10-18T19:28:19.567356+0000 mon.a (mon.0) 225 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.b" in cluster log

dead 7429223 2023-10-16 12:50:56 2023-10-18 19:09:18 2023-10-18 22:35:48 3:26:30 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
fail 7429224 2023-10-16 12:50:56 2023-10-18 19:09:48 2023-10-18 19:51:08 0:41:20 0:29:29 0:11:51 smithi main rhel 8.4 fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} 2
Failure Reason:

"2023-10-18T19:49:36.755100+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: Reduced data availability: 1 pg down (PG_AVAILABILITY)" in cluster log

fail 7429225 2023-10-16 12:50:57 2023-10-18 19:11:49 2023-10-18 19:42:56 0:31:07 0:12:43 0:18:24 smithi main ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
Failure Reason:

Command failed on smithi049 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a'

fail 7429226 2023-10-16 12:50:58 2023-10-18 19:15:20 2023-10-18 19:46:56 0:31:36 0:14:40 0:16:56 smithi main centos 8.stream fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} 2
Failure Reason:

Command failed on smithi130 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'

fail 7429227 2023-10-16 12:50:59 2023-10-18 19:19:51 2023-10-18 19:44:43 0:24:52 0:12:10 0:12:42 smithi main ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
Failure Reason:

Command failed on smithi158 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c'

fail 7429228 2023-10-16 12:50:59 2023-10-18 19:20:01 2023-10-18 19:48:26 0:28:25 0:16:44 0:11:41 smithi main ubuntu 20.04 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} 2
Failure Reason:

"2023-10-18T19:43:54.841244+0000 mon.a (mon.0) 221 : cluster [WRN] Replacing daemon mds.b as rank 0 with standby daemon mds.a" in cluster log

fail 7429229 2023-10-16 12:51:00 2023-10-18 19:20:02 2023-10-18 19:48:38 0:28:36 0:17:08 0:11:28 smithi main ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} 2
Failure Reason:

"2023-10-18T19:43:10.139010+0000 mon.a (mon.0) 170 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.b" in cluster log

fail 7429230 2023-10-16 12:51:01 2023-10-18 19:20:32 2023-10-18 19:55:27 0:34:55 0:21:45 0:13:10 smithi main centos 8.stream fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} 2
Failure Reason:

"2023-10-18T19:47:29.409490+0000 mon.a (mon.0) 215 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.c" in cluster log

dead 7429231 2023-10-16 12:51:02 2023-10-18 20:03:56 2023-10-18 22:34:47 2:30:51 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7429232 2023-10-16 12:51:02 2023-10-18 20:05:36 2023-10-18 20:29:38 0:24:02 0:12:37 0:11:25 smithi main ubuntu 20.04 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
Failure Reason:

"2023-10-18T20:27:59.281568+0000 mon.a (mon.0) 293 : cluster [WRN] Health check failed: Reduced data availability: 13 pgs down (PG_AVAILABILITY)" in cluster log

fail 7429233 2023-10-16 12:51:03 2023-10-18 20:05:37 2023-10-18 20:41:38 0:36:01 0:13:30 0:22:31 smithi main ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} 5
Failure Reason:

"2023-10-18T20:40:01.331202+0000 mon.a (mon.0) 235 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs down (PG_AVAILABILITY)" in cluster log

dead 7429234 2023-10-16 12:51:04 2023-10-18 20:14:48 2023-10-18 20:36:24 0:21:36 0:11:40 0:09:56 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} 2
Failure Reason:

['<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at']

fail 7429235 2023-10-16 12:51:05 2023-10-18 20:15:39 2023-10-18 20:50:45 0:35:06 0:22:21 0:12:45 smithi main ubuntu 20.04 fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

"2023-10-18T20:40:53.534620+0000 mon.a (mon.0) 216 : cluster [WRN] Replacing daemon mds.l as rank 0 with standby daemon mds.i" in cluster log

dead 7429236 2023-10-16 12:51:05 2023-10-18 20:16:29 2023-10-18 20:38:59 0:22:30 0:10:38 0:11:52 smithi main centos 8.stream fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} 2
Failure Reason:

['<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at']

dead 7429237 2023-10-16 12:51:06 2023-10-18 20:17:30 2023-10-18 20:36:28 0:18:58 0:08:50 0:10:08 smithi main rhel 8.4 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} 2
Failure Reason:

['<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at']

dead 7429238 2023-10-16 12:51:07 2023-10-18 20:18:50 2023-10-18 20:38:35 0:19:45 0:08:46 0:10:59 smithi main rhel 8.4 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} 2
Failure Reason:

['<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Connection timed out', 'Enabling a Copr repository. Please note that this repository is not part', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at']

dead 7429239 2023-10-16 12:51:08 2023-10-18 20:21:31 2023-10-18 20:42:55 0:21:24 0:10:51 0:10:33 smithi main centos 8.stream fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} 2
Failure Reason:

['<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at']

fail 7429240 2023-10-16 12:51:08 2023-10-18 20:22:01 2023-10-18 20:51:09 0:29:08 0:15:34 0:13:34 smithi main centos 8.stream fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

Command failed on smithi155 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'