Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7566717 2024-02-19 19:27:45 2024-02-19 20:28:27 2024-02-19 22:32:46 2:04:19 1:54:37 0:09:42 smithi main ubuntu 18.04 rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/radosbench} 3
Failure Reason:

"2024-02-19T21:10:00.000258+0000 mon.a (mon.0) 1265 : cluster [WRN] pg 3.c is active+remapped+backfill_toofull, acting [5,3,6]" in cluster log

dead 7566718 2024-02-19 19:27:45 2024-02-19 20:28:28 2024-02-20 08:41:24 12:12:56 smithi main ubuntu 20.04 rados/objectstore/{backends/objectstore supported-random-distro$/{ubuntu_latest}} 1
Failure Reason:

hit max job timeout

fail 7566719 2024-02-19 19:27:46 2024-02-19 20:28:28 2024-02-19 21:10:36 0:42:08 0:30:35 0:11:33 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

"2024-02-19T21:00:00.000110+0000 mon.smithi162 (mon.0) 154 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; Reduced data availability: 2 pgs peering" in cluster log

fail 7566720 2024-02-19 19:27:47 2024-02-19 20:28:28 2024-02-19 21:05:56 0:37:28 0:26:36 0:10:52 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} 1
Failure Reason:

"2024-02-19T20:56:41.291449+0000 mon.a (mon.0) 1196 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566721 2024-02-19 19:27:48 2024-02-19 20:28:29 2024-02-19 21:19:50 0:51:21 0:39:47 0:11:34 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
Failure Reason:

expected string or bytes-like object

fail 7566722 2024-02-19 19:27:49 2024-02-19 20:28:29 2024-02-19 20:53:21 0:24:52 0:15:16 0:09:36 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-02-19T20:51:07.016182+0000 mon.a (mon.0) 468 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7566723 2024-02-19 19:27:49 2024-02-19 20:28:29 2024-02-19 21:11:02 0:42:33 0:30:45 0:11:48 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7566724 2024-02-19 19:27:50 2024-02-19 20:28:30 2024-02-21 19:01:46 1 day, 22:33:16 0:19:33 1 day, 22:13:43 smithi main ubuntu 20.04 rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

Command failed on smithi155 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.155:/foouser /mnt/foo'"

fail 7566725 2024-02-19 19:27:51 2024-02-19 20:29:10 2024-02-19 21:06:24 0:37:14 0:25:45 0:11:29 smithi main centos 8.stream rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} 2
Failure Reason:

"2024-02-19T21:04:56.203704+0000 osd.4 (osd.4) 3 : cluster [WRN] Monitor daemon marked osd.4 down, but it is still running" in cluster log

fail 7566726 2024-02-19 19:27:52 2024-02-19 20:31:01 2024-02-19 21:15:36 0:44:35 0:32:26 0:12:09 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-19T20:53:26.499079+0000 mon.smithi138 (mon.0) 332 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566727 2024-02-19 19:27:53 2024-02-19 20:34:12 2024-02-19 21:23:42 0:49:30 0:37:00 0:12:30 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
Failure Reason:

expected string or bytes-like object

fail 7566728 2024-02-19 19:27:53 2024-02-19 20:35:52 2024-02-19 21:10:35 0:34:43 0:21:37 0:13:06 smithi main centos 8.stream rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} 2
Failure Reason:

Command failed on smithi033 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0eef3440-cf69-11ee-95bb-87774f69a715 -e sha1=81bd20d634209c7cb82c18be12b4b5a05643ebf1 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

pass 7566729 2024-02-19 19:27:54 2024-02-19 20:39:23 2024-02-19 21:28:26 0:49:03 0:42:17 0:06:46 smithi main rhel 8.6 rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} 2
pass 7566730 2024-02-19 19:27:55 2024-02-19 20:39:23 2024-02-19 21:04:58 0:25:35 0:16:13 0:09:22 smithi main centos 8.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} tasks/rados_python} 2
pass 7566731 2024-02-19 19:27:56 2024-02-19 20:39:44 2024-02-19 21:06:55 0:27:11 0:17:24 0:09:47 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} 1
fail 7566732 2024-02-19 19:27:57 2024-02-19 20:39:44 2024-02-19 21:19:42 0:39:58 0:30:45 0:09:13 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

"2024-02-19T21:10:00.000173+0000 mon.smithi143 (mon.0) 163 : cluster [WRN] osd.1 (root=default,host=smithi143) is down" in cluster log

fail 7566733 2024-02-19 19:27:58 2024-02-19 20:40:35 2024-02-19 21:39:56 0:59:21 0:47:20 0:12:01 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
Failure Reason:

expected string or bytes-like object

pass 7566734 2024-02-19 19:27:58 2024-02-19 20:42:45 2024-02-19 21:08:52 0:26:07 0:16:39 0:09:28 smithi main ubuntu 18.04 rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
fail 7566735 2024-02-19 19:27:59 2024-02-19 20:43:26 2024-02-19 21:06:36 0:23:10 0:17:10 0:06:00 smithi main rhel 8.6 rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-02-19T21:04:02.957182+0000 mon.smithi082 (mon.0) 591 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7566736 2024-02-19 19:28:00 2024-02-19 20:43:36 2024-02-19 21:35:25 0:51:49 0:33:21 0:18:28 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7566737 2024-02-19 19:28:01 2024-02-19 20:43:36 2024-02-19 21:31:26 0:47:50 0:37:32 0:10:18 smithi main ubuntu 20.04 rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} 2
Failure Reason:

Command failed on smithi084 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 90931110-cf69-11ee-95bb-87774f69a715 -e sha1=81bd20d634209c7cb82c18be12b4b5a05643ebf1 -- bash -c \'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk \'"\'"\'{print $2}\'"\'"\')\''

fail 7566738 2024-02-19 19:28:02 2024-02-19 20:43:37 2024-02-19 21:24:43 0:41:06 0:31:54 0:09:12 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} 1
Failure Reason:

"2024-02-19T21:16:19.328300+0000 mon.a (mon.0) 1200 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566739 2024-02-19 19:28:03 2024-02-19 20:43:47 2024-02-19 21:23:56 0:40:09 0:30:18 0:09:51 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-19T21:04:10.213279+0000 mon.smithi072 (mon.0) 361 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566740 2024-02-19 19:28:03 2024-02-19 20:44:08 2024-02-19 21:40:55 0:56:47 0:34:57 0:21:50 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
Failure Reason:

expected string or bytes-like object

pass 7566741 2024-02-19 19:28:04 2024-02-19 20:55:50 2024-02-19 21:49:07 0:53:17 0:41:31 0:11:46 smithi main ubuntu 20.04 rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} 2
fail 7566742 2024-02-19 19:28:05 2024-02-19 20:57:10 2024-02-19 21:23:05 0:25:55 0:16:09 0:09:46 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} 1
Failure Reason:

"2024-02-19T21:21:51.895794+0000 mon.a (mon.0) 468 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7566743 2024-02-19 19:28:06 2024-02-19 20:57:10 2024-02-19 21:43:30 0:46:20 0:32:51 0:13:29 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 7566744 2024-02-19 19:28:07 2024-02-19 20:59:21 2024-02-19 21:38:24 0:39:03 0:30:14 0:08:49 smithi main centos 8.stream rados/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} 2
pass 7566745 2024-02-19 19:28:07 2024-02-19 20:59:21 2024-02-19 21:42:36 0:43:15 0:33:32 0:09:43 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7566746 2024-02-19 19:28:08 2024-02-19 20:59:22 2024-02-19 21:50:45 0:51:23 0:40:17 0:11:06 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

expected string or bytes-like object

fail 7566747 2024-02-19 19:28:09 2024-02-19 20:59:22 2024-02-19 21:24:47 0:25:25 0:16:00 0:09:25 smithi main ubuntu 18.04 rados/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

Command failed on smithi114 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81bd20d634209c7cb82c18be12b4b5a05643ebf1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ab25ac84-cf6b-11ee-95bb-87774f69a715 -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\''

pass 7566748 2024-02-19 19:28:10 2024-02-19 20:59:22 2024-02-19 21:26:28 0:27:06 0:18:10 0:08:56 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} 1
pass 7566749 2024-02-19 19:28:11 2024-02-19 20:59:23 2024-02-19 21:33:47 0:34:24 0:25:34 0:08:50 smithi main centos 8.stream rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} 1
fail 7566750 2024-02-19 19:28:12 2024-02-19 20:59:23 2024-02-19 21:48:26 0:49:03 0:39:44 0:09:19 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

"2024-02-19T21:30:14.488937+0000 mon.smithi135 (mon.0) 325 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566751 2024-02-19 19:28:12 2024-02-19 20:59:23 2024-02-19 21:42:59 0:43:36 0:32:43 0:10:53 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-19T21:18:32.051729+0000 mon.smithi088 (mon.0) 272 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log

fail 7566752 2024-02-19 19:28:13 2024-02-19 20:59:24 2024-02-19 21:48:31 0:49:07 0:38:25 0:10:42 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} 2
Failure Reason:

expected string or bytes-like object

pass 7566753 2024-02-19 19:28:14 2024-02-19 20:59:24 2024-02-19 21:25:19 0:25:55 0:15:19 0:10:36 smithi main centos 8.stream rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
fail 7566754 2024-02-19 19:28:15 2024-02-19 20:59:35 2024-02-19 21:43:13 0:43:38 0:33:14 0:10:24 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

"2024-02-19T21:25:07.544534+0000 mon.smithi008 (mon.0) 321 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566755 2024-02-19 19:28:16 2024-02-19 20:59:35 2024-02-19 21:41:06 0:41:31 0:32:22 0:09:09 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} 1
Failure Reason:

"2024-02-19T21:33:23.389929+0000 mon.a (mon.0) 1207 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566756 2024-02-19 19:28:17 2024-02-19 20:59:35 2024-02-19 22:03:42 1:04:07 0:51:26 0:12:41 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
Failure Reason:

expected string or bytes-like object

fail 7566757 2024-02-19 19:28:17 2024-02-19 20:59:36 2024-02-19 21:36:15 0:36:39 0:20:58 0:15:41 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-02-19T21:33:37.948812+0000 mon.a (mon.0) 463 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7566758 2024-02-19 19:28:18 2024-02-19 21:05:07 2024-02-19 21:52:12 0:47:05 0:36:37 0:10:28 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7566759 2024-02-19 19:28:19 2024-02-19 21:40:17 1569 smithi main centos 8.stream rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} 2
Failure Reason:

"2024-02-19T21:39:19.333057+0000 mon.a (mon.0) 552 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566760 2024-02-19 19:28:20 2024-02-19 21:05:17 2024-02-19 21:52:04 0:46:47 0:32:02 0:14:45 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-19T21:28:49.085064+0000 mon.smithi100 (mon.0) 328 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566761 2024-02-19 19:28:21 2024-02-19 21:08:58 2024-02-19 22:02:58 0:54:00 0:43:56 0:10:04 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

expected string or bytes-like object

fail 7566762 2024-02-19 19:28:22 2024-02-19 21:09:09 2024-02-19 21:55:19 0:46:10 0:32:53 0:13:17 smithi main ubuntu 20.04 rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} 2
Failure Reason:

"2024-02-19T21:30:00.000240+0000 mon.a (mon.0) 319 : cluster [WRN] pg 1.0 is stuck inactive for 93s, current state undersized+peered, last acting [0]" in cluster log

fail 7566763 2024-02-19 19:28:22 2024-02-19 21:11:09 2024-02-19 21:34:43 0:23:34 smithi main ubuntu 18.04 rados/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

Failed to reconnect to smithi155

fail 7566764 2024-02-19 19:28:23 2024-02-19 21:14:50 2024-02-19 21:42:16 0:27:26 0:18:29 0:08:57 smithi main rhel 8.6 rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-02-19T21:38:42.465998+0000 mon.smithi026 (mon.0) 632 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

dead 7566765 2024-02-19 19:28:24 2024-02-19 21:14:51 2024-02-19 21:34:00 0:19:09 smithi main rhel 8.6 rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds

pass 7566766 2024-02-19 19:28:25 2024-02-19 21:14:51 2024-02-19 21:41:44 0:26:53 0:17:34 0:09:19 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_cephadm} 1
pass 7566767 2024-02-19 19:28:26 2024-02-19 21:14:51 2024-02-19 22:04:01 0:49:10 0:38:51 0:10:19 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7566768 2024-02-19 19:28:27 2024-02-19 21:14:52 2024-02-19 22:04:12 0:49:20 0:39:24 0:09:56 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

expected string or bytes-like object

pass 7566769 2024-02-19 19:28:27 2024-02-19 21:14:52 2024-02-19 22:06:12 0:51:20 0:41:11 0:10:09 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 7566770 2024-02-19 19:28:28 2024-02-19 21:14:53 2024-02-19 22:17:29 1:02:36 0:52:01 0:10:35 smithi main ubuntu 20.04 rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} 2
fail 7566771 2024-02-19 19:28:29 2024-02-19 22:02:12 1650 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} 1
Failure Reason:

"2024-02-19T21:53:58.056229+0000 mon.a (mon.0) 1196 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7566772 2024-02-19 19:28:30 2024-02-19 21:25:25 2024-02-19 22:39:03 1:13:38 1:03:07 0:10:31 smithi main ubuntu 18.04 rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/mimic-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{ubuntu_18.04} mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/radosbench} 3
fail 7566773 2024-02-19 19:28:31 2024-02-19 21:25:25 2024-02-19 22:09:36 0:44:11 0:30:46 0:13:25 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-19T21:48:05.042519+0000 mon.smithi131 (mon.0) 350 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566774 2024-02-19 19:28:31 2024-02-19 21:28:36 2024-02-19 22:21:18 0:52:42 0:42:50 0:09:52 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

expected string or bytes-like object

pass 7566775 2024-02-19 19:28:32 2024-02-19 21:30:17 2024-02-19 22:17:04 0:46:47 0:38:09 0:08:38 smithi main centos 8.stream rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} 2
fail 7566776 2024-02-19 19:28:33 2024-02-19 21:30:17 2024-02-19 21:55:10 0:24:53 0:15:52 0:09:01 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} 1
Failure Reason:

"2024-02-19T21:53:19.802901+0000 mon.a (mon.0) 466 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

fail 7566777 2024-02-19 19:28:34 2024-02-19 21:30:17 2024-02-19 22:13:21 0:43:04 0:33:24 0:09:40 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

Command failed on smithi195 with status 5: 'sudo systemctl stop ceph-768bba62-cf71-11ee-95bb-87774f69a715@mon.smithi195'

pass 7566778 2024-02-19 19:28:35 2024-02-19 21:30:18 2024-02-19 21:53:28 0:23:10 0:15:56 0:07:14 smithi main rhel 8.6 rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
fail 7566779 2024-02-19 19:28:35 2024-02-19 21:30:28 2024-02-19 22:08:18 0:37:50 0:26:18 0:11:32 smithi main centos 8.stream rados/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} 2
Failure Reason:

"2024-02-19T22:05:50.588121+0000 mon.a (mon.0) 547 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7566781 2024-02-19 19:28:36 2024-02-19 21:30:34 2024-02-19 22:11:35 0:41:01 0:26:29 0:14:32 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

Command failed on smithi173 with status 5: 'sudo systemctl stop ceph-38187086-cf71-11ee-95bb-87774f69a715@mon.smithi173'

fail 7566783 2024-02-19 19:28:37 2024-02-19 21:30:35 2024-02-19 22:26:35 0:56:00 0:45:36 0:10:24 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
Failure Reason:

expected string or bytes-like object

pass 7566785 2024-02-19 19:28:38 2024-02-19 21:30:35 2024-02-19 22:27:47 0:57:12 0:48:22 0:08:50 smithi main centos 8.stream rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/classic} 2
pass 7566787 2024-02-19 19:28:39 2024-02-19 21:30:36 2024-02-19 22:14:03 0:43:27 0:32:53 0:10:34 smithi main rhel 8.6 rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} 2
pass 7566789 2024-02-19 19:28:39 2024-02-19 21:34:17 2024-02-19 22:04:03 0:29:46 0:16:47 0:12:59 smithi main ubuntu 18.04 rados/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
fail 7566791 2024-02-19 19:28:40 2024-02-19 21:36:58 2024-02-19 22:31:59 0:55:01 0:43:04 0:11:57 smithi main ubuntu 18.04 rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

reached maximum tries (301) after waiting for 300 seconds

fail 7566792 2024-02-19 19:28:41 2024-02-19 21:38:29 2024-02-19 22:08:58 0:30:29 0:18:00 0:12:29 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=81bd20d634209c7cb82c18be12b4b5a05643ebf1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7566794 2024-02-19 19:28:42 2024-02-19 21:42:40 2024-02-19 22:40:49 0:58:09 0:44:13 0:13:56 smithi main ubuntu 20.04 rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

reached maximum tries (301) after waiting for 300 seconds

fail 7566796 2024-02-19 19:28:43 2024-02-19 21:45:42 2024-02-19 22:29:54 0:44:12 0:33:10 0:11:02 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

Command failed on smithi203 with status 5: 'sudo systemctl stop ceph-dbbe4146-cf73-11ee-95bb-87774f69a715@mon.smithi203'

fail 7566798 2024-02-19 19:28:43 2024-02-19 21:46:02 2024-02-19 23:25:10 1:39:08 1:28:48 0:10:20 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

Command failed on smithi016 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 504fbe88-cf72-11ee-95bb-87774f69a715 -e sha1=81bd20d634209c7cb82c18be12b4b5a05643ebf1 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7566800 2024-02-19 19:28:44 2024-02-19 21:46:03 2024-02-19 22:38:51 0:52:48 0:42:55 0:09:53 smithi main centos 8.stream rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

reached maximum tries (301) after waiting for 300 seconds

fail 7566802 2024-02-19 19:28:45 2024-02-19 21:46:04 2024-02-19 22:32:47 0:46:43 0:37:25 0:09:18 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

expected string or bytes-like object

pass 7566804 2024-02-19 19:28:46 2024-02-19 21:46:04 2024-02-19 22:31:12 0:45:08 0:39:08 0:06:00 smithi main rhel 8.6 rados/cephadm/with-work/{0-distro/rhel_8.6_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2