Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7570935 2024-02-22 18:17:44 2024-02-22 18:18:33 2024-02-22 19:50:38 1:32:05 1:19:27 0:12:38 smithi main ubuntu 18.04 rados/cephadm/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/mimic backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{ubuntu_18.04} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/radosbench} 3
Failure Reason:

"2024-02-22T19:00:00.000231+0000 mon.a (mon.0) 1129 : cluster [WRN] pg 3.a is active+undersized+degraded, acting [6,4]" in cluster log

dead 7570936 2024-02-22 18:17:45 2024-02-22 18:18:33 2024-02-23 06:34:02 12:15:29 smithi main ubuntu 20.04 rados/objectstore/{backends/objectstore supported-random-distro$/{ubuntu_latest}} 1
Failure Reason:

hit max job timeout

pass 7570937 2024-02-22 18:17:46 2024-02-22 18:18:33 2024-02-22 19:43:39 1:25:06 0:59:59 0:25:07 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 7570938 2024-02-22 18:17:47 2024-02-22 18:18:34 2024-02-22 19:42:46 1:24:12 1:01:37 0:22:35 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} 1
fail 7570939 2024-02-22 18:17:47 2024-02-22 18:18:34 2024-02-22 19:53:53 1:35:19 1:09:27 0:25:52 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
Failure Reason:

"2024-02-22T19:25:51.460296+0000 mon.a (mon.0) 161 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.27:3301/0,v1:172.21.15.27:6790/0] is down (out of quorum)" in cluster log

pass 7570940 2024-02-22 18:17:48 2024-02-22 18:18:34 2024-02-22 19:30:48 1:12:14 0:49:56 0:22:18 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} 1
fail 7570941 2024-02-22 18:17:49 2024-02-22 18:18:35 2024-02-22 18:50:43 0:32:08 0:20:25 0:11:43 smithi main ubuntu 20.04 rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

Command failed on smithi094 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.94:/foouser /mnt/foo'"

pass 7570942 2024-02-22 18:17:50 2024-02-22 18:18:35 2024-02-22 19:42:26 1:23:51 0:58:46 0:25:05 smithi main centos 8.stream rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} 2
fail 7570943 2024-02-22 18:17:50 2024-02-22 18:18:35 2024-02-22 19:30:38 1:12:03 0:43:58 0:28:05 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-22T19:08:45.430894+0000 mon.smithi145 (mon.0) 331 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7570944 2024-02-22 18:17:51 2024-02-22 18:18:36 2024-02-22 19:52:44 1:34:08 1:07:14 0:26:54 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-02-22T19:34:46.645296+0000 mon.a (mon.0) 858 : cluster [WRN] Health check failed: 17 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7570945 2024-02-22 18:17:52 2024-02-22 18:18:36 2024-02-22 19:18:25 0:59:49 0:35:02 0:24:47 smithi main centos 8.stream rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} 2
Failure Reason:

Command failed on smithi062 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2bc0d14-d1b4-11ee-95bf-87774f69a715 -e sha1=eb66ed921e744301e6863be6353618d63967ea59 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7570946 2024-02-22 18:17:53 2024-02-22 18:18:36 2024-02-22 19:46:27 1:27:51 1:01:38 0:26:13 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

"2024-02-22T19:40:00.000127+0000 mon.smithi050 (mon.0) 534 : cluster [WRN] [WRN] FS_DEGRADED: 1 filesystem is degraded" in cluster log

fail 7570947 2024-02-22 18:17:54 2024-02-22 18:18:37 2024-02-22 20:01:33 1:42:56 1:17:47 0:25:09 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-02-22T19:40:00.000296+0000 mon.a (mon.0) 1263 : cluster [WRN] [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull" in cluster log

pass 7570948 2024-02-22 18:17:54 2024-02-22 18:18:37 2024-02-22 18:45:33 0:26:56 0:15:58 0:10:58 smithi main rhel 8.6 rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
fail 7570949 2024-02-22 18:17:55 2024-02-22 18:18:37 2024-02-22 19:09:33 0:50:56 0:37:01 0:13:55 smithi main ubuntu 20.04 rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls mon_election/connectivity} 2
Failure Reason:

Command failed on smithi040 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15.2.9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 42039b9c-d1b1-11ee-95bf-87774f69a715 -e sha1=eb66ed921e744301e6863be6353618d63967ea59 -- bash -c \'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk \'"\'"\'{print $2}\'"\'"\')\''

pass 7570950 2024-02-22 18:17:56 2024-02-22 18:18:38 2024-02-22 19:44:25 1:25:47 1:00:51 0:24:56 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} 1
fail 7570951 2024-02-22 18:17:57 2024-02-22 18:18:38 2024-02-22 19:28:33 1:09:55 0:43:36 0:26:19 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-22T19:07:04.499787+0000 mon.smithi037 (mon.0) 357 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7570952 2024-02-22 18:17:58 2024-02-22 18:18:38 2024-02-22 19:51:00 1:32:22 1:04:12 0:28:10 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
Failure Reason:

"2024-02-22T19:40:00.000085+0000 mon.a (mon.0) 1466 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull" in cluster log

pass 7570953 2024-02-22 18:17:58 2024-02-22 18:18:39 2024-02-22 19:32:40 1:14:01 0:49:15 0:24:46 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} 1
pass 7570954 2024-02-22 18:17:59 2024-02-22 18:18:39 2024-02-22 19:44:17 1:25:38 1:00:17 0:25:21 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 7570955 2024-02-22 18:18:00 2024-02-22 18:18:39 2024-02-22 19:40:25 1:21:46 0:56:30 0:25:16 smithi main centos 8.stream rados/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} 2
pass 7570956 2024-02-22 18:18:01 2024-02-22 18:18:40 2024-02-22 19:44:14 1:25:34 1:00:23 0:25:11 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7570957 2024-02-22 18:18:02 2024-02-22 18:18:40 2024-02-22 19:56:24 1:37:44 1:12:48 0:24:56 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-02-22T19:40:00.000200+0000 mon.a (mon.0) 1403 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull" in cluster log

fail 7570958 2024-02-22 18:18:02 2024-02-22 18:18:40 2024-02-22 18:46:35 0:27:55 0:15:46 0:12:09 smithi main ubuntu 18.04 rados/cephadm/osds/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

Command failed on smithi137 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:eb66ed921e744301e6863be6353618d63967ea59 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b624a52-d1b1-11ee-95bf-87774f69a715 -- bash -c \'set -e\nset -x\nceph orch ps\nceph orch device ls\nDEVID=$(ceph device ls | grep osd.1 | awk \'"\'"\'{print $1}\'"\'"\')\nHOST=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $1}\'"\'"\')\nDEV=$(ceph orch device ls | grep $DEVID | awk \'"\'"\'{print $2}\'"\'"\')\necho "host $HOST, dev $DEV, devid $DEVID"\nceph orch osd rm 1\nwhile ceph orch osd rm status | grep ^1 ; do sleep 5 ; done\nceph orch device zap $HOST $DEV --force\nceph orch daemon add osd $HOST:$DEV\nwhile ! ceph osd dump | grep osd.1 | grep up ; do sleep 5 ; done\n\''

fail 7570959 2024-02-22 18:18:03 2024-02-22 18:18:41 2024-02-22 19:34:40 1:15:59 0:48:39 0:27:20 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=391ea41edcaa85aa674d6e816fb2469974fc7d8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7570960 2024-02-22 18:18:04 2024-02-22 18:18:41 2024-02-26 13:04:15 3 days, 18:45:34 3 days, 16:19:15 2:26:19 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

reached maximum tries (181) after waiting for 180 seconds

fail 7570961 2024-02-22 18:18:05 2024-02-22 18:18:41 2024-02-22 19:28:39 1:09:58 0:44:37 0:25:21 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-22T19:02:56.683998+0000 mon.smithi169 (mon.0) 66 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7570962 2024-02-22 18:18:05 2024-02-22 18:18:42 2024-02-22 19:52:27 1:33:45 1:07:41 0:26:04 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-02-22T19:34:41.703528+0000 mon.a (mon.0) 1029 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL)" in cluster log

pass 7570963 2024-02-22 18:18:06 2024-02-22 18:18:42 2024-02-22 19:46:37 1:27:55 1:04:16 0:23:39 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 7570964 2024-02-22 18:18:07 2024-02-22 18:18:42 2024-02-22 19:42:46 1:24:04 1:00:41 0:23:23 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_nfs} 1
fail 7570965 2024-02-22 18:18:08 2024-02-22 18:18:43 2024-02-22 20:03:33 1:44:50 1:18:55 0:25:55 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
Failure Reason:

"2024-02-22T19:40:00.001667+0000 mon.a (mon.0) 1197 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 371/114306 objects degraded (0.325%), 2 pgs degraded" in cluster log

pass 7570966 2024-02-22 18:18:08 2024-02-22 18:18:43 2024-02-22 19:32:14 1:13:31 0:49:04 0:24:27 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_orch_cli} 1
pass 7570967 2024-02-22 18:18:09 2024-02-22 18:18:43 2024-02-22 19:40:03 1:21:20 0:54:39 0:26:41 smithi main centos 8.stream rados/cephadm/dashboard/{0-distro/centos_8.stream_container_tools task/test_e2e} 2
fail 7570968 2024-02-22 18:18:10 2024-02-22 18:18:44 2024-02-22 19:31:27 1:12:43 0:44:18 0:28:25 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-22T19:08:48.496572+0000 mon.smithi090 (mon.0) 328 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7570969 2024-02-22 18:18:11 2024-02-22 18:18:44 2024-02-22 19:50:24 1:31:40 1:04:58 0:26:42 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-02-22T19:40:00.000133+0000 mon.a (mon.0) 1509 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down" in cluster log

fail 7570970 2024-02-22 18:18:12 2024-02-22 18:18:45 2024-02-22 19:05:54 0:47:09 0:33:00 0:14:09 smithi main ubuntu 20.04 rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls mon_election/classic} 2
Failure Reason:

"2024-02-22T18:52:02.392720+0000 mon.a (mon.0) 754 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7570971 2024-02-22 18:18:12 2024-02-22 18:18:45 2024-02-22 19:21:11 1:02:26 0:40:13 0:22:13 smithi main ubuntu 18.04 rados/cephadm/with-work/{0-distro/ubuntu_18.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

"2024-02-22T19:01:53.293606+0000 mon.a (mon.0) 863 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7570972 2024-02-22 18:18:13 2024-02-22 18:18:45 2024-02-22 18:47:43 0:28:58 0:18:39 0:10:19 smithi main rhel 8.6 rados/cephadm/osds/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7570973 2024-02-22 18:18:14 2024-02-22 18:18:46 2024-02-22 18:49:49 0:31:03 0:21:11 0:09:52 smithi main rhel 8.6 rados/cephadm/smoke-roleless/{0-distro/rhel_8.6_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
pass 7570974 2024-02-22 18:18:15 2024-02-22 18:18:46 2024-02-22 19:46:24 1:27:38 1:00:48 0:26:50 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 7570975 2024-02-22 18:18:15 2024-02-22 18:18:46 2024-02-22 19:56:06 1:37:20 1:10:24 0:26:56 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
pass 7570976 2024-02-22 18:18:16 2024-02-22 18:18:47 2024-02-22 19:42:21 1:23:34 0:59:56 0:23:38 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_nfs} 1
fail 7570977 2024-02-22 18:18:17 2024-02-22 18:18:47 2024-02-22 19:27:42 1:08:55 0:43:29 0:25:26 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-02-22T19:07:10.651565+0000 mon.smithi012 (mon.0) 357 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7570978 2024-02-22 18:18:18 2024-02-22 18:18:47 2024-02-22 19:56:28 1:37:41 1:09:48 0:27:53 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

"2024-02-22T19:35:23.844690+0000 mon.a (mon.0) 1023 : cluster [WRN] Health check failed: 1 pool(s) full (POOL_FULL)" in cluster log

pass 7570979 2024-02-22 18:18:19 2024-02-22 18:18:48 2024-02-22 19:32:35 1:13:47 0:49:14 0:24:33 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/connectivity task/test_orch_cli} 1
pass 7570980 2024-02-22 18:18:19 2024-02-22 18:18:48 2024-02-22 19:46:26 1:27:38 1:00:43 0:26:55 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 7570981 2024-02-22 18:18:20 2024-02-22 18:18:48 2024-02-22 19:38:26 1:19:38 0:53:19 0:26:19 smithi main centos 8.stream rados/cephadm/dashboard/{0-distro/ignorelist_health task/test_e2e} 2
pass 7570982 2024-02-22 18:18:21 2024-02-22 18:18:49 2024-02-22 19:45:49 1:27:00 1:00:11 0:26:49 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7570983 2024-02-22 18:18:22 2024-02-22 18:18:49 2024-02-22 20:08:56 1:50:07 1:20:03 0:30:04 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-02-22T19:50:00.000480+0000 mon.a (mon.0) 1714 : cluster [WRN] Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull; Degraded data redundancy: 24012/315519 objects degraded (7.610%), 6 pgs degraded" in cluster log

pass 7570984 2024-02-22 18:18:22 2024-02-22 18:18:49 2024-02-22 18:51:00 0:32:11 0:18:22 0:13:49 smithi main ubuntu 18.04 rados/cephadm/smoke-roleless/{0-distro/ubuntu_18.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
fail 7570985 2024-02-22 18:18:23 2024-02-22 18:18:50 2024-02-22 19:34:53 1:16:03 0:48:23 0:27:40 smithi main centos 8.stream rados/cephadm/workunits/{0-distro/centos_8.stream_container_tools mon_election/classic task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi161 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=391ea41edcaa85aa674d6e816fb2469974fc7d8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7570986 2024-02-22 18:18:24 2024-02-22 18:18:50 2024-02-22 18:52:43 0:33:53 0:19:15 0:14:38 smithi main ubuntu 20.04 rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

Command failed on smithi088 with status 32: "sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mount -t nfs 10.0.31.88:/fake /mnt/foo'"

pass 7570987 2024-02-22 18:18:25 2024-02-22 18:18:50 2024-02-22 19:48:34 1:29:44 1:05:08 0:24:36 smithi main centos 8.stream rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 7570988 2024-02-22 18:18:25 2024-02-22 18:18:51 2024-02-22 19:20:45 1:01:54 0:34:03 0:27:51 smithi main centos 8.stream rados/cephadm/mgr-nfs-upgrade/{0-distro/centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

Command failed on smithi078 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v15 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0fffde9a-d1b5-11ee-95bf-87774f69a715 -e sha1=eb66ed921e744301e6863be6353618d63967ea59 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7570989 2024-02-22 18:18:26 2024-02-22 18:18:51 2024-02-22 19:14:23 0:55:32 0:31:12 0:24:20 smithi main centos 8.stream rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-02-22T19:11:20.274203+0000 mon.smithi131 (mon.0) 780 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7570990 2024-02-22 18:18:27 2024-02-22 18:18:51 2024-02-22 19:50:42 1:31:51 1:04:26 0:27:25 smithi main centos 8.stream rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-02-22T19:27:01.840738+0000 mon.a (mon.0) 163 : cluster [WRN] mon.c (rank 2) addr [v2:172.21.15.107:3301/0,v1:172.21.15.107:6790/0] is down (out of quorum)" in cluster log