Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
dead 7630058 2024-03-29 21:08:32 2024-03-30 12:54:05 2024-03-31 01:01:47 12:07:42 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

hit max job timeout

fail 7630059 2024-03-29 21:08:33 2024-03-30 12:54:05 2024-03-30 13:12:54 0:18:49 0:07:36 0:11:13 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/calico rook/master} 1
Failure Reason:

Command failed on smithi023 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7630060 2024-03-29 21:08:34 2024-03-30 12:54:06 2024-03-30 13:19:33 0:25:27 0:14:39 0:10:48 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-03-30T13:11:31.907325+0000 mon.smithi110 (mon.0) 270 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630061 2024-03-29 21:08:34 2024-03-30 12:54:06 2024-03-30 13:16:22 0:22:16 0:11:47 0:10:29 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
pass 7630062 2024-03-29 21:08:35 2024-03-30 12:55:16 2024-03-30 13:20:05 0:24:49 0:13:50 0:10:59 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
fail 7630063 2024-03-29 21:08:36 2024-03-30 12:55:17 2024-03-30 13:20:57 0:25:40 0:14:46 0:10:54 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-03-30T13:11:58.431138+0000 mon.smithi062 (mon.0) 267 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630064 2024-03-29 21:08:37 2024-03-30 12:55:17 2024-03-30 13:20:15 0:24:58 0:15:12 0:09:46 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
fail 7630065 2024-03-29 21:08:38 2024-03-30 12:55:18 2024-03-30 14:09:06 1:13:48 0:59:47 0:14:01 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi172 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74d4e596-ee97-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

pass 7630066 2024-03-29 21:08:39 2024-03-30 12:55:38 2024-03-30 13:45:21 0:49:43 0:37:22 0:12:21 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} 5
fail 7630067 2024-03-29 21:08:40 2024-03-30 12:57:19 2024-03-30 13:27:48 0:30:29 0:19:49 0:10:40 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-03-30T13:18:20.793239+0000 mon.smithi176 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630068 2024-03-29 21:08:41 2024-03-30 12:57:59 2024-03-30 13:36:03 0:38:04 0:26:13 0:11:51 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_monitoring_stack_basic} 3
fail 7630069 2024-03-29 21:08:41 2024-03-30 12:59:40 2024-03-30 13:31:59 0:32:19 0:20:55 0:11:24 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-03-30T13:20:46.976475+0000 mon.smithi026 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630070 2024-03-29 21:08:42 2024-03-30 13:00:10 2024-03-30 13:58:04 0:57:54 0:48:13 0:09:41 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-30T13:21:36.731750+0000 mon.a (mon.0) 201 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7630071 2024-03-29 21:08:43 2024-03-30 13:00:31 2024-03-30 13:27:41 0:27:10 0:15:40 0:11:30 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-03-30T13:21:15.982005+0000 mon.smithi121 (mon.0) 270 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630072 2024-03-29 21:08:44 2024-03-30 13:00:31 2024-03-30 13:29:08 0:28:37 0:18:15 0:10:22 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-03-30T13:16:46.053352+0000 mon.smithi003 (mon.0) 267 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630073 2024-03-29 21:08:45 2024-03-30 13:00:32 2024-03-30 14:01:33 1:01:01 0:52:39 0:08:22 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-30T13:30:00.000288+0000 mon.a (mon.0) 2499 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled" in cluster log

pass 7630074 2024-03-29 21:08:46 2024-03-30 13:00:32 2024-03-30 13:27:26 0:26:54 0:16:55 0:09:59 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_rgw_multisite} 3
fail 7630075 2024-03-29 21:08:47 2024-03-30 13:00:32 2024-03-30 13:32:00 0:31:28 0:21:38 0:09:50 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

"2024-03-30T13:20:21.712665+0000 mon.smithi102 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630076 2024-03-29 21:08:48 2024-03-30 13:00:33 2024-03-30 13:51:45 0:51:12 0:39:59 0:11:13 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7630077 2024-03-29 21:08:49 2024-03-30 13:00:33 2024-03-30 13:48:22 0:47:49 0:36:32 0:11:17 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-03-30T13:28:02.079163+0000 mon.smithi159 (mon.0) 858 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7630078 2024-03-29 21:08:49 2024-03-30 13:01:14 2024-03-30 13:56:09 0:54:55 0:45:00 0:09:55 smithi main ubuntu 22.04 orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
pass 7630079 2024-03-29 21:08:50 2024-03-30 13:02:44 2024-03-30 13:23:53 0:21:09 0:12:07 0:09:02 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_adoption} 1
pass 7630080 2024-03-29 21:08:51 2024-03-30 13:02:44 2024-03-30 13:28:48 0:26:04 0:12:09 0:13:55 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} 2
fail 7630081 2024-03-29 21:08:52 2024-03-30 13:03:25 2024-03-30 13:31:41 0:28:16 0:15:09 0:13:07 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-03-30T13:24:18.736472+0000 mon.smithi050 (mon.0) 273 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630082 2024-03-29 21:08:53 2024-03-30 13:04:45 2024-03-30 13:39:47 0:35:02 0:24:57 0:10:05 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-03-30T13:20:01.886223+0000 mon.a (mon.0) 208 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7630083 2024-03-29 21:08:54 2024-03-30 13:04:46 2024-03-30 13:29:22 0:24:36 0:13:53 0:10:43 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-03-30T13:23:27.101423+0000 mon.smithi104 (mon.0) 275 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630084 2024-03-29 21:08:55 2024-03-30 13:06:27 2024-03-30 13:27:20 0:20:53 0:11:20 0:09:33 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} 1
pass 7630085 2024-03-29 21:08:56 2024-03-30 13:06:27 2024-03-30 13:32:29 0:26:02 0:11:54 0:14:08 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
fail 7630086 2024-03-29 21:08:57 2024-03-30 13:06:47 2024-03-30 13:59:50 0:53:03 0:40:58 0:12:05 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

"2024-03-30T13:26:28.453268+0000 mon.a (mon.0) 373 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7630087 2024-03-29 21:08:57 2024-03-30 13:07:48 2024-03-30 13:35:36 0:27:48 0:17:25 0:10:23 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_set_mon_crush_locations} 3
fail 7630088 2024-03-29 21:08:58 2024-03-30 13:08:38 2024-03-30 13:36:18 0:27:40 0:14:14 0:13:26 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-03-30T13:28:42.643779+0000 mon.smithi040 (mon.0) 269 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630089 2024-03-29 21:08:59 2024-03-30 13:09:09 2024-03-30 13:54:34 0:45:25 0:36:28 0:08:57 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-30T13:40:00.000142+0000 mon.a (mon.0) 2770 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log

pass 7630090 2024-03-29 21:09:00 2024-03-30 13:09:29 2024-03-30 13:36:43 0:27:14 0:16:12 0:11:02 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
fail 7630091 2024-03-29 21:09:01 2024-03-30 13:09:50 2024-03-30 13:43:49 0:33:59 0:20:29 0:13:30 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-03-30T13:34:05.174645+0000 mon.smithi049 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630092 2024-03-29 21:09:02 2024-03-30 13:10:30 2024-03-30 13:39:23 0:28:53 0:15:08 0:13:45 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_ca_signed_key} 2
pass 7630093 2024-03-29 21:09:03 2024-03-30 13:11:41 2024-03-30 13:49:10 0:37:29 0:23:40 0:13:49 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} 2
fail 7630094 2024-03-29 21:09:04 2024-03-30 13:12:01 2024-03-30 13:37:54 0:25:53 0:14:42 0:11:11 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-03-30T13:29:23.519159+0000 mon.smithi123 (mon.0) 272 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630095 2024-03-29 21:09:05 2024-03-30 13:12:52 2024-03-30 13:40:52 0:28:00 0:14:04 0:13:56 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-03-30T13:34:43.017076+0000 mon.smithi083 (mon.0) 269 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630096 2024-03-29 21:09:06 2024-03-30 13:15:42 2024-03-30 16:42:30 3:26:48 3:11:37 0:15:11 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi066 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 204a8f46-ee9a-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

pass 7630097 2024-03-29 21:09:07 2024-03-30 13:15:43 2024-03-30 13:40:52 0:25:09 0:14:23 0:10:46 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_cephadm_timeout} 1
fail 7630098 2024-03-29 21:09:07 2024-03-30 13:15:43 2024-03-30 13:43:03 0:27:20 0:16:36 0:10:44 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-03-30T13:35:36.351848+0000 mon.smithi042 (mon.0) 272 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630099 2024-03-29 21:09:08 2024-03-30 13:15:43 2024-03-30 13:43:24 0:27:41 0:16:56 0:10:45 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5af0400050a123dc31f105425174f106e057276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7630100 2024-03-29 21:09:09 2024-03-30 13:15:44 2024-03-30 13:49:05 0:33:21 0:22:26 0:10:55 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-03-30T13:37:19.514084+0000 mon.smithi019 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630101 2024-03-29 21:09:10 2024-03-30 13:15:44 2024-03-30 13:34:14 0:18:30 0:07:31 0:10:59 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/1.7.2} 3
Failure Reason:

Command failed on smithi064 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7630102 2024-03-29 21:09:11 2024-03-30 13:15:45 2024-03-30 14:29:07 1:13:22 1:01:04 0:12:18 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-30T13:50:00.000116+0000 mon.a (mon.0) 1104 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

fail 7630103 2024-03-29 21:09:12 2024-03-30 13:15:45 2024-03-30 13:40:42 0:24:57 0:15:59 0:08:58 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-03-30T13:32:36.054862+0000 mon.smithi046 (mon.0) 272 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630104 2024-03-29 21:09:13 2024-03-30 13:15:45 2024-03-30 13:42:02 0:26:17 0:12:56 0:13:21 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-30T13:35:41.561455+0000 mon.a (mon.0) 267 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7630105 2024-03-29 21:09:14 2024-03-30 13:15:56 2024-03-30 13:34:42 0:18:46 0:06:41 0:12:05 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm_repos} 1
fail 7630106 2024-03-29 21:09:15 2024-03-30 13:15:56 2024-03-30 13:45:28 0:29:32 0:20:11 0:09:21 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-03-30T13:36:36.240546+0000 mon.smithi073 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630107 2024-03-29 21:09:15 2024-03-30 13:15:56 2024-03-30 14:17:27 1:01:31 0:51:28 0:10:03 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

"2024-03-30T13:32:17.671317+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7630108 2024-03-29 21:09:16 2024-03-30 13:15:57 2024-03-30 13:43:36 0:27:39 0:16:44 0:10:55 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-03-30T13:35:31.884284+0000 mon.smithi037 (mon.0) 275 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630109 2024-03-29 21:09:17 2024-03-30 13:16:27 2024-03-30 13:52:15 0:35:48 0:26:15 0:09:33 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
fail 7630110 2024-03-29 21:09:18 2024-03-30 13:16:28 2024-03-30 14:09:53 0:53:25 0:39:51 0:13:34 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} 2
Failure Reason:

"2024-03-30T13:42:31.134647+0000 mon.a (mon.0) 199 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7630111 2024-03-29 21:09:19 2024-03-30 13:20:19 2024-03-30 14:15:23 0:55:04 0:41:08 0:13:56 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7630112 2024-03-29 21:09:20 2024-03-30 13:23:59 2024-03-30 13:56:55 0:32:56 0:22:06 0:10:50 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli} 1
fail 7630113 2024-03-29 21:09:21 2024-03-30 13:24:10 2024-03-30 14:01:20 0:37:10 0:23:45 0:13:25 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-03-30T13:47:58.746069+0000 mon.smithi116 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630114 2024-03-29 21:09:22 2024-03-30 13:27:21 2024-03-30 13:50:33 0:23:12 0:14:31 0:08:41 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_extra_daemon_features} 2
fail 7630115 2024-03-29 21:09:23 2024-03-30 13:27:31 2024-03-30 13:56:20 0:28:49 0:17:43 0:11:06 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-03-30T13:46:25.183518+0000 mon.smithi031 (mon.0) 280 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630116 2024-03-29 21:09:24 2024-03-30 13:27:42 2024-03-30 14:02:22 0:34:40 0:14:55 0:19:45 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-03-30T13:55:23.474555+0000 mon.smithi094 (mon.0) 271 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630117 2024-03-29 21:09:24 2024-03-30 13:28:52 2024-03-30 14:26:41 0:57:49 0:45:50 0:11:59 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-30T13:52:28.627456+0000 mon.a (mon.0) 202 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7630118 2024-03-29 21:09:25 2024-03-30 13:31:03 2024-03-30 14:01:48 0:30:45 0:17:03 0:13:42 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-03-30T13:53:08.979041+0000 mon.smithi114 (mon.0) 273 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630119 2024-03-29 21:09:26 2024-03-30 13:31:03 2024-03-30 14:06:56 0:35:53 0:24:53 0:11:00 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_host_drain} 3
Failure Reason:

"2024-03-30T14:00:00.000276+0000 mon.a (mon.0) 752 : cluster 3 [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm" in cluster log

fail 7630120 2024-03-29 21:09:27 2024-03-30 13:31:24 2024-03-30 14:02:51 0:31:27 0:21:32 0:09:55 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-03-30T13:52:28.703312+0000 mon.smithi018 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

dead 7630121 2024-03-29 21:09:28 2024-03-30 13:31:24 2024-03-31 01:40:32 12:09:08 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

hit max job timeout

pass 7630122 2024-03-29 21:09:29 2024-03-30 13:31:24 2024-03-30 14:07:16 0:35:52 0:23:45 0:12:07 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli_mon} 5
pass 7630123 2024-03-29 21:09:30 2024-03-30 13:31:25 2024-03-30 13:52:06 0:20:41 0:12:00 0:08:41 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} 2
fail 7630124 2024-03-29 21:09:31 2024-03-30 13:31:25 2024-03-30 13:57:00 0:25:35 0:14:24 0:11:11 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-03-30T13:49:24.537462+0000 mon.smithi082 (mon.0) 272 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630125 2024-03-29 21:09:33 2024-03-30 13:31:26 2024-03-30 13:59:13 0:27:47 0:12:27 0:15:20 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-30T13:53:03.241636+0000 mon.a (mon.0) 266 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7630126 2024-03-29 21:09:34 2024-03-30 13:31:26 2024-03-30 14:36:16 1:04:50 0:54:02 0:10:48 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-30T14:00:00.165657+0000 mon.a (mon.0) 1105 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log

pass 7630127 2024-03-29 21:09:34 2024-03-30 13:31:27 2024-03-30 13:54:18 0:22:51 0:13:11 0:09:40 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
fail 7630128 2024-03-29 21:09:35 2024-03-30 13:31:27 2024-03-30 13:59:09 0:27:42 0:14:56 0:12:46 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-03-30T13:50:34.758476+0000 mon.smithi148 (mon.0) 272 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630129 2024-03-29 21:09:36 2024-03-30 13:31:27 2024-03-30 13:58:20 0:26:53 0:14:26 0:12:27 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-03-30T13:50:32.085446+0000 mon.smithi107 (mon.0) 270 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630130 2024-03-29 21:09:37 2024-03-30 13:32:38 2024-03-30 13:59:48 0:27:10 0:16:16 0:10:54 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
fail 7630131 2024-03-29 21:09:38 2024-03-30 13:32:38 2024-03-30 14:23:50 0:51:12 0:37:33 0:13:39 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-03-30T13:53:05.117347+0000 mon.a (mon.0) 169 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7630132 2024-03-29 21:09:39 2024-03-30 13:35:39 2024-03-30 14:22:35 0:46:56 0:36:30 0:10:26 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-30T14:00:00.000128+0000 mon.a (mon.0) 871 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

fail 7630133 2024-03-29 21:09:40 2024-03-30 13:35:39 2024-03-30 14:14:38 0:38:59 0:27:19 0:11:40 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-03-30T13:58:24.272750+0000 mon.smithi090 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630134 2024-03-29 21:09:41 2024-03-30 13:36:10 2024-03-30 14:05:43 0:29:33 0:19:17 0:10:16 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
fail 7630135 2024-03-29 21:09:42 2024-03-30 13:36:50 2024-03-30 14:07:13 0:30:23 0:15:56 0:14:27 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

"2024-03-30T13:57:25.566937+0000 mon.smithi072 (mon.0) 267 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630136 2024-03-29 21:09:43 2024-03-30 13:39:31 2024-03-30 14:28:41 0:49:10 0:33:17 0:15:53 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1711808080.808271 mon.smithi103 (mon.0) 914 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7630137 2024-03-29 21:09:43 2024-03-30 13:45:22 2024-03-30 14:04:39 0:19:17 0:10:05 0:09:12 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_adoption} 1
fail 7630138 2024-03-29 21:09:44 2024-03-30 13:45:23 2024-03-30 14:15:03 0:29:40 0:19:23 0:10:17 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-03-30T14:06:50.022734+0000 mon.smithi047 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630139 2024-03-29 21:09:45 2024-03-30 13:45:23 2024-03-30 14:12:40 0:27:17 0:15:08 0:12:09 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-03-30T14:07:33.477475+0000 mon.smithi040 (mon.0) 271 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630140 2024-03-29 21:09:46 2024-03-30 13:46:44 2024-03-30 14:16:25 0:29:41 0:20:06 0:09:35 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_rgw_multisite} 3
fail 7630141 2024-03-29 21:09:47 2024-03-30 13:46:44 2024-03-30 14:19:21 0:32:37 0:21:06 0:11:31 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-03-30T14:09:10.639928+0000 mon.smithi167 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630142 2024-03-29 21:09:48 2024-03-30 13:46:44 2024-03-30 14:03:47 0:17:03 0:07:24 0:09:39 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/host rook/master} 1
Failure Reason:

Command failed on smithi053 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7630143 2024-03-29 21:09:49 2024-03-30 13:46:45 2024-03-30 14:20:22 0:33:37 0:23:39 0:09:58 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} 2
fail 7630144 2024-03-29 21:09:50 2024-03-30 13:46:45 2024-03-30 14:12:50 0:26:05 0:15:01 0:11:04 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-03-30T14:05:42.198537+0000 mon.smithi076 (mon.0) 271 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630145 2024-03-29 21:09:51 2024-03-30 13:46:45 2024-03-30 14:10:40 0:23:55 0:12:43 0:11:12 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-30T14:05:37.656706+0000 mon.a (mon.0) 211 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7630146 2024-03-29 21:09:51 2024-03-30 13:46:46 2024-03-30 14:35:45 0:48:59 0:40:00 0:08:59 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-30T14:10:00.000074+0000 mon.a (mon.0) 892 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

pass 7630147 2024-03-29 21:09:52 2024-03-30 13:46:46 2024-03-30 14:14:18 0:27:32 0:17:36 0:09:56 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
pass 7630148 2024-03-29 21:09:53 2024-03-30 13:46:47 2024-03-30 14:17:48 0:31:01 0:16:49 0:14:12 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
fail 7630149 2024-03-29 21:09:54 2024-03-30 13:46:47 2024-03-30 14:15:40 0:28:53 0:15:19 0:13:34 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-03-30T14:07:31.188384+0000 mon.smithi073 (mon.0) 269 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630150 2024-03-29 21:09:55 2024-03-30 13:46:47 2024-03-30 14:09:42 0:22:55 0:13:42 0:09:13 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-03-30T14:02:53.151120+0000 mon.smithi046 (mon.0) 271 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630151 2024-03-29 21:09:56 2024-03-30 13:46:48 2024-03-30 14:52:32 1:05:44 0:54:16 0:11:28 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7630152 2024-03-29 21:09:57 2024-03-30 13:46:48 2024-03-30 14:17:55 0:31:07 0:19:28 0:11:39 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_cephadm_timeout} 1
fail 7630153 2024-03-29 21:09:58 2024-03-30 13:46:48 2024-03-30 14:19:21 0:32:33 0:21:04 0:11:29 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-03-30T14:09:03.823591+0000 mon.smithi087 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630154 2024-03-29 21:09:59 2024-03-30 13:46:49 2024-03-30 14:37:30 0:50:41 0:37:27 0:13:14 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-03-30T14:21:11.978531+0000 mon.a (mon.0) 1048 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7630155 2024-03-29 21:10:00 2024-03-30 13:49:19 2024-03-30 14:15:37 0:26:18 0:16:34 0:09:44 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_ca_signed_key} 2
fail 7630156 2024-03-29 21:10:00 2024-03-30 13:50:40 2024-03-30 14:17:04 0:26:24 0:15:28 0:10:56 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-03-30T14:08:48.669674+0000 mon.smithi003 (mon.0) 271 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630157 2024-03-29 21:10:02 2024-03-30 13:52:11 2024-03-30 14:42:01 0:49:50 0:39:06 0:10:44 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
Failure Reason:

"2024-03-30T14:13:31.341592+0000 mon.a (mon.0) 204 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7630158 2024-03-29 21:10:02 2024-03-30 13:52:21 2024-03-30 14:20:10 0:27:49 0:15:23 0:12:26 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-03-30T14:12:14.340397+0000 mon.smithi174 (mon.0) 266 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630159 2024-03-29 21:10:03 2024-03-30 13:55:02 2024-03-30 14:21:55 0:26:53 0:14:39 0:12:14 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-03-30T14:15:19.942541+0000 mon.smithi089 (mon.0) 271 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630160 2024-03-29 21:10:04 2024-03-30 13:55:02 2024-03-30 15:19:59 1:24:57 1:11:57 0:13:00 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-30T14:21:13.073407+0000 mon.a (mon.0) 199 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7630161 2024-03-29 21:10:05 2024-03-30 13:57:03 2024-03-30 14:23:02 0:25:59 0:12:30 0:13:29 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5af0400050a123dc31f105425174f106e057276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7630162 2024-03-29 21:10:06 2024-03-30 13:59:53 2024-03-30 14:33:51 0:33:58 0:21:48 0:12:10 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-03-30T14:22:14.756999+0000 mon.smithi094 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630163 2024-03-29 21:10:07 2024-03-30 14:02:14 2024-03-30 14:48:53 0:46:39 0:32:53 0:13:46 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1711809171.5770833 mon.smithi019 (mon.0) 893 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7630164 2024-03-29 21:10:08 2024-03-30 14:02:14 2024-03-30 14:47:02 0:44:48 0:34:26 0:10:22 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-03-30T14:27:14.339071+0000 mon.smithi031 (mon.0) 873 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7630165 2024-03-29 21:10:09 2024-03-30 14:02:15 2024-03-30 14:47:04 0:44:49 0:35:50 0:08:59 smithi main centos 9.stream orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} 1
pass 7630166 2024-03-29 21:10:10 2024-03-30 14:02:15 2024-03-30 14:33:01 0:30:46 0:17:18 0:13:28 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_orch_cli} 1
pass 7630167 2024-03-29 21:10:10 2024-03-30 14:02:15 2024-03-30 14:25:35 0:23:20 0:14:16 0:09:04 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} 2
fail 7630168 2024-03-29 21:10:11 2024-03-30 14:02:16 2024-03-30 14:38:39 0:36:23 0:24:44 0:11:39 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-03-30T14:19:52.433175+0000 mon.a (mon.0) 208 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7630169 2024-03-29 21:10:12 2024-03-30 14:02:16 2024-03-30 14:27:19 0:25:03 0:15:59 0:09:04 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-03-30T14:18:25.276724+0000 mon.smithi067 (mon.0) 269 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630170 2024-03-29 21:10:13 2024-03-30 14:02:27 2024-03-30 14:23:21 0:20:54 0:11:21 0:09:33 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} 1
fail 7630171 2024-03-29 21:10:14 2024-03-30 14:02:27 2024-03-30 14:29:15 0:26:48 0:11:30 0:15:18 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-30T14:23:38.743820+0000 mon.a (mon.0) 209 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7630172 2024-03-29 21:10:15 2024-03-30 14:02:27 2024-03-30 14:19:58 0:17:31 0:08:48 0:08:43 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_cephadm_repos} 1
Failure Reason:

Command failed (workunit test cephadm/test_repos.sh) on smithi186 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5af0400050a123dc31f105425174f106e057276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

pass 7630173 2024-03-29 21:10:16 2024-03-30 14:02:28 2024-03-30 14:38:40 0:36:12 0:24:54 0:11:18 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
fail 7630174 2024-03-29 21:10:17 2024-03-30 14:02:28 2024-03-30 14:28:05 0:25:37 0:16:01 0:09:36 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-03-30T14:19:14.781500+0000 mon.smithi116 (mon.0) 270 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630175 2024-03-29 21:10:17 2024-03-30 14:02:39 2024-03-30 14:32:27 0:29:48 0:19:26 0:10:22 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-03-30T14:22:45.035638+0000 mon.smithi084 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630176 2024-03-29 21:10:18 2024-03-30 14:02:39 2024-03-30 14:34:56 0:32:17 0:20:51 0:11:26 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-03-30T14:23:51.900542+0000 mon.smithi148 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630177 2024-03-29 21:10:19 2024-03-30 14:02:39 2024-03-30 14:43:50 0:41:11 0:30:37 0:10:34 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-30T14:30:00.000168+0000 mon.a (mon.0) 1172 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

fail 7630178 2024-03-29 21:10:20 2024-03-30 14:02:40 2024-03-30 15:17:48 1:15:08 1:01:53 0:13:15 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-30T14:28:30.929051+0000 mon.a (mon.0) 202 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7630179 2024-03-29 21:10:21 2024-03-30 14:05:50 2024-03-30 14:30:50 0:25:00 0:14:46 0:10:14 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_extra_daemon_features} 2
fail 7630180 2024-03-29 21:10:22 2024-03-30 14:05:51 2024-03-30 14:32:39 0:26:48 0:14:30 0:12:18 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-03-30T14:25:05.240816+0000 mon.smithi062 (mon.0) 274 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630181 2024-03-29 21:10:23 2024-03-30 14:07:22 2024-03-30 15:04:42 0:57:20 0:42:59 0:14:21 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

Command failed on smithi146 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 27dbfb3a-eea1-11ee-b647-cb9ed24678a4 -e sha1=4e50bbcdd1630c7760b708f3a5b4d38a121e66cb -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''

fail 7630182 2024-03-29 21:10:24 2024-03-30 14:07:22 2024-03-30 15:19:34 1:12:12 0:56:12 0:16:00 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7630183 2024-03-29 21:10:25 2024-03-30 14:14:23 2024-03-30 14:59:20 0:44:57 0:28:54 0:16:03 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli_mon} 5
Failure Reason:

"2024-03-30T14:53:18.433554+0000 mon.a (mon.0) 1365 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log

fail 7630184 2024-03-29 21:10:25 2024-03-30 14:16:14 2024-03-30 14:34:01 0:17:47 0:07:24 0:10:23 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/calico rook/1.7.2} 1
Failure Reason:

Command failed on smithi152 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7630185 2024-03-29 21:10:26 2024-03-30 14:16:14 2024-03-30 14:41:44 0:25:30 0:14:40 0:10:50 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-03-30T14:33:54.734474+0000 mon.smithi083 (mon.0) 269 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630186 2024-03-29 21:10:27 2024-03-30 14:16:35 2024-03-30 14:46:02 0:29:27 0:16:51 0:12:36 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_host_drain} 3
fail 7630187 2024-03-29 21:10:28 2024-03-30 14:17:35 2024-03-30 14:42:13 0:24:38 0:16:05 0:08:33 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-03-30T14:34:57.445829+0000 mon.smithi026 (mon.0) 268 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630188 2024-03-29 21:10:29 2024-03-30 14:17:36 2024-03-30 14:49:33 0:31:57 0:20:21 0:11:36 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-03-30T14:38:56.206998+0000 mon.smithi104 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630189 2024-03-29 21:10:30 2024-03-30 14:17:36 2024-03-30 14:50:24 0:32:48 0:18:00 0:14:48 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-03-30T14:38:58.663389+0000 mon.smithi050 (mon.0) 269 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630190 2024-03-29 21:10:31 2024-03-30 14:17:37 2024-03-30 14:43:42 0:26:05 0:12:54 0:13:11 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-30T14:37:25.420419+0000 mon.a (mon.0) 267 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7630191 2024-03-29 21:10:32 2024-03-30 14:17:37 2024-03-30 14:45:27 0:27:50 0:16:11 0:11:39 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7630192 2024-03-29 21:10:32 2024-03-30 14:17:37 2024-03-30 14:40:25 0:22:48 0:13:27 0:09:21 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
fail 7630193 2024-03-29 21:10:33 2024-03-30 14:17:38 2024-03-30 15:05:52 0:48:14 0:36:40 0:11:34 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-30T14:50:00.000248+0000 mon.a (mon.0) 1347 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

dead 7630194 2024-03-29 21:10:34 2024-03-30 14:17:38 2024-03-30 14:37:04 0:19:26 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds

pass 7630195 2024-03-29 21:10:35 2024-03-30 14:17:39 2024-03-30 14:51:56 0:34:17 0:22:56 0:11:21 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} 2
fail 7630196 2024-03-29 21:10:36 2024-03-30 14:17:39 2024-03-30 15:03:33 0:45:54 0:31:40 0:14:14 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1711810178.8476303 mon.smithi090 (mon.0) 978 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7630197 2024-03-29 21:10:37 2024-03-30 14:17:39 2024-03-30 14:40:35 0:22:56 0:12:15 0:10:41 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_adoption} 1
fail 7630198 2024-03-29 21:10:38 2024-03-30 14:17:40 2024-03-30 14:41:57 0:24:17 0:13:38 0:10:39 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-03-30T14:36:00.706982+0000 mon.smithi079 (mon.0) 270 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630199 2024-03-29 21:10:39 2024-03-30 14:17:40 2024-03-30 14:47:02 0:29:22 0:19:15 0:10:07 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-03-30T14:38:51.497873+0000 mon.smithi047 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630200 2024-03-29 21:10:40 2024-03-30 14:17:40 2024-03-30 14:47:33 0:29:53 0:19:42 0:10:11 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_monitoring_stack_basic} 3
fail 7630201 2024-03-29 21:10:41 2024-03-30 14:17:41 2024-03-30 14:44:57 0:27:16 0:14:35 0:12:41 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-03-30T14:38:57.147347+0000 mon.smithi099 (mon.0) 271 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630202 2024-03-29 21:10:41 2024-03-30 14:17:41 2024-03-30 14:43:58 0:26:17 0:14:50 0:11:27 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-03-30T14:37:04.481014+0000 mon.smithi161 (mon.0) 268 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630203 2024-03-29 21:10:42 2024-03-30 14:17:42 2024-03-30 15:10:26 0:52:44 0:41:27 0:11:17 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

"2024-03-30T14:36:25.900574+0000 mon.a (mon.0) 201 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7630204 2024-03-29 21:10:43 2024-03-30 14:17:42 2024-03-30 14:50:41 0:32:59 0:20:02 0:12:57 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_rgw_multisite} 3
fail 7630205 2024-03-29 21:10:44 2024-03-30 14:17:52 2024-03-30 14:50:11 0:32:19 0:20:26 0:11:53 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-03-30T14:41:09.642993+0000 mon.smithi042 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630206 2024-03-29 21:10:45 2024-03-30 14:20:23 2024-03-30 14:55:23 0:35:00 0:19:28 0:15:32 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-03-30T14:45:49.157941+0000 mon.smithi114 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630207 2024-03-29 21:10:46 2024-03-30 14:25:44 2024-03-30 15:20:21 0:54:37 0:38:06 0:16:31 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-30T15:00:00.000177+0000 mon.a (mon.0) 1711 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled" in cluster log

fail 7630208 2024-03-29 21:10:47 2024-03-30 14:29:15 2024-03-30 15:23:58 0:54:43 0:41:41 0:13:02 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7630209 2024-03-29 21:10:48 2024-03-30 14:30:56 2024-03-30 14:53:49 0:22:53 0:14:21 0:08:32 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_cephadm_timeout} 1
pass 7630210 2024-03-29 21:10:49 2024-03-30 14:30:56 2024-03-30 14:57:58 0:27:02 0:14:15 0:12:47 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} 2
fail 7630211 2024-03-29 21:10:49 2024-03-30 14:33:07 2024-03-30 15:03:03 0:29:56 0:15:39 0:14:17 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-03-30T14:55:19.026983+0000 mon.smithi094 (mon.0) 276 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7630212 2024-03-29 21:10:50 2024-03-30 14:33:07 2024-03-30 15:02:43 0:29:36 0:19:44 0:09:52 smithi main ubuntu 22.04 orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} 1
fail 7630213 2024-03-29 21:10:51 2024-03-30 14:33:08 2024-03-30 14:57:39 0:24:31 0:12:18 0:12:13 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-30T14:51:59.413336+0000 mon.a (mon.0) 267 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7630214 2024-03-29 21:10:52 2024-03-30 14:33:08 2024-03-30 15:06:15 0:33:07 0:23:55 0:09:12 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys} 2
pass 7630215 2024-03-29 21:10:53 2024-03-30 14:33:08 2024-03-30 14:58:05 0:24:57 0:16:42 0:08:15 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7630216 2024-03-29 21:10:54 2024-03-30 14:33:09 2024-03-30 15:05:28 0:32:19 0:22:03 0:10:16 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_set_mon_crush_locations} 3
fail 7630217 2024-03-29 21:10:55 2024-03-30 14:33:09 2024-03-30 14:58:02 0:24:53 0:16:11 0:08:42 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-03-30T14:50:56.172709+0000 mon.smithi043 (mon.0) 274 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630218 2024-03-29 21:10:56 2024-03-30 14:33:09 2024-03-30 15:04:56 0:31:47 0:21:37 0:10:10 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-03-30T14:53:38.496239+0000 mon.smithi067 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630219 2024-03-29 21:10:57 2024-03-30 14:33:10 2024-03-30 14:56:56 0:23:46 0:14:10 0:09:36 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-03-30T14:50:32.171053+0000 mon.smithi082 (mon.0) 272 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630220 2024-03-29 21:10:58 2024-03-30 14:33:10 2024-03-30 14:54:41 0:21:31 0:11:19 0:10:12 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_ca_signed_key} 2
Failure Reason:

Command failed on smithi060 with status 5: 'sudo systemctl stop ceph-e17c2526-eea4-11ee-b647-cb9ed24678a4@mon.a'

fail 7630221 2024-03-29 21:10:59 2024-03-30 14:33:11 2024-03-30 14:58:12 0:25:01 0:15:38 0:09:23 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-03-30T14:49:59.354172+0000 mon.smithi170 (mon.0) 270 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630222 2024-03-29 21:10:59 2024-03-30 14:33:11 2024-03-30 15:47:38 1:14:27 1:04:14 0:10:13 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-30T14:55:22.312595+0000 mon.a (mon.0) 200 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7630223 2024-03-29 21:11:00 2024-03-30 14:33:21 2024-03-30 15:17:25 0:44:04 0:34:39 0:09:25 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1711811084.483746 mon.smithi110 (mon.0) 921 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7630224 2024-03-29 21:11:01 2024-03-30 14:33:22 2024-03-30 14:56:53 0:23:31 0:13:29 0:10:02 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli} 1
fail 7630225 2024-03-29 21:11:02 2024-03-30 14:33:22 2024-03-30 15:01:08 0:27:46 0:16:54 0:10:52 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-03-30T14:50:25.841186+0000 mon.smithi084 (mon.0) 270 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7630226 2024-03-29 21:11:03 2024-03-30 14:33:22 2024-03-30 14:52:23 0:19:01 0:07:27 0:11:34 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/flannel rook/master} 3
Failure Reason:

Command failed on smithi087 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7630227 2024-03-29 21:11:04 2024-03-30 14:33:23 2024-03-30 15:33:47 1:00:24 0:45:10 0:15:14 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-30T14:59:35.251374+0000 mon.a (mon.0) 198 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7630228 2024-03-29 21:11:05 2024-03-30 14:37:14 2024-03-30 15:00:19 0:23:05 0:13:16 0:09:49 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi171 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5af0400050a123dc31f105425174f106e057276a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7630229 2024-03-29 21:11:06 2024-03-30 14:37:14 2024-03-30 15:12:46 0:35:32 0:24:15 0:11:17 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-03-30T15:00:20.697477+0000 mon.smithi012 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log