Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7621916 2024-03-25 21:08:45 2024-03-26 01:31:20 2024-03-26 02:24:53 0:53:33 0:42:59 0:10:34 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-26T02:00:00.000218+0000 mon.a (mon.0) 1470 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log

fail 7621917 2024-03-25 21:08:46 2024-03-26 01:31:20 2024-03-26 02:05:01 0:33:41 0:23:32 0:10:09 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

"2024-03-26T01:51:30.646476+0000 mon.smithi080 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621918 2024-03-25 21:08:47 2024-03-26 01:31:21 2024-03-26 01:49:09 0:17:48 0:08:33 0:09:15 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_cephadm_repos} 1
Failure Reason:

Command failed (workunit test cephadm/test_repos.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

fail 7621919 2024-03-25 21:08:48 2024-03-26 01:31:21 2024-03-26 02:04:28 0:33:07 0:23:09 0:09:58 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ae3e77b2-eb12-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

fail 7621920 2024-03-25 21:08:49 2024-03-26 01:31:52 2024-03-26 02:16:40 0:44:48 0:35:16 0:09:32 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-03-26T01:50:42.933185+0000 mon.smithi019 (mon.0) 504 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

pass 7621921 2024-03-25 21:08:50 2024-03-26 01:31:52 2024-03-26 02:27:21 0:55:29 0:45:19 0:10:10 smithi main ubuntu 22.04 orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
pass 7621922 2024-03-25 21:08:51 2024-03-26 01:31:52 2024-03-26 01:55:47 0:23:55 0:14:11 0:09:44 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} 2
fail 7621923 2024-03-25 21:08:52 2024-03-26 01:31:53 2024-03-26 01:54:39 0:22:46 0:13:14 0:09:32 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-03-26T01:48:50.171158+0000 mon.smithi038 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621924 2024-03-25 21:08:53 2024-03-26 01:31:53 2024-03-26 02:08:52 0:36:59 0:24:41 0:12:18 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
fail 7621925 2024-03-25 21:08:54 2024-03-26 01:33:54 2024-03-26 02:00:52 0:26:58 0:12:38 0:14:20 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-03-26T01:55:11.272109+0000 mon.smithi164 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621926 2024-03-25 21:08:55 2024-03-26 01:37:35 2024-03-26 01:58:31 0:20:56 0:11:34 0:09:22 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} 1
fail 7621927 2024-03-25 21:08:56 2024-03-26 01:37:45 2024-03-26 02:02:28 0:24:43 0:11:22 0:13:21 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-26T01:56:48.619706+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7621928 2024-03-25 21:08:57 2024-03-26 01:37:46 2024-03-26 02:37:37 0:59:51 0:48:31 0:11:20 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-03-26T01:54:15.621834+0000 mon.a (mon.0) 167 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7621929 2024-03-25 21:08:58 2024-03-26 01:38:16 2024-03-26 02:01:15 0:22:59 0:14:00 0:08:59 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-03-26T01:55:14.928771+0000 mon.smithi133 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621930 2024-03-25 21:08:59 2024-03-26 01:38:17 2024-03-26 02:01:25 0:23:08 0:14:15 0:08:53 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_cephadm_timeout} 1
fail 7621931 2024-03-25 21:09:00 2024-03-26 01:38:17 2024-03-26 02:41:50 1:03:33 0:53:05 0:10:28 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-26T02:10:00.000151+0000 mon.a (mon.0) 2538 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled" in cluster log

fail 7621932 2024-03-25 21:09:01 2024-03-26 01:38:27 2024-03-26 01:57:39 0:19:12 0:07:29 0:11:43 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/calico rook/master} 3
Failure Reason:

Command failed on smithi105 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7621933 2024-03-25 21:09:03 2024-03-26 01:39:38 2024-03-26 02:12:58 0:33:20 0:21:49 0:11:31 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-03-26T02:01:38.237244+0000 mon.smithi103 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621934 2024-03-25 21:09:04 2024-03-26 01:40:38 2024-03-26 02:07:37 0:26:59 0:13:52 0:13:07 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_extra_daemon_features} 2
fail 7621935 2024-03-25 21:09:05 2024-03-26 01:44:29 2024-03-26 02:07:48 0:23:19 0:14:01 0:09:18 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-03-26T02:01:04.407760+0000 mon.smithi117 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621936 2024-03-25 21:09:06 2024-03-26 01:44:40 2024-03-26 02:09:58 0:25:18 0:14:19 0:10:59 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-03-26T02:03:53.162660+0000 mon.smithi150 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621937 2024-03-25 21:09:07 2024-03-26 01:46:40 2024-03-26 03:06:33 1:19:53 1:09:29 0:10:24 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-26T02:09:14.286778+0000 mon.a (mon.0) 183 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7621938 2024-03-25 21:09:08 2024-03-26 01:46:41 2024-03-26 02:36:26 0:49:45 0:39:37 0:10:08 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7621939 2024-03-25 21:09:09 2024-03-26 01:47:21 2024-03-26 02:12:30 0:25:09 0:14:12 0:10:57 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-03-26T02:04:57.669826+0000 mon.smithi057 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621940 2024-03-25 21:09:10 2024-03-26 01:47:22 2024-03-26 02:12:07 0:24:45 0:15:37 0:09:08 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
fail 7621941 2024-03-25 21:09:11 2024-03-26 01:47:22 2024-03-26 02:22:37 0:35:15 0:22:42 0:12:33 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_host_drain} 3
Failure Reason:

"2024-03-26T02:20:00.000201+0000 mon.a (mon.0) 642 : cluster 3 [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering" in cluster log

fail 7621942 2024-03-25 21:09:12 2024-03-26 01:47:22 2024-03-26 02:19:29 0:32:07 0:22:06 0:10:01 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-03-26T02:08:54.547369+0000 mon.smithi119 (mon.0) 253 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621943 2024-03-25 21:09:13 2024-03-26 01:47:33 2024-03-26 02:14:42 0:27:09 0:15:29 0:11:40 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-03-26T02:06:51.163840+0000 mon.smithi033 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621944 2024-03-25 21:09:14 2024-03-26 01:47:33 2024-03-26 02:14:41 0:27:08 0:12:17 0:14:51 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-26T02:08:17.465556+0000 mon.a (mon.0) 249 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7621945 2024-03-25 21:09:15 2024-03-26 03:20:19 2024-03-26 03:43:04 0:22:45 0:13:28 0:09:17 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
fail 7621946 2024-03-25 21:09:16 2024-03-26 03:20:19 2024-03-26 03:52:29 0:32:10 0:20:50 0:11:20 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-03-26T03:42:44.487791+0000 mon.smithi055 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621947 2024-03-25 21:09:17 2024-03-26 03:20:20 2024-03-26 04:02:35 0:42:15 0:32:29 0:09:46 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} 2
Failure Reason:

"2024-03-26T03:42:22.603425+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7621948 2024-03-25 21:09:18 2024-03-26 03:20:20 2024-03-26 04:27:05 1:06:45 0:56:56 0:09:49 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-03-26T03:38:31.555918+0000 mon.a (mon.0) 170 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7621949 2024-03-25 21:09:19 2024-03-26 03:20:20 2024-03-26 03:46:04 0:25:44 0:15:36 0:10:08 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-03-26T03:37:30.714020+0000 mon.smithi073 (mon.0) 266 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621950 2024-03-25 21:09:20 2024-03-26 03:20:21 2024-03-26 04:01:40 0:41:19 0:31:46 0:09:33 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-26T03:50:00.000121+0000 mon.a (mon.0) 1328 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

pass 7621951 2024-03-25 21:09:21 2024-03-26 03:20:21 2024-03-26 03:50:33 0:30:12 0:19:36 0:10:36 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_monitoring_stack_basic} 3
pass 7621952 2024-03-25 21:09:22 2024-03-26 03:20:41 2024-03-26 04:06:43 0:46:02 0:34:15 0:11:47 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7621953 2024-03-25 21:09:23 2024-03-26 03:21:02 2024-03-26 03:56:52 0:35:50 0:23:58 0:11:52 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-03-26T03:43:54.200799+0000 mon.smithi007 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621954 2024-03-25 21:09:24 2024-03-26 03:23:13 2024-03-26 03:52:26 0:29:13 0:17:39 0:11:34 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-03-26T03:42:50.721791+0000 mon.smithi018 (mon.0) 268 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621955 2024-03-25 21:09:25 2024-03-26 03:25:23 2024-03-26 03:57:22 0:31:59 0:19:29 0:12:30 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_orch_cli} 1
Failure Reason:

Test failure: test_cephfs_mirror (tasks.cephadm_cases.test_cli.TestCephadmCLI)

fail 7621956 2024-03-25 21:09:26 2024-03-26 03:27:44 2024-03-26 03:58:42 0:30:58 0:13:44 0:17:14 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-03-26T03:52:58.184431+0000 mon.smithi001 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621957 2024-03-25 21:09:28 2024-03-26 03:36:12 2024-03-26 04:00:58 0:24:46 0:14:55 0:09:51 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
fail 7621958 2024-03-25 21:09:29 2024-03-26 03:36:12 2024-03-26 04:07:45 0:31:33 0:21:16 0:10:17 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-03-26T03:56:17.714813+0000 mon.smithi026 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621959 2024-03-25 21:09:30 2024-03-26 03:36:13 2024-03-26 04:10:37 0:34:24 0:24:18 0:10:06 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_orch_cli_mon} 5
pass 7621960 2024-03-25 21:09:31 2024-03-26 03:36:13 2024-03-26 04:12:46 0:36:33 0:25:35 0:10:58 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} 2
fail 7621961 2024-03-25 21:09:32 2024-03-26 03:36:14 2024-03-26 04:22:02 0:45:48 0:35:41 0:10:07 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-26T04:10:00.000130+0000 mon.a (mon.0) 1726 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

fail 7621962 2024-03-25 21:09:33 2024-03-26 03:36:14 2024-03-26 04:24:12 0:47:58 0:36:39 0:11:19 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7621963 2024-03-25 21:09:34 2024-03-26 03:36:14 2024-03-26 03:59:47 0:23:33 0:13:57 0:09:36 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} 2
fail 7621964 2024-03-25 21:09:35 2024-03-26 03:36:15 2024-03-26 04:00:43 0:24:28 0:13:54 0:10:34 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-03-26T03:54:49.827127+0000 mon.smithi113 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621965 2024-03-25 21:09:36 2024-03-26 03:36:15 2024-03-26 03:57:02 0:20:47 0:12:36 0:08:11 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
pass 7621966 2024-03-25 21:09:37 2024-03-26 03:36:15 2024-03-26 03:58:53 0:22:38 0:12:09 0:10:29 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
pass 7621967 2024-03-25 21:09:38 2024-03-26 03:36:16 2024-03-26 04:03:06 0:26:50 0:16:10 0:10:40 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_rgw_multisite} 3
fail 7621968 2024-03-25 21:09:39 2024-03-26 03:36:16 2024-03-26 04:01:53 0:25:37 0:14:42 0:10:55 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-03-26T03:54:27.815752+0000 mon.smithi100 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621969 2024-03-25 21:09:40 2024-03-26 03:36:17 2024-03-26 04:02:25 0:26:08 0:14:53 0:11:15 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-03-26T03:54:35.320947+0000 mon.smithi052 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621970 2024-03-25 21:09:41 2024-03-26 03:36:17 2024-03-26 04:32:21 0:56:04 0:45:59 0:10:05 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

"2024-03-26T03:52:56.609262+0000 mon.a (mon.0) 197 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7621971 2024-03-25 21:09:42 2024-03-26 03:36:17 2024-03-26 03:53:50 0:17:33 0:07:33 0:10:00 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/flannel rook/1.7.2} 1
Failure Reason:

Command failed on smithi047 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7621972 2024-03-25 21:09:43 2024-03-26 03:36:18 2024-03-26 04:12:34 0:36:16 0:25:29 0:10:47 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-03-26T03:57:49.297489+0000 mon.smithi046 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621973 2024-03-25 21:09:44 2024-03-26 03:36:18 2024-03-26 04:08:44 0:32:26 0:21:46 0:10:40 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_set_mon_crush_locations} 3
fail 7621974 2024-03-25 21:09:45 2024-03-26 03:36:18 2024-03-26 04:07:11 0:30:53 0:16:16 0:14:37 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

"2024-03-26T03:57:43.783805+0000 mon.smithi119 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621975 2024-03-25 21:09:46 2024-03-26 03:45:48 2024-03-26 04:34:53 0:49:05 0:39:17 0:09:48 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7621976 2024-03-25 21:09:47 2024-03-26 03:45:48 2024-03-26 04:18:25 0:32:37 0:18:41 0:13:56 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-03-26T04:09:41.740027+0000 mon.smithi012 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621977 2024-03-25 21:09:49 2024-03-26 03:48:29 2024-03-26 04:26:22 0:37:53 0:26:01 0:11:52 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
fail 7621978 2024-03-25 21:09:50 2024-03-26 03:50:40 2024-03-26 04:13:58 0:23:18 0:13:23 0:09:55 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-03-26T04:07:59.942648+0000 mon.smithi102 (mon.0) 269 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621979 2024-03-25 21:09:51 2024-03-26 03:50:40 2024-03-26 04:53:47 1:03:07 0:52:35 0:10:32 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-26T04:20:00.000140+0000 mon.a (mon.0) 2320 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

fail 7621980 2024-03-25 21:09:52 2024-03-26 03:50:41 2024-03-26 04:56:04 1:05:23 0:55:49 0:09:34 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-26T04:20:00.066943+0000 mon.a (mon.0) 869 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 20 pool(s) do not have an application enabled" in cluster log

pass 7621981 2024-03-25 21:09:53 2024-03-26 03:51:21 2024-03-26 04:18:42 0:27:21 0:13:18 0:14:03 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_adoption} 1
fail 7621982 2024-03-25 21:09:54 2024-03-26 03:51:31 2024-03-26 04:22:41 0:31:10 0:22:14 0:08:56 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-03-26T04:13:35.151852+0000 mon.smithi033 (mon.0) 272 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621983 2024-03-25 21:09:55 2024-03-26 03:51:32 2024-03-26 04:15:33 0:24:01 0:11:09 0:12:52 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_ca_signed_key} 2
fail 7621984 2024-03-25 21:09:56 2024-03-26 03:51:32 2024-03-26 04:16:20 0:24:48 0:14:41 0:10:07 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-03-26T04:08:34.011829+0000 mon.smithi019 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621985 2024-03-25 21:09:57 2024-03-26 03:51:33 2024-03-26 04:15:13 0:23:40 0:11:50 0:11:50 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-26T04:09:07.276203+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7621986 2024-03-25 21:09:58 2024-03-26 03:51:33 2024-03-26 04:17:59 0:26:26 0:14:49 0:11:37 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-03-26T04:10:11.919867+0000 mon.smithi164 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621987 2024-03-25 21:09:59 2024-03-26 03:51:33 2024-03-26 04:15:19 0:23:46 0:13:41 0:10:05 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-03-26T04:09:34.455735+0000 mon.smithi038 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621988 2024-03-25 21:10:00 2024-03-26 03:51:34 2024-03-26 04:21:05 0:29:31 0:20:19 0:09:12 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_cephadm} 1
fail 7621989 2024-03-25 21:10:01 2024-03-26 03:51:34 2024-03-26 04:25:51 0:34:17 0:23:36 0:10:41 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi003 with status 5: 'sudo systemctl stop ceph-73f48d58-eb26-11ee-95d0-87774f69a715@mon.smithi003'

fail 7621990 2024-03-25 21:10:02 2024-03-26 03:51:34 2024-03-26 04:22:02 0:30:28 0:20:35 0:09:53 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-03-26T04:12:44.007321+0000 mon.smithi114 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621991 2024-03-25 21:10:03 2024-03-26 03:51:35 2024-03-26 05:08:25 1:16:50 1:05:38 0:11:12 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-26T04:14:34.877351+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7621992 2024-03-25 21:10:04 2024-03-26 03:51:35 2024-03-26 04:52:21 1:00:46 0:49:56 0:10:50 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi144 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4aca1812-eb26-11ee-95d0-87774f69a715 -e sha1=5e70124b46c2a671efcaae4690cd9ea72670710a -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''

fail 7621993 2024-03-25 21:10:05 2024-03-26 03:51:35 2024-03-26 04:25:17 0:33:42 0:22:15 0:11:27 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} 2
Failure Reason:

"2024-03-26T04:20:00.000213+0000 mon.a (mon.0) 1174 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log

pass 7621994 2024-03-25 21:10:06 2024-03-26 03:51:36 2024-03-26 04:06:57 0:15:21 0:06:10 0:09:11 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm_repos} 1
fail 7621995 2024-03-25 21:10:07 2024-03-26 03:51:36 2024-03-26 04:16:36 0:25:00 0:15:26 0:09:34 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-03-26T04:08:45.002557+0000 mon.smithi055 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621996 2024-03-25 21:10:08 2024-03-26 03:51:36 2024-03-26 04:17:17 0:25:41 0:14:45 0:10:56 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
fail 7621997 2024-03-25 21:10:09 2024-03-26 03:51:37 2024-03-26 04:17:19 0:25:42 0:16:12 0:09:30 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-03-26T04:08:49.372372+0000 mon.smithi161 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7621998 2024-03-25 21:10:10 2024-03-26 03:51:37 2024-03-26 04:15:44 0:24:07 0:14:11 0:09:56 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-03-26T04:09:42.061705+0000 mon.smithi115 (mon.0) 264 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7621999 2024-03-25 21:10:11 2024-03-26 03:51:38 2024-03-26 04:15:21 0:23:43 0:13:59 0:09:44 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_timeout} 1
fail 7622000 2024-03-25 21:10:13 2024-03-26 03:51:48 2024-03-26 04:24:32 0:32:44 0:21:46 0:10:58 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-03-26T04:13:00.329076+0000 mon.smithi099 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622001 2024-03-25 21:10:14 2024-03-26 03:51:48 2024-03-26 04:52:25 1:00:37 0:43:44 0:16:53 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7622002 2024-03-25 21:10:15 2024-03-26 04:07:07 2024-03-26 04:55:23 0:48:16 0:36:46 0:11:30 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-03-26T04:29:51.414859+0000 mon.smithi018 (mon.0) 505 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

pass 7622003 2024-03-25 21:10:16 2024-03-26 04:07:08 2024-03-26 05:02:41 0:55:33 0:45:11 0:10:22 smithi main ubuntu 22.04 orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
pass 7622004 2024-03-25 21:10:17 2024-03-26 04:07:08 2024-03-26 04:29:54 0:22:46 0:13:44 0:09:02 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} 2
pass 7622005 2024-03-25 21:10:18 2024-03-26 04:07:08 2024-03-26 04:46:52 0:39:44 0:25:23 0:14:21 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
fail 7622006 2024-03-25 21:10:19 2024-03-26 04:07:09 2024-03-26 04:37:50 0:30:41 0:17:00 0:13:41 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-03-26T04:28:18.283224+0000 mon.smithi113 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622007 2024-03-25 21:10:20 2024-03-26 04:07:09 2024-03-26 04:29:26 0:22:17 0:11:28 0:10:49 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} 1
fail 7622008 2024-03-25 21:10:21 2024-03-26 04:07:09 2024-03-26 04:29:37 0:22:28 0:11:22 0:11:06 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-26T04:24:54.217538+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7622009 2024-03-25 21:10:22 2024-03-26 04:07:10 2024-03-26 04:37:46 0:30:36 0:19:30 0:11:06 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_extra_daemon_features} 2
fail 7622010 2024-03-25 21:10:23 2024-03-26 04:07:10 2024-03-26 04:49:07 0:41:57 0:30:35 0:11:22 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-26T04:40:00.000138+0000 mon.a (mon.0) 1549 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

fail 7622011 2024-03-25 21:10:24 2024-03-26 04:07:11 2024-03-26 04:35:02 0:27:51 0:17:25 0:10:26 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-03-26T04:25:18.795451+0000 mon.smithi052 (mon.0) 261 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622012 2024-03-25 21:10:25 2024-03-26 04:07:11 2024-03-26 04:25:25 0:18:14 0:07:29 0:10:45 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/host rook/master} 3
Failure Reason:

Command failed on smithi047 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7622013 2024-03-25 21:10:26 2024-03-26 04:07:11 2024-03-26 04:58:31 0:51:20 0:38:55 0:12:25 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} 2
Failure Reason:

"2024-03-26T04:30:17.028796+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7622014 2024-03-25 21:10:27 2024-03-26 04:07:12 2024-03-26 04:38:41 0:31:29 0:19:34 0:11:55 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-03-26T04:29:19.406056+0000 mon.smithi080 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622015 2024-03-25 21:10:28 2024-03-26 04:08:52 2024-03-26 04:38:19 0:29:27 0:15:37 0:13:50 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_host_drain} 3
fail 7622016 2024-03-25 21:10:29 2024-03-26 04:10:43 2024-03-26 04:42:36 0:31:53 0:21:22 0:10:31 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-03-26T04:31:45.651935+0000 mon.smithi155 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622017 2024-03-25 21:10:30 2024-03-26 04:10:43 2024-03-26 04:37:55 0:27:12 0:14:36 0:12:36 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-03-26T04:29:44.942625+0000 mon.smithi050 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622018 2024-03-25 21:10:31 2024-03-26 04:12:54 2024-03-26 04:54:38 0:41:44 0:28:12 0:13:32 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-03-26T04:31:32.298227+0000 mon.a (mon.0) 169 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7622019 2024-03-25 21:10:32 2024-03-26 04:15:25 2024-03-26 04:38:32 0:23:07 0:13:45 0:09:22 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7622020 2024-03-25 21:10:33 2024-03-26 04:15:35 2024-03-26 04:42:28 0:26:53 0:16:15 0:10:38 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
fail 7622021 2024-03-25 21:10:34 2024-03-26 04:17:26 2024-03-26 05:11:28 0:54:02 0:37:30 0:16:32 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi181 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

fail 7622022 2024-03-25 21:10:35 2024-03-26 04:18:46 2024-03-26 04:45:12 0:26:26 0:13:37 0:12:49 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-03-26T04:38:31.584692+0000 mon.smithi117 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622023 2024-03-25 21:10:36 2024-03-26 04:22:37 2024-03-26 04:48:13 0:25:36 0:14:51 0:10:45 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-03-26T04:40:07.804030+0000 mon.smithi063 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622024 2024-03-25 21:10:37 2024-03-26 04:22:37 2024-03-26 05:07:32 0:44:55 0:36:30 0:08:25 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-26T04:50:00.000209+0000 mon.a (mon.0) 1262 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

fail 7622025 2024-03-25 21:10:38 2024-03-26 04:22:38 2024-03-26 04:53:03 0:30:25 0:20:05 0:10:20 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-03-26T04:44:09.723626+0000 mon.smithi149 (mon.0) 256 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622026 2024-03-25 21:10:39 2024-03-26 04:22:38 2024-03-26 04:59:11 0:36:33 0:25:04 0:11:29 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_monitoring_stack_basic} 3
fail 7622027 2024-03-25 21:10:41 2024-03-26 04:22:38 2024-03-26 04:50:33 0:27:55 0:17:20 0:10:35 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-03-26T04:39:57.669045+0000 mon.smithi121 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622028 2024-03-25 21:10:42 2024-03-26 04:22:39 2024-03-26 04:45:35 0:22:56 0:12:12 0:10:44 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-26T04:38:59.972242+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7622029 2024-03-25 21:10:43 2024-03-26 04:22:39 2024-03-26 05:08:15 0:45:36 0:35:01 0:10:35 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-26T04:50:00.000191+0000 mon.a (mon.0) 1630 : cluster 3 [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg peering" in cluster log

fail 7622030 2024-03-25 21:10:44 2024-03-26 04:22:39 2024-03-26 04:47:40 0:25:01 0:15:44 0:09:17 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_orch_cli} 1
Failure Reason:

Test failure: test_daemon_restart (tasks.cephadm_cases.test_cli.TestCephadmCLI)

dead 7622031 2024-03-25 21:10:45 2024-03-26 04:22:40 2024-03-26 16:30:25 12:07:45 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

hit max job timeout

fail 7622032 2024-03-25 21:10:46 2024-03-26 04:22:40 2024-03-26 05:10:12 0:47:32 0:37:50 0:09:42 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7622033 2024-03-25 21:10:47 2024-03-26 04:22:41 2024-03-26 04:45:42 0:23:01 0:12:27 0:10:34 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-03-26T04:39:23.337778+0000 mon.smithi161 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622034 2024-03-25 21:10:48 2024-03-26 04:22:41 2024-03-26 04:52:43 0:30:02 0:19:10 0:10:52 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-03-26T04:43:46.346465+0000 mon.smithi131 (mon.0) 255 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622035 2024-03-25 21:10:49 2024-03-26 04:22:42 2024-03-26 05:06:17 0:43:35 0:23:58 0:19:37 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_orch_cli_mon} 5
fail 7622036 2024-03-25 21:10:50 2024-03-26 04:30:03 2024-03-26 05:00:50 0:30:47 0:13:03 0:17:44 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-03-26T04:54:21.881065+0000 mon.smithi135 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622037 2024-03-25 21:10:51 2024-03-26 04:37:45 2024-03-26 05:51:07 1:13:22 1:04:00 0:09:22 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-26T04:58:27.498389+0000 mon.a (mon.0) 182 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7622038 2024-03-25 21:10:52 2024-03-26 04:37:45 2024-03-26 05:11:44 0:33:59 0:24:34 0:09:25 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
fail 7622039 2024-03-25 21:10:53 2024-03-26 04:37:55 2024-03-26 05:00:29 0:22:34 0:14:39 0:07:55 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-03-26T04:54:03.960881+0000 mon.smithi033 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622040 2024-03-25 21:10:54 2024-03-26 04:38:06 2024-03-26 05:08:41 0:30:35 0:20:01 0:10:34 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} 3
fail 7622041 2024-03-25 21:10:55 2024-03-26 04:38:06 2024-03-26 05:27:51 0:49:45 0:38:12 0:11:33 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-03-26T04:57:37.450575+0000 mon.a (mon.0) 170 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7622042 2024-03-25 21:10:57 2024-03-26 04:38:06 2024-03-26 05:12:34 0:34:28 0:22:23 0:12:05 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} 2
fail 7622043 2024-03-25 21:10:58 2024-03-26 04:38:17 2024-03-26 05:59:17 1:21:00 1:10:16 0:10:44 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

reached maximum tries (301) after waiting for 300 seconds

fail 7622044 2024-03-25 21:10:59 2024-03-26 04:38:17 2024-03-26 05:09:18 0:31:01 0:20:19 0:10:42 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-03-26T05:00:30.557049+0000 mon.smithi003 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622045 2024-03-25 21:11:00 2024-03-26 04:38:18 2024-03-26 05:06:24 0:28:06 0:17:46 0:10:20 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} 3
fail 7622046 2024-03-25 21:11:01 2024-03-26 04:38:18 2024-03-26 08:28:39 3:50:21 3:36:13 0:14:08 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi194 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

pass 7622047 2024-03-25 21:11:02 2024-03-26 04:38:18 2024-03-26 05:01:56 0:23:38 0:12:26 0:11:12 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli} 2
fail 7622048 2024-03-25 21:11:03 2024-03-26 04:38:19 2024-03-26 05:03:35 0:25:16 0:15:15 0:10:01 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-03-26T04:55:50.568878+0000 mon.smithi029 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622049 2024-03-25 21:11:04 2024-03-26 04:38:19 2024-03-26 05:11:08 0:32:49 0:20:43 0:12:06 smithi main ubuntu 22.04 orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} 1
pass 7622050 2024-03-25 21:11:05 2024-03-26 04:38:19 2024-03-26 05:01:14 0:22:55 0:11:54 0:11:01 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
fail 7622051 2024-03-25 21:11:06 2024-03-26 04:38:20 2024-03-26 05:03:36 0:25:16 0:15:43 0:09:33 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-03-26T04:55:21.014431+0000 mon.smithi142 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622052 2024-03-25 21:11:07 2024-03-26 04:38:20 2024-03-26 04:57:12 0:18:52 0:07:32 0:11:20 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/calico rook/1.7.2} 3
Failure Reason:

Command failed on smithi088 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7622053 2024-03-25 21:11:08 2024-03-26 04:38:20 2024-03-26 05:46:04 1:07:44 0:56:51 0:10:53 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-26T05:10:00.000257+0000 mon.a (mon.0) 1299 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

pass 7622054 2024-03-25 21:11:09 2024-03-26 04:38:21 2024-03-26 04:58:00 0:19:39 0:10:24 0:09:15 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_adoption} 1
fail 7622055 2024-03-25 21:11:10 2024-03-26 04:38:21 2024-03-26 05:14:10 0:35:49 0:21:41 0:14:08 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-03-26T05:02:51.964643+0000 mon.smithi037 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622056 2024-03-25 21:11:11 2024-03-26 04:42:32 2024-03-26 05:10:01 0:27:29 0:13:10 0:14:19 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-03-26T05:03:30.891711+0000 mon.smithi081 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622057 2024-03-25 21:11:12 2024-03-26 04:46:53 2024-03-26 05:08:25 0:21:32 0:11:04 0:10:28 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_ca_signed_key} 2
Failure Reason:

Command failed on smithi007 with status 5: 'sudo systemctl stop ceph-5b8931a8-eb2e-11ee-95d0-87774f69a715@mon.a'

fail 7622058 2024-03-25 21:11:13 2024-03-26 04:46:53 2024-03-26 05:26:19 0:39:26 0:23:20 0:16:06 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-03-26T05:16:07.207818+0000 mon.smithi006 (mon.0) 315 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622059 2024-03-25 21:11:14 2024-03-26 04:53:24 2024-03-26 05:18:42 0:25:18 0:14:56 0:10:22 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
fail 7622060 2024-03-25 21:11:15 2024-03-26 04:53:45 2024-03-26 05:44:23 0:50:38 0:39:01 0:11:37 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} 2
Failure Reason:

"2024-03-26T05:16:26.556496+0000 mon.a (mon.0) 180 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7622061 2024-03-25 21:11:16 2024-03-26 04:53:45 2024-03-26 05:43:42 0:49:57 0:37:49 0:12:08 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7622062 2024-03-25 21:11:17 2024-03-26 04:53:45 2024-03-26 05:23:05 0:29:20 0:17:01 0:12:19 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-03-26T05:13:05.325172+0000 mon.smithi137 (mon.0) 265 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622063 2024-03-25 21:11:18 2024-03-26 04:53:46 2024-03-26 05:24:33 0:30:47 0:22:14 0:08:33 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm} 1
fail 7622064 2024-03-25 21:11:20 2024-03-26 04:53:46 2024-03-26 05:30:18 0:36:32 0:24:47 0:11:45 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-03-26T05:15:44.300432+0000 mon.smithi072 (mon.0) 252 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622065 2024-03-25 21:11:21 2024-03-26 04:53:47 2024-03-26 05:57:41 1:03:54 0:53:43 0:10:11 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

"2024-03-26T05:11:04.844524+0000 mon.a (mon.0) 360 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7622066 2024-03-25 21:11:22 2024-03-26 04:53:47 2024-03-26 05:16:37 0:22:50 0:14:02 0:08:48 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-03-26T05:10:42.798068+0000 mon.smithi038 (mon.0) 262 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622067 2024-03-25 21:11:23 2024-03-26 04:53:47 2024-03-26 05:35:21 0:41:34 0:30:54 0:10:40 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-26T05:20:00.000125+0000 mon.a (mon.0) 1022 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

fail 7622068 2024-03-25 21:11:24 2024-03-26 04:53:48 2024-03-26 05:21:28 0:27:40 0:15:18 0:12:22 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-03-26T05:12:42.437201+0000 mon.smithi159 (mon.0) 263 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622069 2024-03-25 21:11:25 2024-03-26 04:53:48 2024-03-26 05:14:49 0:21:01 0:12:02 0:08:59 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-26T05:10:14.943090+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7622070 2024-03-25 21:11:26 2024-03-26 04:53:48 2024-03-26 05:09:38 0:15:50 0:06:16 0:09:34 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos} 1
fail 7622071 2024-03-25 21:11:27 2024-03-26 04:53:49 2024-03-26 05:17:40 0:23:51 0:14:16 0:09:35 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-03-26T05:11:09.818687+0000 mon.smithi161 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622072 2024-03-25 21:11:28 2024-03-26 04:53:49 2024-03-26 05:24:16 0:30:27 0:19:30 0:10:57 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_timeout} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi150 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py'

fail 7622073 2024-03-25 21:11:29 2024-03-26 04:53:50 2024-03-26 08:45:32 3:51:42 3:38:17 0:13:25 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi148 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

fail 7622074 2024-03-25 21:11:30 2024-03-26 04:53:50 2024-03-26 05:23:48 0:29:58 0:19:33 0:10:25 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-03-26T05:14:13.344710+0000 mon.smithi129 (mon.0) 250 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622075 2024-03-25 21:11:31 2024-03-26 04:53:51 2024-03-26 06:00:42 1:06:51 0:51:39 0:15:12 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-26T05:30:00.000148+0000 mon.a (mon.0) 2465 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log

fail 7622076 2024-03-25 21:11:32 2024-03-26 23:38:02 2024-03-27 00:11:01 0:32:59 0:21:14 0:11:45 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-03-27T00:00:14.029428+0000 mon.smithi071 (mon.0) 251 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

fail 7622077 2024-03-25 21:11:33 2024-03-26 23:39:12 2024-03-27 00:04:46 0:25:34 0:14:14 0:11:20 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-03-26T23:56:41.849696+0000 mon.smithi042 (mon.0) 260 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log

pass 7622078 2024-03-25 21:11:34 2024-03-26 23:39:43 2024-03-27 00:07:12 0:27:29 0:15:46 0:11:43 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7622079 2024-03-25 21:11:35 2024-03-26 23:39:53 2024-03-27 00:02:48 0:22:55 0:13:35 0:09:20 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_extra_daemon_features} 2
fail 7622080 2024-03-25 21:11:37 2024-03-26 23:39:54 2024-03-27 00:40:05 1:00:11 0:49:12 0:10:59 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-27T00:01:21.519588+0000 mon.a (mon.0) 186 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7622081 2024-03-25 21:11:38 2024-03-26 23:40:24 2024-03-27 00:10:03 0:29:39 0:18:10 0:11:29 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-03-26T23:58:27.773734+0000 mon.smithi098 (mon.0) 259 : cluster 3 [WRN] CEPHADM_DAEMON_PLACE_FAIL: Failed to place 1 daemon(s)" in cluster log