Pulpito

teuthology-2024-06-07_21:08:01-orch-squid-distro-default-smithi

User Scheduled Started Updated Runtime Suite Branch Machine Type Revision Pass Fail Dead
teuthology 2024-06-07 21:08:01 2024-06-08 22:39:10 2024-06-09 12:51:27 14:12:17 orch squid smithi 18001b1 99 66 20

Average wait time: 0:13:48

See other runs of suite 'orch' on branch 'squid'?

See other runs on branch 'squid'?

See other runs scheduled on 2024-06-07?

See detail view?

Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7745269 2024-06-07 21:08:52 2024-06-08 22:38:50 2024-06-08 23:08:45 0:29:55 0:19:19 0:10:36 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7745270 2024-06-07 21:08:52 2024-06-08 22:39:10 2024-06-09 00:10:53 1:31:43 1:08:23 0:23:20 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2
fail 7745271 2024-06-07 21:08:53 2024-06-08 22:39:10 2024-06-08 23:32:11 0:53:01 0:31:20 0:21:41 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
Failure Reason:

"2024-06-08T23:27:22.886435+0000 mon.a (mon.0) 233 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7745272 2024-06-07 21:08:54 2024-06-08 22:39:11 2024-06-08 23:08:38 0:29:27 0:20:37 0:08:50 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-06-08T23:04:34.927918+0000 mon.smithi096 (mon.0) 805 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7745273 2024-06-07 21:08:55 2024-06-08 22:39:11 2024-06-08 23:36:56 0:57:45 0:38:48 0:18:57 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

reached maximum tries (301) after waiting for 300 seconds

fail 7745274 2024-06-07 21:08:56 2024-06-08 22:40:02 2024-06-08 23:29:10 0:49:08 0:23:05 0:26:03 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
Failure Reason:

"2024-06-08T23:20:27.611724+0000 mon.a (mon.0) 420 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745275 2024-06-07 21:08:57 2024-06-08 22:40:22 2024-06-08 23:40:07 0:59:45 0:49:54 0:09:51 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
dead 7745276 2024-06-07 21:08:58 2024-06-08 22:40:32 2024-06-09 10:49:37 12:09:05 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

pass 7745277 2024-06-07 21:08:59 2024-06-08 22:40:33 2024-06-08 23:47:34 1:07:01 0:45:44 0:21:17 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_monitoring_stack_basic} 3
pass 7745278 2024-06-07 21:09:00 2024-06-08 22:40:43 2024-06-08 23:14:51 0:34:08 0:21:13 0:12:55 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} 2
fail 7745279 2024-06-07 21:09:01 2024-06-08 22:41:24 2024-06-08 23:40:25 0:59:01 0:40:12 0:18:49 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1717889311.5125897 mgr.smithi057.lqzkil (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7745280 2024-06-07 21:09:02 2024-06-08 22:41:24 2024-06-09 00:00:24 1:19:00 0:55:26 0:23:34 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-06-08T23:29:50.284491+0000 mon.smithi016 (mon.0) 774 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7745281 2024-06-07 21:09:03 2024-06-08 22:41:34 2024-06-08 23:46:22 1:04:48 0:54:47 0:10:01 smithi main ubuntu 22.04 orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
Failure Reason:

"2024-06-08T23:08:37.120698+0000 mon.a (mon.0) 333 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log

pass 7745282 2024-06-07 21:09:04 2024-06-08 22:42:45 2024-06-08 23:29:42 0:46:57 0:27:26 0:19:31 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_adoption} 1
dead 7745283 2024-06-07 21:09:04 2024-06-08 22:42:45 2024-06-08 22:43:59 0:01:14 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi154

pass 7745284 2024-06-07 21:09:05 2024-06-08 22:42:55 2024-06-08 23:25:00 0:42:05 0:21:45 0:20:20 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} 2
fail 7745285 2024-06-07 21:09:06 2024-06-08 22:43:46 2024-06-08 23:51:06 1:07:20 0:41:28 0:25:52 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-06-08T23:21:09.545374+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

dead 7745286 2024-06-07 21:09:07 2024-06-08 22:43:56 2024-06-08 22:45:20 0:01:24 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi045

pass 7745287 2024-06-07 21:09:08 2024-06-08 22:44:17 2024-06-08 23:21:53 0:37:36 0:20:03 0:17:33 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final} 2
pass 7745288 2024-06-07 21:09:09 2024-06-08 22:44:37 2024-06-08 23:21:39 0:37:02 0:18:42 0:18:20 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} 1
fail 7745289 2024-06-07 21:09:10 2024-06-08 22:44:37 2024-06-08 23:03:23 0:18:46 0:07:39 0:11:07 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/flannel rook/master} 1
Failure Reason:

Command failed on smithi029 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7745290 2024-06-07 21:09:11 2024-06-08 22:44:38 2024-06-08 23:28:36 0:43:58 0:17:50 0:26:08 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
fail 7745291 2024-06-07 21:09:12 2024-06-08 22:45:08 2024-06-08 23:36:51 0:51:43 0:40:29 0:11:14 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

"2024-06-08T23:07:24.633054+0000 mon.a (mon.0) 455 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745292 2024-06-07 21:09:13 2024-06-08 22:45:08 2024-06-08 23:41:10 0:56:02 0:26:29 0:29:33 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_rgw_multisite} 3
pass 7745293 2024-06-07 21:09:14 2024-06-08 22:45:09 2024-06-08 23:30:04 0:44:55 0:20:40 0:24:15 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
pass 7745294 2024-06-07 21:09:15 2024-06-08 22:45:09 2024-06-08 23:44:04 0:58:55 0:30:41 0:28:14 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} 2
pass 7745295 2024-06-07 21:09:16 2024-06-08 22:53:11 2024-06-08 23:24:40 0:31:29 0:20:24 0:11:05 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
fail 7745296 2024-06-07 21:09:16 2024-06-08 22:54:51 2024-06-09 00:19:47 1:24:56 1:01:17 0:23:39 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

"2024-06-08T23:43:09.332788+0000 mon.a (mon.0) 1026 : cluster [WRN] Health check failed: 17 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745297 2024-06-07 21:09:17 2024-06-08 22:56:42 2024-06-08 23:32:33 0:35:51 0:22:55 0:12:56 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_set_mon_crush_locations} 3
pass 7745298 2024-06-07 21:09:18 2024-06-08 23:00:53 2024-06-08 23:38:17 0:37:24 0:25:51 0:11:33 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
dead 7745299 2024-06-07 21:09:19 2024-06-08 23:02:33 2024-06-08 23:04:07 0:01:34 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi139

pass 7745300 2024-06-07 21:09:20 2024-06-08 23:03:04 2024-06-08 23:38:45 0:35:41 0:15:18 0:20:23 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
fail 7745301 2024-06-07 21:09:21 2024-06-08 23:03:24 2024-06-09 00:22:57 1:19:33 1:02:26 0:17:07 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7745302 2024-06-07 21:09:22 2024-06-08 23:03:35 2024-06-08 23:48:38 0:45:03 0:25:08 0:19:55 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_cephadm_timeout} 1
Failure Reason:

"2024-06-08T23:45:59.315512+0000 mon.a (mon.0) 212 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7745303 2024-06-07 21:09:23 2024-06-08 23:03:35 2024-06-08 23:37:01 0:33:26 0:15:30 0:17:56 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} 2
pass 7745304 2024-06-07 21:09:24 2024-06-08 23:05:56 2024-06-08 23:41:09 0:35:13 0:14:57 0:20:16 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final} 2
fail 7745305 2024-06-07 21:09:25 2024-06-08 23:07:46 2024-06-08 23:36:07 0:28:21 0:17:16 0:11:05 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_ca_signed_key} 2
Failure Reason:

"2024-06-08T23:32:29.075052+0000 mon.a (mon.0) 340 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745306 2024-06-07 21:09:26 2024-06-08 23:08:47 2024-06-08 23:38:22 0:29:35 0:21:12 0:08:23 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7745307 2024-06-07 21:09:27 2024-06-08 23:08:47 2024-06-09 00:01:47 0:53:00 0:39:00 0:14:00 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
dead 7745308 2024-06-07 21:09:27 2024-06-08 23:12:08 2024-06-08 23:13:32 0:01:24 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi155

pass 7745309 2024-06-07 21:09:28 2024-06-08 23:12:28 2024-06-08 23:47:21 0:34:53 0:18:42 0:16:11 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
pass 7745310 2024-06-07 21:09:29 2024-06-08 23:14:09 2024-06-08 23:46:21 0:32:12 0:19:09 0:13:03 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7745311 2024-06-07 21:09:30 2024-06-08 23:14:59 2024-06-09 00:33:23 1:18:24 1:04:47 0:13:37 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
pass 7745312 2024-06-07 21:09:31 2024-06-08 23:16:10 2024-06-08 23:49:15 0:33:05 0:21:29 0:11:36 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_cephadm} 1
fail 7745313 2024-06-07 21:09:32 2024-06-08 23:16:10 2024-06-09 00:21:52 1:05:42 0:52:08 0:13:34 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-06-08T23:54:16.917709+0000 mon.a (mon.0) 1022 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7745314 2024-06-07 21:09:33 2024-06-08 23:16:11 2024-06-08 23:53:35 0:37:24 0:21:40 0:15:44 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
fail 7745315 2024-06-07 21:09:34 2024-06-08 23:20:12 2024-06-09 00:01:26 0:41:14 0:26:17 0:14:57 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1717890584.3540394 mon.smithi069 (mon.0) 121 : cluster [WRN] Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log

fail 7745316 2024-06-07 21:09:35 2024-06-08 23:22:02 2024-06-08 23:55:42 0:33:40 0:23:41 0:09:59 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-06-08T23:52:08.520332+0000 mon.a (mon.0) 558 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7745317 2024-06-07 21:09:36 2024-06-08 23:22:03 2024-06-08 23:54:40 0:32:37 0:19:23 0:13:14 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} 2
fail 7745318 2024-06-07 21:09:37 2024-06-08 23:24:43 2024-06-08 23:58:48 0:34:05 0:24:12 0:09:53 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-08T23:52:01.300232+0000 mon.smithi001 (mon.0) 824 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7745319 2024-06-07 21:09:38 2024-06-08 23:25:04 2024-06-08 23:50:06 0:25:02 0:06:47 0:18:15 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_cephadm_repos} 1
dead 7745320 2024-06-07 21:09:39 2024-06-08 23:28:45 2024-06-08 23:47:48 0:19:03 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
Failure Reason:

Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds

pass 7745321 2024-06-07 21:09:40 2024-06-08 23:28:45 2024-06-08 23:58:06 0:29:21 0:16:43 0:12:38 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7745322 2024-06-07 21:09:40 2024-06-08 23:29:15 2024-06-08 23:59:00 0:29:45 0:16:48 0:12:57 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7745323 2024-06-07 21:09:41 2024-06-08 23:30:06 2024-06-09 00:00:37 0:30:31 0:16:21 0:14:10 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-08T23:56:00.023534+0000 mon.smithi154 (mon.0) 854 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7745324 2024-06-07 21:09:42 2024-06-08 23:31:36 2024-06-09 00:26:18 0:54:42 0:44:24 0:10:18 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
fail 7745325 2024-06-07 21:09:43 2024-06-08 23:32:37 2024-06-09 00:47:10 1:14:33 1:01:26 0:13:07 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

"2024-06-09T00:10:16.021340+0000 mon.a (mon.0) 977 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745326 2024-06-07 21:09:44 2024-06-08 23:35:28 2024-06-09 00:07:24 0:31:56 0:19:12 0:12:44 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_extra_daemon_features} 2
fail 7745327 2024-06-07 21:09:45 2024-06-08 23:36:08 2024-06-09 00:10:35 0:34:27 0:20:29 0:13:58 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-06-09T00:06:38.931868+0000 mon.smithi028 (mon.0) 764 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

fail 7745328 2024-06-07 21:09:46 2024-06-08 23:36:59 2024-06-09 00:18:03 0:41:04 0:27:58 0:13:06 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-06-09T00:06:29.277786+0000 mgr.smithi149.mafmku (mgr.14242) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

pass 7745329 2024-06-07 21:09:47 2024-06-08 23:36:59 2024-06-09 00:31:32 0:54:33 0:40:24 0:14:09 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_orch_cli_mon} 5
pass 7745330 2024-06-07 21:09:48 2024-06-08 23:38:20 2024-06-09 00:02:00 0:23:40 0:14:06 0:09:34 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} 2
pass 7745331 2024-06-07 21:09:49 2024-06-08 23:38:30 2024-06-09 00:06:26 0:27:56 0:14:52 0:13:04 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} 2
dead 7745332 2024-06-07 21:09:50 2024-06-08 23:38:50 2024-06-08 23:41:14 0:02:24 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi120

pass 7745333 2024-06-07 21:09:51 2024-06-08 23:40:11 2024-06-09 00:04:52 0:24:41 0:13:23 0:11:18 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
fail 7745334 2024-06-07 21:09:52 2024-06-08 23:40:31 2024-06-08 23:59:09 0:18:38 0:08:35 0:10:03 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/host rook/1.7.2} 3
Failure Reason:

Command failed on smithi042 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

dead 7745335 2024-06-07 21:09:52 2024-06-08 23:41:12 2024-06-08 23:42:16 0:01:04 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi088

fail 7745336 2024-06-07 21:09:53 2024-06-08 23:41:12 2024-06-09 00:29:25 0:48:13 0:31:13 0:17:00 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_host_drain} 3
Failure Reason:

"2024-06-09T00:23:40.055163+0000 mon.a (mon.0) 460 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7745337 2024-06-07 21:09:54 2024-06-08 23:46:23 2024-06-09 00:17:18 0:30:55 0:18:48 0:12:07 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-06-09T00:13:40.754372+0000 mon.smithi122 (mon.0) 824 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7745338 2024-06-07 21:09:55 2024-06-08 23:46:24 2024-06-09 00:37:56 0:51:32 0:35:40 0:15:52 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

reached maximum tries (301) after waiting for 300 seconds

pass 7745339 2024-06-07 21:09:56 2024-06-08 23:47:04 2024-06-09 00:54:40 1:07:36 0:57:10 0:10:26 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
dead 7745340 2024-06-07 21:09:57 2024-06-08 23:47:04 2024-06-09 11:56:49 12:09:45 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

pass 7745341 2024-06-07 21:09:58 2024-06-08 23:47:05 2024-06-09 00:16:26 0:29:21 0:19:03 0:10:18 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7745342 2024-06-07 21:09:59 2024-06-08 23:47:05 2024-06-09 00:27:47 0:40:42 0:30:04 0:10:38 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7745343 2024-06-07 21:10:00 2024-06-08 23:47:05 2024-06-09 00:51:06 1:04:01 0:51:19 0:12:42 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
pass 7745344 2024-06-07 21:10:01 2024-06-08 23:47:16 2024-06-09 00:16:25 0:29:09 0:17:52 0:11:17 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final} 2
pass 7745345 2024-06-07 21:10:02 2024-06-08 23:49:17 2024-06-09 00:36:42 0:47:25 0:35:57 0:11:28 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} 2
fail 7745346 2024-06-07 21:10:03 2024-06-08 23:51:07 2024-06-09 00:34:19 0:43:12 0:30:14 0:12:58 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1717892642.4285612 mgr.smithi148.vwdskp (mgr.14234) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

pass 7745347 2024-06-07 21:10:04 2024-06-08 23:51:08 2024-06-09 00:23:06 0:31:58 0:16:56 0:15:02 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_adoption} 1
pass 7745348 2024-06-07 21:10:05 2024-06-08 23:53:39 2024-06-09 00:26:08 0:32:29 0:18:40 0:13:49 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7745349 2024-06-07 21:10:05 2024-06-08 23:54:39 2024-06-09 00:27:39 0:33:00 0:24:00 0:09:00 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} 2
pass 7745350 2024-06-07 21:10:06 2024-06-08 23:54:49 2024-06-09 00:31:00 0:36:11 0:13:28 0:22:43 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final} 2
fail 7745351 2024-06-07 21:10:07 2024-06-08 23:55:50 2024-06-09 00:33:25 0:37:35 0:24:44 0:12:51 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_monitoring_stack_basic} 3
Failure Reason:

"2024-06-09T00:21:24.139266+0000 mon.a (mon.0) 432 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745352 2024-06-07 21:10:08 2024-06-08 23:57:51 2024-06-09 00:31:42 0:33:51 0:20:16 0:13:35 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
dead 7745353 2024-06-07 21:10:09 2024-06-08 23:58:11 2024-06-08 23:59:55 0:01:44 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi001

pass 7745354 2024-06-07 21:10:10 2024-06-08 23:58:51 2024-06-09 00:34:10 0:35:19 0:13:11 0:22:08 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
dead 7745355 2024-06-07 21:10:11 2024-06-08 23:59:02 2024-06-09 00:00:16 0:01:14 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_rgw_multisite} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi152

fail 7745356 2024-06-07 21:10:12 2024-06-08 23:59:12 2024-06-09 00:33:18 0:34:06 0:15:12 0:18:54 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-06-09T00:29:28.459316+0000 mon.smithi016 (mon.0) 812 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7745357 2024-06-07 21:10:13 2024-06-09 00:01:29 2024-06-09 00:34:52 0:33:23 0:14:00 0:19:23 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
fail 7745358 2024-06-07 21:10:14 2024-06-09 00:01:29 2024-06-09 01:08:55 1:07:26 0:51:48 0:15:38 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-06-09T00:33:29.264676+0000 mon.a (mon.0) 1199 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7745359 2024-06-07 21:10:15 2024-06-09 00:01:49 2024-06-09 00:59:07 0:57:18 0:39:54 0:17:24 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7745360 2024-06-07 21:10:16 2024-06-09 00:02:10 2024-06-09 00:31:40 0:29:30 0:19:21 0:10:09 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_cephadm_timeout} 1
Failure Reason:

"2024-06-09T00:29:02.883494+0000 mon.a (mon.0) 213 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7745361 2024-06-07 21:10:17 2024-06-09 00:02:10 2024-06-09 00:32:14 0:30:04 0:19:33 0:10:31 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} 2
pass 7745362 2024-06-07 21:10:18 2024-06-09 00:02:30 2024-06-09 00:35:02 0:32:32 0:21:18 0:11:14 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7745363 2024-06-07 21:10:18 2024-06-09 00:02:31 2024-06-09 00:47:47 0:45:16 0:25:44 0:19:32 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys} 2
fail 7745364 2024-06-07 21:10:19 2024-06-09 00:02:31 2024-06-09 00:46:18 0:43:47 0:27:18 0:16:29 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
Failure Reason:

"2024-06-09T00:32:16.707233+0000 mon.a (mon.0) 412 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

dead 7745365 2024-06-07 21:10:20 2024-06-09 00:05:02 2024-06-09 12:25:39 12:20:37 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

hit max job timeout

pass 7745366 2024-06-07 21:10:21 2024-06-09 00:06:32 2024-06-09 00:41:05 0:34:33 0:17:33 0:17:00 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_set_mon_crush_locations} 3
pass 7745367 2024-06-07 21:10:22 2024-06-09 00:10:43 2024-06-09 00:40:37 0:29:54 0:15:54 0:14:00 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7745368 2024-06-07 21:10:23 2024-06-09 00:11:04 2024-06-09 00:41:35 0:30:31 0:15:24 0:15:07 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
pass 7745369 2024-06-07 21:10:24 2024-06-09 00:16:35 2024-06-09 00:41:59 0:25:24 0:13:43 0:11:41 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7745370 2024-06-07 21:10:25 2024-06-09 00:16:35 2024-06-09 00:42:16 0:25:41 0:16:12 0:09:29 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_ca_signed_key} 2
fail 7745371 2024-06-07 21:10:26 2024-06-09 00:17:26 2024-06-09 00:49:31 0:32:05 0:22:55 0:09:10 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-06-09T00:46:53.914806+0000 mon.smithi059 (mon.0) 962 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745372 2024-06-07 21:10:27 2024-06-09 00:17:56 2024-06-09 01:33:12 1:15:16 1:04:16 0:11:00 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
fail 7745373 2024-06-07 21:10:28 2024-06-09 00:18:07 2024-06-09 00:54:52 0:36:45 0:25:56 0:10:49 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1717893861.3535802 mgr.smithi129.smlmff (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

pass 7745374 2024-06-07 21:10:28 2024-06-09 00:19:57 2024-06-09 01:16:09 0:56:12 0:45:06 0:11:06 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
fail 7745375 2024-06-07 21:10:29 2024-06-09 00:21:08 2024-06-09 01:25:32 1:04:24 0:54:20 0:10:04 smithi main ubuntu 22.04 orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
Failure Reason:

"2024-06-09T00:47:15.725849+0000 mon.a (mon.0) 329 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log

fail 7745376 2024-06-07 21:10:30 2024-06-09 00:21:58 2024-06-09 00:49:41 0:27:43 0:16:33 0:11:10 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-06-09T00:47:31.903209+0000 mon.a (mon.0) 550 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

dead 7745377 2024-06-07 21:10:31 2024-06-09 00:21:59 2024-06-09 00:24:03 0:02:04 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi043

fail 7745378 2024-06-07 21:10:32 2024-06-09 00:22:59 2024-06-09 01:07:31 0:44:32 0:26:29 0:18:03 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-06-09T00:48:29.706818+0000 mon.a (mon.0) 204 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7745379 2024-06-07 21:10:33 2024-06-09 00:26:10 2024-06-09 00:43:54 0:17:44 0:08:15 0:09:29 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/calico rook/master} 1
Failure Reason:

Command failed on smithi138 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7745380 2024-06-07 21:10:34 2024-06-09 00:26:20 2024-06-09 00:54:35 0:28:15 0:15:57 0:12:18 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} 2
fail 7745381 2024-06-07 21:10:35 2024-06-09 00:27:41 2024-06-09 00:56:36 0:28:55 0:17:22 0:11:33 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-09T00:51:18.530056+0000 mon.smithi132 (mon.0) 864 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7745382 2024-06-07 21:10:36 2024-06-09 00:27:51 2024-06-09 00:54:16 0:26:25 0:12:01 0:14:24 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} 1
pass 7745383 2024-06-07 21:10:37 2024-06-09 00:29:32 2024-06-09 00:54:07 0:24:35 0:12:14 0:12:21 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
fail 7745384 2024-06-07 21:10:38 2024-06-09 00:31:02 2024-06-09 01:41:24 1:10:22 1:00:12 0:10:10 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} 2
Failure Reason:

"2024-06-09T01:05:22.618592+0000 mon.a (mon.0) 987 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745385 2024-06-07 21:10:39 2024-06-09 00:31:43 2024-06-09 01:01:06 0:29:23 0:19:27 0:09:56 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_cephadm} 1
pass 7745386 2024-06-07 21:10:40 2024-06-09 00:31:43 2024-06-09 01:00:49 0:29:06 0:18:24 0:10:42 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7745387 2024-06-07 21:10:41 2024-06-09 00:31:43 2024-06-09 01:01:18 0:29:35 0:18:49 0:10:46 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7745388 2024-06-07 21:10:42 2024-06-09 00:31:44 2024-06-09 01:03:30 0:31:46 0:20:32 0:11:14 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-09T00:58:19.595536+0000 mon.smithi150 (mon.0) 830 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

fail 7745389 2024-06-07 21:10:43 2024-06-09 00:31:44 2024-06-09 00:57:17 0:25:33 0:15:14 0:10:19 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-06-09T00:50:39.458466+0000 mon.a (mon.0) 526 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7745390 2024-06-07 21:10:44 2024-06-09 00:32:25 2024-06-09 00:50:17 0:17:52 0:07:25 0:10:27 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_cephadm_repos} 1
fail 7745391 2024-06-07 21:10:45 2024-06-09 00:33:25 2024-06-09 00:58:43 0:25:18 0:14:18 0:11:00 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-06-09T00:54:59.961995+0000 mon.smithi016 (mon.0) 787 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

fail 7745392 2024-06-07 21:10:46 2024-06-09 00:33:25 2024-06-09 01:33:08 0:59:43 0:46:01 0:13:42 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi143 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b08deafc-25fa-11ef-bc9b-c7b262605968 -e sha1=18001b18ccec6ef0e131e4537cb87f419f579be0 -- bash -c \'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk \'"\'"\'{print $2}\'"\'"\')\''

pass 7745393 2024-06-07 21:10:46 2024-06-09 00:33:26 2024-06-09 01:33:09 0:59:43 0:45:26 0:14:17 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
fail 7745394 2024-06-07 21:10:47 2024-06-09 00:33:26 2024-06-09 01:08:37 0:35:11 0:24:39 0:10:32 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-06-09T00:58:01.446928+0000 mgr.smithi162.scxuqi (mgr.14244) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

pass 7745395 2024-06-07 21:10:48 2024-06-09 00:33:26 2024-06-09 01:13:32 0:40:06 0:30:09 0:09:57 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} 5
pass 7745396 2024-06-07 21:10:49 2024-06-09 00:34:27 2024-06-09 00:56:20 0:21:53 0:11:49 0:10:04 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} 2
pass 7745397 2024-06-07 21:10:50 2024-06-09 00:34:58 2024-06-09 01:00:37 0:25:39 0:15:21 0:10:18 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
fail 7745398 2024-06-07 21:10:51 2024-06-09 00:35:08 2024-06-09 01:02:22 0:27:14 0:14:58 0:12:16 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-06-09T00:59:41.155775+0000 mon.smithi027 (mon.0) 815 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

dead 7745399 2024-06-07 21:10:52 2024-06-09 00:36:49 2024-06-09 00:39:03 0:02:14 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/classic task/test_extra_daemon_features} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi114

pass 7745400 2024-06-07 21:10:53 2024-06-09 00:37:59 2024-06-09 01:10:51 0:32:52 0:19:54 0:12:58 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
pass 7745401 2024-06-07 21:10:54 2024-06-09 00:40:40 2024-06-09 01:13:35 0:32:55 0:23:49 0:09:06 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} 2
dead 7745402 2024-06-07 21:10:55 2024-06-09 00:41:10 2024-06-09 12:51:27 12:10:17 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

fail 7745403 2024-06-07 21:10:56 2024-06-09 00:41:41 2024-06-09 01:05:27 0:23:46 0:13:13 0:10:33 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-06-09T01:00:48.719124+0000 mon.a (mon.0) 424 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7745404 2024-06-07 21:10:57 2024-06-09 00:42:01 2024-06-09 01:21:29 0:39:28 0:24:19 0:15:09 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/connectivity task/test_host_drain} 3
Failure Reason:

"2024-06-09T01:18:56.155019+0000 mon.a (mon.0) 758 : cluster [WRN] Health check failed: 1 stray host(s) with 1 daemon(s) not managed by cephadm (CEPHADM_STRAY_HOST)" in cluster log

pass 7745405 2024-06-07 21:10:58 2024-06-09 00:44:02 2024-06-09 01:17:03 0:33:01 0:20:02 0:12:59 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
dead 7745406 2024-06-07 21:10:59 2024-06-09 00:46:23 2024-06-09 00:48:17 0:01:54 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi086

pass 7745407 2024-06-07 21:10:59 2024-06-09 00:47:13 2024-06-09 01:09:26 0:22:13 0:12:16 0:09:57 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} 1
pass 7745408 2024-06-07 21:11:00 2024-06-09 00:47:53 2024-06-09 01:19:13 0:31:20 0:19:39 0:11:41 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7745409 2024-06-07 21:11:01 2024-06-09 00:47:54 2024-06-09 01:18:26 0:30:32 0:19:49 0:10:43 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} 2
pass 7745410 2024-06-07 21:11:02 2024-06-09 00:48:54 2024-06-09 01:24:01 0:35:07 0:22:01 0:13:06 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
pass 7745411 2024-06-07 21:11:03 2024-06-09 00:48:55 2024-06-09 01:19:10 0:30:15 0:19:09 0:11:06 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
pass 7745412 2024-06-07 21:11:04 2024-06-09 00:48:55 2024-06-09 01:50:09 1:01:14 0:51:26 0:09:48 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} 2
pass 7745413 2024-06-07 21:11:05 2024-06-09 00:49:05 2024-06-09 01:18:36 0:29:31 0:16:36 0:12:55 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
fail 7745414 2024-06-07 21:11:06 2024-06-09 00:49:36 2024-06-09 01:27:12 0:37:36 0:25:19 0:12:17 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
Failure Reason:

"2024-06-09T01:22:31.850240+0000 mon.a (mon.0) 295 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7745415 2024-06-07 21:11:07 2024-06-09 00:50:26 2024-06-09 01:23:10 0:32:44 0:19:31 0:13:13 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
fail 7745416 2024-06-07 21:11:08 2024-06-09 00:51:07 2024-06-09 01:33:11 0:42:04 0:29:59 0:12:05 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-06-09T01:22:07.217200+0000 mon.a (mon.0) 1031 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7745417 2024-06-07 21:11:09 2024-06-09 00:54:07 2024-06-09 01:37:13 0:43:06 0:32:36 0:10:30 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} 2
fail 7745418 2024-06-07 21:11:10 2024-06-09 00:54:18 2024-06-09 01:26:17 0:31:59 0:21:58 0:10:01 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-06-09T01:20:59.739518+0000 mon.smithi119 (mon.0) 791 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7745419 2024-06-07 21:11:11 2024-06-09 00:54:38 2024-06-09 01:37:55 0:43:17 0:23:28 0:19:49 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_monitoring_stack_basic} 3
pass 7745420 2024-06-07 21:11:12 2024-06-09 00:54:59 2024-06-09 01:26:19 0:31:20 0:19:22 0:11:58 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
fail 7745421 2024-06-07 21:11:12 2024-06-09 00:56:29 2024-06-09 01:33:36 0:37:07 0:21:51 0:15:16 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1717896206.6750019 mgr.smithi170.xdnjqq (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7745422 2024-06-07 21:11:13 2024-06-09 00:56:40 2024-06-09 01:29:34 0:32:54 0:17:51 0:15:03 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} 1
Failure Reason:

"2024-06-09T01:27:02.579440+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7745423 2024-06-07 21:11:14 2024-06-09 00:56:40 2024-06-09 00:58:24 0:01:44 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi167

fail 7745424 2024-06-07 21:11:15 2024-06-09 00:57:20 2024-06-09 01:18:57 0:21:37 0:07:40 0:13:57 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/flannel rook/1.7.2} 3
Failure Reason:

Command failed on smithi016 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7745425 2024-06-07 21:11:16 2024-06-09 00:59:11 2024-06-09 01:26:06 0:26:55 0:11:16 0:15:39 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} 2
dead 7745426 2024-06-07 21:11:17 2024-06-09 01:00:42 2024-06-09 01:01:56 0:01:14 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi063

dead 7745427 2024-06-07 21:11:18 2024-06-09 01:00:52 2024-06-09 01:01:56 0:01:04 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
Failure Reason:

Error reimaging machines: Failed to power on smithi169

dead 7745428 2024-06-07 21:11:19 2024-06-09 01:00:52 2024-06-09 01:02:26 0:01:34 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi050

pass 7745429 2024-06-07 21:11:20 2024-06-09 01:01:23 2024-06-09 02:07:59 1:06:36 0:51:33 0:15:03 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} 2
pass 7745430 2024-06-07 21:11:21 2024-06-09 01:02:23 2024-06-09 01:37:29 0:35:06 0:16:22 0:18:44 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
fail 7745431 2024-06-07 21:11:22 2024-06-09 01:02:34 2024-06-09 01:36:15 0:33:41 0:21:19 0:12:22 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_rgw_multisite} 3
Failure Reason:

"2024-06-09T01:30:08.123934+0000 mon.a (mon.0) 531 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7745432 2024-06-07 21:11:23 2024-06-09 01:04:35 2024-06-09 01:40:54 0:36:19 0:25:42 0:10:37 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-06-09T01:27:30.048404+0000 mon.a (mon.0) 463 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7745433 2024-06-07 21:11:24 2024-06-09 01:04:35 2024-06-09 01:36:11 0:31:36 0:21:59 0:09:37 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-06-09T01:32:47.557910+0000 mon.smithi050 (mon.0) 935 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745434 2024-06-07 21:11:25 2024-06-09 01:04:35 2024-06-09 01:33:41 0:29:06 0:13:44 0:15:22 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7745435 2024-06-07 21:11:25 2024-06-09 01:04:36 2024-06-09 01:30:17 0:25:41 0:15:28 0:10:13 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
fail 7745436 2024-06-07 21:11:26 2024-06-09 01:05:06 2024-06-09 02:01:06 0:56:00 0:45:27 0:10:33 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-06-09T01:39:42.069946+0000 mon.a (mon.0) 1051 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7745437 2024-06-07 21:11:27 2024-06-09 01:05:37 2024-06-09 01:39:59 0:34:22 0:17:47 0:16:35 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_set_mon_crush_locations} 3
fail 7745438 2024-06-07 21:11:28 2024-06-09 01:06:47 2024-06-09 01:44:37 0:37:50 0:21:52 0:15:58 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-06-09T01:33:24.887956+0000 mgr.smithi121.dvrtmx (mgr.14238) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7745439 2024-06-07 21:11:29 2024-06-09 01:07:38 2024-06-09 01:38:54 0:31:16 0:17:02 0:14:14 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-06-09T01:36:55.608456+0000 mon.a (mon.0) 548 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7745440 2024-06-07 21:11:30 2024-06-09 01:07:38 2024-06-09 01:35:56 0:28:18 0:14:52 0:13:26 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} 2
fail 7745441 2024-06-07 21:11:31 2024-06-09 01:08:39 2024-06-09 01:37:53 0:29:14 0:17:12 0:12:02 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-09T01:32:28.360470+0000 mon.smithi105 (mon.0) 881 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7745442 2024-06-07 21:11:32 2024-06-09 01:08:59 2024-06-09 02:27:16 1:18:17 1:04:58 0:13:19 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
pass 7745443 2024-06-07 21:11:33 2024-06-09 01:09:29 2024-06-09 01:43:18 0:33:49 0:23:33 0:10:16 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
fail 7745444 2024-06-07 21:11:34 2024-06-09 01:09:30 2024-06-09 01:48:35 0:39:05 0:27:36 0:11:29 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-06-09T01:36:51.961279+0000 mon.a (mon.0) 1027 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7745445 2024-06-07 21:11:35 2024-06-09 01:11:00 2024-06-09 01:34:56 0:23:56 0:13:17 0:10:39 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_ca_signed_key} 2
pass 7745446 2024-06-07 21:11:35 2024-06-09 01:11:51 2024-06-09 01:38:17 0:26:26 0:14:15 0:12:11 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7745447 2024-06-07 21:11:36 2024-06-09 01:13:41 2024-06-09 01:38:42 0:25:01 0:15:09 0:09:52 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-09T01:34:08.158900+0000 mon.smithi002 (mon.0) 851 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7745448 2024-06-07 21:11:37 2024-06-09 01:13:42 2024-06-09 01:36:47 0:23:05 0:12:53 0:10:12 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
fail 7745449 2024-06-07 21:11:38 2024-06-09 01:13:42 2024-06-09 01:38:39 0:24:57 0:14:58 0:09:59 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-06-09T01:33:41.248793+0000 mon.smithi131 (mon.0) 777 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7745450 2024-06-07 21:11:39 2024-06-09 01:13:43 2024-06-09 01:42:54 0:29:11 0:18:13 0:10:58 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm} 1
pass 7745451 2024-06-07 21:11:40 2024-06-09 01:15:53 2024-06-09 01:40:59 0:25:06 0:15:12 0:09:54 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
pass 7745452 2024-06-07 21:11:41 2024-06-09 01:16:14 2024-06-09 01:51:50 0:35:36 0:25:34 0:10:02 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} 2
fail 7745453 2024-06-07 21:11:42 2024-06-09 01:17:04 2024-06-09 01:51:19 0:34:15 0:24:18 0:09:57 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1717897267.3848078 mgr.smithi120.evncdt (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log