Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7739549 2024-06-03 14:55:03 2024-06-03 15:03:49 2024-06-03 16:31:39 1:27:50 0:57:04 0:30:46 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} 2
fail 7739550 2024-06-03 14:55:05 2024-06-03 15:03:49 2024-06-03 16:53:07 1:49:18 1:16:14 0:33:04 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} 3
Failure Reason:

"2024-06-03T16:46:30.595808+0000 mon.a (mon.0) 490 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7739551 2024-06-03 14:55:06 2024-06-03 15:10:00 2024-06-03 16:39:14 1:29:14 1:00:03 0:29:11 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

ceph version 17.2.7-984.g79d5a009 was not installed, found 17.2.7-980.g658e3c70.el9.

pass 7739552 2024-06-03 14:55:07 2024-06-03 15:10:11 2024-06-03 16:34:11 1:24:00 0:54:28 0:29:32 smithi main centos 9.stream orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
fail 7739553 2024-06-03 14:55:08 2024-06-03 15:11:01 2024-06-03 16:14:39 1:03:38 0:53:33 0:10:05 smithi main ubuntu 22.04 orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
Failure Reason:

"2024-06-03T15:36:13.056902+0000 mon.a (mon.0) 332 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log

fail 7739554 2024-06-03 14:55:10 2024-06-03 15:11:02 2024-06-03 16:49:07 1:38:05 1:09:42 0:28:23 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-06-03T16:46:53.181692+0000 mon.a (mon.0) 553 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7739555 2024-06-03 14:55:11 2024-06-03 15:11:52 2024-06-03 16:46:09 1:34:17 1:02:11 0:32:06 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli} 2
fail 7739556 2024-06-03 14:55:12 2024-06-03 15:13:23 2024-06-03 17:14:49 2:01:26 1:31:05 0:30:21 smithi main centos 9.stream orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-06-03T15:59:10.287741+0000 mon.a (mon.0) 205 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7739557 2024-06-03 14:55:14 2024-06-03 15:13:23 2024-06-03 16:50:18 1:36:55 1:00:40 0:36:15 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} 2
fail 7739558 2024-06-03 14:55:15 2024-06-03 15:18:54 2024-06-03 16:17:42 0:58:48 0:28:30 0:30:18 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-03T16:11:28.551599+0000 mon.smithi053 (mon.0) 860 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7739559 2024-06-03 14:55:16 2024-06-03 15:22:55 2024-06-03 15:51:54 0:28:59 0:18:13 0:10:46 smithi main ubuntu 22.04 orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} 1
pass 7739560 2024-06-03 14:55:17 2024-06-03 15:24:35 2024-06-03 16:26:39 1:02:04 0:24:22 0:37:42 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
pass 7739561 2024-06-03 14:55:19 2024-06-03 15:25:36 2024-06-03 18:00:16 2:34:40 1:59:01 0:35:39 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
fail 7739562 2024-06-03 14:55:20 2024-06-03 15:26:06 2024-06-03 17:02:21 1:36:15 1:00:35 0:35:40 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-06-03T16:41:11.182792+0000 mon.a (mon.0) 1046 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7739563 2024-06-03 14:55:21 2024-06-03 15:27:07 2024-06-03 17:18:12 1:51:05 1:19:27 0:31:38 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} 2
pass 7739564 2024-06-03 14:55:23 2024-06-03 15:27:07 2024-06-03 17:01:29 1:34:22 1:04:24 0:29:58 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
fail 7739565 2024-06-03 14:55:24 2024-06-03 15:30:38 2024-06-03 16:36:52 1:06:14 0:28:07 0:38:07 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-06-03T16:33:16.104675+0000 mon.smithi069 (mon.0) 1016 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7739566 2024-06-03 14:55:25 2024-06-03 15:37:49 2024-06-03 16:10:56 0:33:07 0:19:46 0:13:21 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7739567 2024-06-03 14:55:26 2024-06-03 15:40:30 2024-06-03 16:12:29 0:31:59 0:20:46 0:11:13 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-03T16:08:02.387972+0000 mon.smithi105 (mon.0) 859 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

fail 7739568 2024-06-03 14:55:28 2024-06-03 15:42:50 2024-06-03 16:23:57 0:41:07 0:25:01 0:16:06 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
Failure Reason:

"2024-06-03T16:10:10.865974+0000 mon.a (mon.0) 462 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7739569 2024-06-03 14:55:29 2024-06-03 15:48:21 2024-06-03 17:52:57 2:04:36 1:30:41 0:33:55 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
pass 7739570 2024-06-03 14:55:30 2024-06-03 15:52:02 2024-06-03 16:47:55 0:55:53 0:45:16 0:10:37 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} 2
fail 7739571 2024-06-03 14:55:31 2024-06-03 15:52:13 2024-06-03 16:33:24 0:41:11 0:27:14 0:13:57 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
Failure Reason:

"2024-06-03T16:21:19.659264+0000 mon.a (mon.0) 462 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7739572 2024-06-03 14:55:33 2024-06-03 15:55:43 2024-06-03 18:18:10 2:22:27 1:48:31 0:33:56 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7739573 2024-06-03 14:55:34 2024-06-03 15:59:24 2024-06-03 16:54:08 0:54:44 0:23:41 0:31:03 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-06-03T16:50:49.528791+0000 mon.smithi038 (mon.0) 777 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7739574 2024-06-03 14:55:35 2024-06-03 16:01:15 2024-06-03 18:08:26 2:07:11 1:30:49 0:36:22 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} 5
pass 7739575 2024-06-03 14:55:36 2024-06-03 16:11:07 2024-06-03 17:08:38 0:57:31 0:28:21 0:29:10 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} 2
pass 7739576 2024-06-03 14:55:38 2024-06-03 16:11:17 2024-06-03 17:11:35 1:00:18 0:27:48 0:32:30 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7739577 2024-06-03 14:55:39 2024-06-03 16:12:38 2024-06-03 17:43:40 1:31:02 0:56:33 0:34:29 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} 2
pass 7739578 2024-06-03 14:55:40 2024-06-03 16:14:38 2024-06-03 17:55:40 1:41:02 1:11:51 0:29:11 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} 3
fail 7739579 2024-06-03 14:55:42 2024-06-03 16:14:49 2024-06-03 17:10:41 0:55:52 0:23:28 0:32:24 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-06-03T17:06:12.668899+0000 mon.smithi053 (mon.0) 811 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7739580 2024-06-03 14:55:43 2024-06-03 16:17:49 2024-06-03 16:56:31 0:38:42 0:20:45 0:17:57 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
dead 7739581 2024-06-03 14:55:44 2024-06-03 16:23:50 2024-06-03 16:25:04 0:01:14 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi064

dead 7739582 2024-06-03 14:55:45 2024-06-03 16:24:01 2024-06-03 16:27:45 0:03:44 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi046

dead 7739583 2024-06-03 14:55:47 2024-06-03 16:26:42 2024-06-04 04:36:05 12:09:23 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

fail 7739584 2024-06-03 14:55:48 2024-06-03 16:26:42 2024-06-03 17:29:03 1:02:21 0:28:08 0:34:13 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-06-03T17:23:24.172661+0000 mon.a (mon.0) 420 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7739585 2024-06-03 14:55:49 2024-06-03 16:30:03 2024-06-03 17:24:21 0:54:18 0:42:53 0:11:25 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

"2024-06-03T16:51:37.105714+0000 mon.a (mon.0) 391 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7739586 2024-06-03 14:55:50 2024-06-03 16:30:03 2024-06-03 18:27:21 1:57:18 1:25:19 0:31:59 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} 2
fail 7739587 2024-06-03 14:55:51 2024-06-03 16:31:44 2024-06-03 18:19:40 1:47:56 1:10:58 0:36:58 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
Failure Reason:

"2024-06-03T18:12:18.800313+0000 mon.a (mon.0) 398 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7739588 2024-06-03 14:55:53 2024-06-03 16:32:24 2024-06-03 17:35:24 1:03:00 0:28:56 0:34:04 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
pass 7739589 2024-06-03 14:55:54 2024-06-03 16:32:44 2024-06-03 16:56:48 0:24:04 0:13:03 0:11:01 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} 1
pass 7739590 2024-06-03 14:55:55 2024-06-03 16:32:45 2024-06-03 17:30:36 0:57:51 0:25:40 0:32:11 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7739591 2024-06-03 14:55:57 2024-06-03 16:33:25 2024-06-03 17:03:41 0:30:16 0:19:21 0:10:55 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} 2
fail 7739592 2024-06-03 14:55:58 2024-06-03 16:34:16 2024-06-03 17:39:51 1:05:35 0:25:41 0:39:54 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-06-03T17:31:38.550416+0000 mon.a (mon.0) 410 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7739593 2024-06-03 14:55:59 2024-06-03 16:36:56 2024-06-03 17:08:56 0:32:00 0:20:42 0:11:18 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
pass 7739594 2024-06-03 14:56:00 2024-06-03 16:39:17 2024-06-03 18:01:20 1:22:03 1:06:38 0:15:25 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
pass 7739595 2024-06-03 14:56:02 2024-06-03 16:43:28 2024-06-03 19:06:38 2:23:10 1:50:28 0:32:42 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} 2
pass 7739596 2024-06-03 14:56:03 2024-06-03 16:43:28 2024-06-03 18:24:24 1:40:56 1:04:32 0:36:24 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} 2
fail 7739597 2024-06-03 14:56:04 2024-06-03 16:45:29 2024-06-03 19:21:03 2:35:34 2:00:09 0:35:25 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7739598 2024-06-03 14:56:05 2024-06-03 16:45:39 2024-06-03 17:49:12 1:03:33 0:24:24 0:39:09 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
fail 7739599 2024-06-03 14:56:07 2024-06-03 16:48:00 2024-06-03 17:33:06 0:45:06 0:32:20 0:12:46 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} 2
Failure Reason:

"2024-06-03T17:29:11.481390+0000 mon.a (mon.0) 1395 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7739600 2024-06-03 14:56:08 2024-06-03 16:49:10 2024-06-03 17:22:20 0:33:10 0:20:46 0:12:24 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} 1
pass 7739601 2024-06-03 14:56:09 2024-06-03 16:50:21 2024-06-03 17:47:15 0:56:54 0:25:47 0:31:07 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
pass 7739602 2024-06-03 14:56:11 2024-06-03 16:50:21 2024-06-03 19:15:31 2:25:10 1:53:31 0:31:39 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
fail 7739603 2024-06-03 14:56:12 2024-06-03 16:50:21 2024-06-03 17:24:39 0:34:18 0:21:00 0:13:18 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-06-03T17:20:18.502200+0000 mon.smithi052 (mon.0) 809 : cluster [WRN] Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7739604 2024-06-03 14:56:13 2024-06-03 16:53:12 2024-06-03 17:23:46 0:30:34 0:19:59 0:10:35 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
pass 7739605 2024-06-03 14:56:14 2024-06-03 16:54:13 2024-06-03 18:42:04 1:47:51 1:12:46 0:35:05 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} 2
pass 7739606 2024-06-03 14:56:16 2024-06-03 16:56:33 2024-06-03 17:47:20 0:50:47 0:20:21 0:30:26 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
pass 7739607 2024-06-03 14:56:17 2024-06-03 16:56:34 2024-06-03 18:52:45 1:56:11 1:19:48 0:36:23 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7739608 2024-06-03 14:56:18 2024-06-03 17:01:04 2024-06-03 18:38:14 1:37:10 1:06:57 0:30:13 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} 1
Failure Reason:

"2024-06-03T18:35:33.345024+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7739609 2024-06-03 14:56:19 2024-06-03 17:01:05 2024-06-03 18:42:47 1:41:42 1:09:59 0:31:43 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} 2
pass 7739610 2024-06-03 14:56:21 2024-06-03 17:03:45 2024-06-03 17:58:03 0:54:18 0:20:43 0:33:35 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} 2
pass 7739611 2024-06-03 14:56:22 2024-06-03 17:08:46 2024-06-03 18:00:07 0:51:21 0:22:05 0:29:16 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7739612 2024-06-03 14:56:23 2024-06-03 17:08:57 2024-06-03 17:55:54 0:46:57 0:18:41 0:28:16 smithi main centos 9.stream orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
fail 7739613 2024-06-03 14:56:25 2024-06-03 17:08:57 2024-06-03 18:08:32 0:59:35 0:24:18 0:35:17 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-06-03T18:03:41.141596+0000 mon.a (mon.0) 453 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7739614 2024-06-03 14:56:26 2024-06-03 17:11:38 2024-06-03 18:48:13 1:36:35 1:03:35 0:33:00 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

"2024-06-03T18:03:18.310724+0000 mon.a (mon.0) 550 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7739615 2024-06-03 14:56:27 2024-06-03 17:29:13 2024-06-03 18:26:46 0:57:33 0:28:15 0:29:18 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7739616 2024-06-03 14:56:29 2024-06-03 17:29:14 2024-06-03 18:33:47 1:04:33 0:26:50 0:37:43 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
pass 7739617 2024-06-03 14:56:30 2024-06-03 17:30:44 2024-06-03 19:30:40 1:59:56 1:24:18 0:35:38 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
pass 7739618 2024-06-03 14:56:31 2024-06-03 17:35:29 2024-06-03 19:59:14 2:23:45 1:45:03 0:38:42 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} 2
fail 7739619 2024-06-03 14:56:32 2024-06-03 17:39:59 2024-06-03 19:24:51 1:44:52 1:05:46 0:39:06 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} 2
Failure Reason:

"2024-06-03T19:22:52.105512+0000 mon.a (mon.0) 455 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7739620 2024-06-03 14:56:34 2024-06-03 17:43:50 2024-06-03 18:18:50 0:35:00 0:21:48 0:13:12 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-06-03T18:15:14.751413+0000 mon.smithi144 (mon.0) 954 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7739621 2024-06-03 14:56:35 2024-06-03 17:47:11 2024-06-03 18:44:24 0:57:13 0:22:28 0:34:45 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
fail 7739622 2024-06-03 14:56:36 2024-06-03 17:49:22 2024-06-03 19:43:59 1:54:37 1:20:35 0:34:02 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

dead 7739623 2024-06-03 14:56:38 2024-06-03 17:53:03 2024-06-03 17:56:47 0:03:44 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi142

dead 7739624 2024-06-03 14:56:39 2024-06-03 17:55:43 2024-06-03 17:56:47 0:01:04 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi087

fail 7739625 2024-06-03 14:56:40 2024-06-03 17:55:44 2024-06-03 18:34:09 0:38:25 0:23:27 0:14:58 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} 3
Failure Reason:

"2024-06-03T18:31:23.356173+0000 mon.a (mon.0) 747 : cluster [WRN] Health check failed: 1 stray host(s) with 1 daemon(s) not managed by cephadm (CEPHADM_STRAY_HOST)" in cluster log

fail 7739626 2024-06-03 14:56:41 2024-06-03 17:58:04 2024-06-03 19:39:08 1:41:04 1:11:01 0:30:03 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-06-03T19:37:46.729635+0000 mon.a (mon.0) 542 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7739627 2024-06-03 14:56:43 2024-06-03 18:00:15 2024-06-03 19:44:22 1:44:07 1:12:51 0:31:16 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} 2
fail 7739628 2024-06-03 14:56:44 2024-06-03 18:00:25 2024-06-03 18:58:41 0:58:16 0:27:55 0:30:21 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-03T18:52:22.862004+0000 mon.smithi183 (mon.0) 864 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7739629 2024-06-03 14:56:45 2024-06-03 18:01:26 2024-06-03 19:02:28 1:01:02 0:46:48 0:14:14 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} 2
pass 7739630 2024-06-03 14:56:47 2024-06-03 18:02:46 2024-06-03 19:55:35 1:52:49 1:20:23 0:32:26 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} 2
fail 7739631 2024-06-03 14:56:48 2024-06-03 18:02:47 2024-06-03 19:37:37 1:34:50 1:02:27 0:32:23 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
Failure Reason:

"2024-06-03T19:31:46.742790+0000 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7739632 2024-06-03 14:56:49 2024-06-03 18:02:47 2024-06-03 18:38:13 0:35:26 0:24:08 0:11:18 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7739633 2024-06-03 14:56:50 2024-06-03 18:04:37 2024-06-03 18:55:45 0:51:08 0:22:11 0:28:57 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7739634 2024-06-03 14:56:51 2024-06-03 18:04:58 2024-06-03 19:57:57 1:52:59 1:23:18 0:29:41 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7739635 2024-06-03 14:56:53 2024-06-03 18:04:58 2024-06-03 18:58:39 0:53:41 0:24:38 0:29:03 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-03T18:55:38.599967+0000 mon.smithi031 (mon.0) 853 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

dead 7739636 2024-06-03 14:56:54 2024-06-03 18:05:19 2024-06-03 18:07:53 0:02:34 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi002

pass 7739637 2024-06-03 14:56:55 2024-06-03 18:06:49 2024-06-03 18:54:46 0:47:57 0:37:01 0:10:56 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
fail 7739638 2024-06-03 14:56:56 2024-06-03 18:07:20 2024-06-03 18:45:28 0:38:08 0:26:01 0:12:07 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-06-03T18:30:19.108856+0000 mon.a (mon.0) 389 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7739639 2024-06-03 14:56:58 2024-06-03 18:07:40 2024-06-03 20:23:12 2:15:32 1:42:53 0:32:39 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} 2
fail 7739640 2024-06-03 14:56:59 2024-06-03 18:08:00 2024-06-03 20:17:59 2:09:59 1:49:15 0:20:44 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-06-03T19:42:31.460900+0000 mon.a (mon.0) 896 : cluster [WRN] Health check failed: 17 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7739641 2024-06-03 14:57:00 2024-06-03 18:08:31 2024-06-03 19:57:23 1:48:52 1:16:06 0:32:46 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} 3
fail 7739642 2024-06-03 14:57:02 2024-06-03 18:08:31 2024-06-03 19:03:35 0:55:04 0:25:12 0:29:52 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-06-03T18:56:08.206683+0000 mon.smithi105 (mon.0) 235 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7739643 2024-06-03 14:57:03 2024-06-03 18:08:42 2024-06-03 19:01:49 0:53:07 0:41:23 0:11:44 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} 5
pass 7739644 2024-06-03 14:57:04 2024-06-03 18:09:52 2024-06-03 18:40:33 0:30:41 0:19:11 0:11:30 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} 2
pass 7739645 2024-06-03 14:57:05 2024-06-03 18:13:03 2024-06-03 18:47:39 0:34:36 0:19:29 0:15:07 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
fail 7739646 2024-06-03 14:57:07 2024-06-03 18:18:04 2024-06-03 19:01:42 0:43:38 0:33:35 0:10:03 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} 2
Failure Reason:

"2024-06-03T18:58:27.993316+0000 mon.a (mon.0) 1386 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7739647 2024-06-03 14:57:08 2024-06-03 18:18:04 2024-06-03 18:49:32 0:31:28 0:19:53 0:11:35 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} 3
Failure Reason:

"2024-06-03T18:42:38.848044+0000 mon.a (mon.0) 435 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7739648 2024-06-03 14:57:09 2024-06-03 18:18:05 2024-06-03 18:49:49 0:31:44 0:21:13 0:10:31 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-06-03T18:45:00.924503+0000 mon.smithi043 (mon.0) 800 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7739649 2024-06-03 14:57:10 2024-06-03 18:18:05 2024-06-03 20:13:26 1:55:21 1:24:59 0:30:22 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

dead 7739650 2024-06-03 14:57:12 2024-06-03 18:18:05 2024-06-03 18:25:30 0:07:25 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi033

pass 7739651 2024-06-03 14:57:13 2024-06-03 18:24:26 2024-06-03 20:32:31 2:08:05 1:36:05 0:32:00 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
dead 7739652 2024-06-03 14:57:14 2024-06-03 18:26:47 2024-06-04 06:38:52 12:12:05 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

pass 7739653 2024-06-03 14:57:15 2024-06-03 18:27:28 2024-06-03 20:12:24 1:44:56 1:09:33 0:35:23 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
pass 7739654 2024-06-03 14:57:16 2024-06-03 18:33:29 2024-06-03 20:16:54 1:43:25 1:10:13 0:33:12 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} 3
pass 7739655 2024-06-03 14:57:18 2024-06-03 18:33:29 2024-06-03 19:07:38 0:34:09 0:23:23 0:10:46 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} 2