Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7759503 2024-06-17 12:06:04 2024-06-17 13:57:23 2024-06-17 14:28:08 0:30:45 0:22:02 0:08:43 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} 2
fail 7759504 2024-06-17 12:06:05 2024-06-17 13:57:23 2024-06-17 14:22:17 0:24:54 0:16:18 0:08:36 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} 3
Failure Reason:

"2024-06-17T14:17:57.961383+0000 mon.a (mon.0) 551 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7759505 2024-06-17 12:06:06 2024-06-17 13:57:24 2024-06-17 14:26:52 0:29:28 0:20:25 0:09:03 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1718633796.4879227 mgr.smithi097.wyoqel (mgr.14234) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

pass 7759506 2024-06-17 12:06:08 2024-06-17 13:57:54 2024-06-17 14:51:02 0:53:08 0:44:10 0:08:58 smithi main centos 9.stream orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
fail 7759507 2024-06-17 12:06:09 2024-06-17 13:58:04 2024-06-17 14:41:03 0:42:59 0:33:53 0:09:06 smithi main centos 9.stream orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignore_mgr_down ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} 1
Failure Reason:

"2024-06-17T14:21:16.597439+0000 mon.a (mon.0) 695 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7759508 2024-06-17 12:06:10 2024-06-17 13:58:05 2024-06-17 14:21:33 0:23:28 0:14:20 0:09:08 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-06-17T14:19:39.909177+0000 mon.a (mon.0) 554 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7759509 2024-06-17 12:06:11 2024-06-17 13:59:15 2024-06-17 14:22:47 0:23:32 0:11:53 0:11:39 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} 2
pass 7759510 2024-06-17 12:06:13 2024-06-17 14:01:46 2024-06-17 14:37:13 0:35:27 0:23:29 0:11:58 smithi main centos 9.stream orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
pass 7759511 2024-06-17 12:06:14 2024-06-17 14:03:57 2024-06-17 14:28:37 0:24:40 0:12:43 0:11:57 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} 2
fail 7759512 2024-06-17 12:06:15 2024-06-17 14:05:27 2024-06-17 14:30:18 0:24:51 0:16:34 0:08:17 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-17T14:25:10.903333+0000 mon.smithi120 (mon.0) 850 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7759513 2024-06-17 12:06:17 2024-06-17 14:05:27 2024-06-17 14:26:18 0:20:51 0:11:08 0:09:43 smithi main centos 9.stream orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} 1
pass 7759514 2024-06-17 12:06:18 2024-06-17 14:05:28 2024-06-17 14:28:30 0:23:02 0:11:09 0:11:53 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
pass 7759515 2024-06-17 12:06:19 2024-06-17 14:07:08 2024-06-17 15:27:48 1:20:40 1:06:57 0:13:43 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
fail 7759516 2024-06-17 12:06:20 2024-06-17 14:10:49 2024-06-17 15:10:43 0:59:54 0:48:03 0:11:51 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-06-17T14:51:09.182854+0000 mon.a (mon.0) 1081 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7759517 2024-06-17 12:06:22 2024-06-17 14:12:30 2024-06-17 14:46:02 0:33:32 0:23:14 0:10:18 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} 2
pass 7759518 2024-06-17 12:06:23 2024-06-17 14:13:10 2024-06-17 14:36:18 0:23:08 0:14:04 0:09:04 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
fail 7759519 2024-06-17 12:06:24 2024-06-17 14:13:11 2024-06-17 14:38:17 0:25:06 0:16:27 0:08:39 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-06-17T14:35:27.330189+0000 mon.smithi152 (mon.0) 1011 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7759520 2024-06-17 12:06:25 2024-06-17 14:13:11 2024-06-17 14:44:44 0:31:33 0:19:51 0:11:42 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7759521 2024-06-17 12:06:27 2024-06-17 14:14:12 2024-06-17 14:46:23 0:32:11 0:21:49 0:10:22 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-17T14:41:03.296022+0000 mon.smithi105 (mon.0) 866 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

fail 7759522 2024-06-17 12:06:28 2024-06-17 14:14:42 2024-06-17 14:50:15 0:35:33 0:26:07 0:09:26 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
Failure Reason:

"2024-06-17T14:35:14.828050+0000 mon.a (mon.0) 382 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7759523 2024-06-17 12:06:29 2024-06-17 14:14:43 2024-06-17 14:55:44 0:41:01 0:31:17 0:09:44 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
fail 7759524 2024-06-17 12:06:30 2024-06-17 14:14:43 2024-06-17 15:10:29 0:55:46 0:45:30 0:10:16 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-06-17T14:49:27.526376+0000 mon.a (mon.0) 999 : cluster [WRN] Health check failed: 17 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7759525 2024-06-17 12:06:32 2024-06-17 14:14:53 2024-06-17 14:51:15 0:36:22 0:25:54 0:10:28 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
fail 7759526 2024-06-17 12:06:33 2024-06-17 14:15:24 2024-06-17 15:37:36 1:22:12 1:07:29 0:14:43 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7759527 2024-06-17 12:06:34 2024-06-17 14:17:34 2024-06-17 14:40:54 0:23:20 0:13:30 0:09:50 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-06-17T14:37:49.291413+0000 mon.smithi003 (mon.0) 774 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7759528 2024-06-17 12:06:35 2024-06-17 14:17:45 2024-06-17 14:57:01 0:39:16 0:25:02 0:14:14 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} 5
pass 7759529 2024-06-17 12:06:37 2024-06-17 14:21:36 2024-06-17 14:43:17 0:21:41 0:11:05 0:10:36 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} 2
pass 7759530 2024-06-17 12:06:38 2024-06-17 14:22:26 2024-06-17 14:45:59 0:23:33 0:13:46 0:09:47 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7759531 2024-06-17 12:06:39 2024-06-17 14:22:57 2024-06-17 14:55:09 0:32:12 0:22:02 0:10:10 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_python} 2
pass 7759532 2024-06-17 12:06:40 2024-06-17 14:24:07 2024-06-17 14:49:43 0:25:36 0:12:57 0:12:39 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} 3
fail 7759533 2024-06-17 12:06:42 2024-06-17 14:26:48 2024-06-17 14:52:00 0:25:12 0:14:26 0:10:46 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-06-17T14:47:20.899089+0000 mon.smithi002 (mon.0) 813 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7759534 2024-06-17 12:06:43 2024-06-17 14:26:58 2024-06-17 14:56:35 0:29:37 0:20:34 0:09:03 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
pass 7759535 2024-06-17 12:06:44 2024-06-17 14:26:59 2024-06-17 15:27:48 1:00:49 0:49:34 0:11:15 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
fail 7759536 2024-06-17 12:06:46 2024-06-17 14:28:09 2024-06-17 14:57:54 0:29:45 0:20:02 0:09:43 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1718635659.5136385 mgr.smithi167.ilgsbl (mgr.14232) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

dead 7759537 2024-06-17 12:06:47 2024-06-17 14:28:40 2024-06-18 02:37:58 12:09:18 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

dead 7759538 2024-06-17 12:06:48 2024-06-17 14:28:40 2024-06-17 14:31:24 0:02:44 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi120

fail 7759539 2024-06-17 12:06:49 2024-06-17 14:30:21 2024-06-17 15:29:53 0:59:32 0:44:07 0:15:25 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

"2024-06-17T14:56:01.178745+0000 mon.a (mon.0) 341 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7759540 2024-06-17 12:06:51 2024-06-17 14:36:12 2024-06-17 15:09:34 0:33:22 0:24:10 0:09:12 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} 2
fail 7759541 2024-06-17 12:06:52 2024-06-17 14:36:12 2024-06-17 15:02:30 0:26:18 0:15:51 0:10:27 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
Failure Reason:

"2024-06-17T14:56:31.062341+0000 mon.a (mon.0) 383 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7759542 2024-06-17 12:06:53 2024-06-17 14:37:23 2024-06-17 15:03:33 0:26:10 0:15:24 0:10:46 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
pass 7759543 2024-06-17 12:06:54 2024-06-17 14:38:23 2024-06-17 14:59:58 0:21:35 0:11:59 0:09:36 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} 1
pass 7759544 2024-06-17 12:06:56 2024-06-17 14:38:24 2024-06-17 15:04:20 0:25:56 0:12:46 0:13:10 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7759545 2024-06-17 12:06:57 2024-06-17 14:40:54 2024-06-17 15:11:04 0:30:10 0:19:03 0:11:07 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} 2
fail 7759546 2024-06-17 12:06:58 2024-06-17 14:43:25 2024-06-17 15:10:33 0:27:08 0:15:18 0:11:50 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-06-17T15:03:23.549383+0000 mon.a (mon.0) 416 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7759547 2024-06-17 12:06:59 2024-06-17 14:44:45 2024-06-17 15:15:32 0:30:47 0:19:05 0:11:42 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
fail 7759548 2024-06-17 12:07:01 2024-06-17 14:46:06 2024-06-17 15:59:47 1:13:41 1:04:38 0:09:03 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-06-17T15:20:35.394009+0000 mon.a (mon.0) 1068 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7759549 2024-06-17 12:07:02 2024-06-17 14:46:06 2024-06-17 15:47:27 1:01:21 0:51:46 0:09:35 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

"2024-06-17T15:10:49.215213+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7759550 2024-06-17 12:07:03 2024-06-17 14:46:27 2024-06-17 15:10:42 0:24:15 0:11:22 0:12:53 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} 2
fail 7759551 2024-06-17 12:07:04 2024-06-17 14:49:48 2024-06-17 15:52:47 1:02:59 0:52:52 0:10:07 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7759552 2024-06-17 12:07:06 2024-06-17 14:49:48 2024-06-17 15:13:09 0:23:21 0:13:39 0:09:42 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
pass 7759553 2024-06-17 12:07:07 2024-06-17 14:50:18 2024-06-17 15:32:47 0:42:29 0:32:18 0:10:11 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} 2
pass 7759554 2024-06-17 12:07:08 2024-06-17 14:51:09 2024-06-17 15:20:44 0:29:35 0:19:29 0:10:06 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} 1
pass 7759555 2024-06-17 12:07:10 2024-06-17 14:51:19 2024-06-17 15:14:29 0:23:10 0:14:01 0:09:09 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
pass 7759556 2024-06-17 12:07:11 2024-06-17 14:51:20 2024-06-17 15:50:34 0:59:14 0:49:14 0:10:00 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
dead 7759557 2024-06-17 12:07:12 2024-06-17 14:51:40 2024-06-17 14:56:14 0:04:34 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi046

pass 7759558 2024-06-17 12:07:13 2024-06-17 14:55:11 2024-06-17 15:25:28 0:30:17 0:19:59 0:10:18 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
pass 7759559 2024-06-17 12:07:15 2024-06-17 14:55:51 2024-06-17 15:29:47 0:33:56 0:22:59 0:10:57 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} 2
pass 7759560 2024-06-17 12:07:16 2024-06-17 14:56:42 2024-06-17 15:11:26 0:14:44 0:06:08 0:08:36 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
fail 7759561 2024-06-17 12:07:17 2024-06-17 14:57:02 2024-06-17 15:28:26 0:31:24 0:20:19 0:11:05 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1718637394.793555 mgr.smithi153.dwxkbv (mgr.14234) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7759562 2024-06-17 12:07:19 2024-06-17 14:57:02 2024-06-17 15:20:32 0:23:30 0:13:20 0:10:10 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} 1
Failure Reason:

"2024-06-17T15:18:25.577291+0000 mon.a (mon.0) 212 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7759563 2024-06-17 12:07:20 2024-06-17 14:57:03 2024-06-17 15:19:09 0:22:06 0:11:55 0:10:11 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli} 2
pass 7759564 2024-06-17 12:07:21 2024-06-17 14:58:03 2024-06-17 15:20:57 0:22:54 0:11:12 0:11:42 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} 2
pass 7759565 2024-06-17 12:07:22 2024-06-17 15:00:04 2024-06-17 15:25:49 0:25:45 0:13:48 0:11:57 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7759566 2024-06-17 12:07:24 2024-06-17 15:03:22 2024-06-17 15:35:19 0:31:57 0:20:49 0:11:08 smithi main ubuntu 22.04 orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} 1
fail 7759567 2024-06-17 12:07:25 2024-06-17 15:03:42 2024-06-17 15:27:42 0:24:00 0:12:13 0:11:47 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-06-17T15:24:30.978050+0000 mon.a (mon.0) 788 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7759568 2024-06-17 12:07:26 2024-06-17 15:07:02 2024-06-17 16:05:03 0:58:01 0:44:43 0:13:18 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7ca98a24-2cbd-11ef-bc9d-c7b262605968 -e sha1=56d3febd0f9fda48a6103c1c961e616c859932b9 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''

pass 7759569 2024-06-17 12:07:27 2024-06-17 15:09:43 2024-06-17 15:35:29 0:25:46 0:14:59 0:10:47 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7759570 2024-06-17 12:07:29 2024-06-17 15:10:33 2024-06-17 15:36:22 0:25:49 0:14:41 0:11:08 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
pass 7759571 2024-06-17 12:07:30 2024-06-17 15:10:34 2024-06-17 15:54:33 0:43:59 0:32:20 0:11:39 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
dead 7759572 2024-06-17 12:07:31 2024-06-17 15:10:44 2024-06-17 15:11:48 0:01:04 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_api_tests} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi045

fail 7759573 2024-06-17 12:07:33 2024-06-17 15:10:44 2024-06-17 15:34:17 0:23:33 0:13:11 0:10:22 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} 2
Failure Reason:

"2024-06-17T15:31:27.082664+0000 mon.a (mon.0) 40 : cluster [WRN] [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)" in cluster log

fail 7759574 2024-06-17 12:07:34 2024-06-17 15:11:15 2024-06-17 15:44:43 0:33:28 0:21:59 0:11:29 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-06-17T15:40:55.494088+0000 mon.smithi033 (mon.0) 970 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7759575 2024-06-17 12:07:35 2024-06-17 15:13:15 2024-06-17 15:37:27 0:24:12 0:13:18 0:10:54 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
fail 7759576 2024-06-17 12:07:36 2024-06-17 15:14:36 2024-06-17 16:04:20 0:49:44 0:39:03 0:10:41 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7759577 2024-06-17 12:07:38 2024-06-17 15:15:36 2024-06-17 15:42:23 0:26:47 0:14:46 0:12:01 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
pass 7759578 2024-06-17 12:07:39 2024-06-17 15:19:17 2024-06-17 16:02:38 0:43:21 0:32:01 0:11:20 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_python} 2
fail 7759579 2024-06-17 12:07:40 2024-06-17 15:20:38 2024-06-17 15:54:44 0:34:06 0:23:54 0:10:12 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} 3
Failure Reason:

"2024-06-17T15:52:32.516803+0000 mon.a (mon.0) 741 : cluster [WRN] Health check failed: 1 stray host(s) with 1 daemon(s) not managed by cephadm (CEPHADM_STRAY_HOST)" in cluster log

fail 7759580 2024-06-17 12:07:42 2024-06-17 15:20:58 2024-06-17 15:44:57 0:23:59 0:14:46 0:09:13 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-06-17T15:43:16.422795+0000 mon.a (mon.0) 551 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7759581 2024-06-17 12:07:43 2024-06-17 15:22:29 2024-06-17 15:47:10 0:24:41 0:12:30 0:12:11 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} 2
fail 7759582 2024-06-17 12:07:44 2024-06-17 15:24:19 2024-06-17 15:50:31 0:26:12 0:16:19 0:09:53 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-17T15:45:07.689242+0000 mon.smithi087 (mon.0) 841 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

fail 7759583 2024-06-17 12:07:45 2024-06-17 15:25:30 2024-06-17 16:27:47 1:02:17 0:51:07 0:11:10 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-06-17T16:19:00.420396+0000 mon.a (mon.0) 2577 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7759584 2024-06-17 12:07:47 2024-06-17 15:25:50 2024-06-17 16:01:07 0:35:17 0:22:56 0:12:21 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} 2
pass 7759585 2024-06-17 12:07:48 2024-06-17 15:27:51 2024-06-17 15:50:29 0:22:38 0:13:10 0:09:28 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7759586 2024-06-17 12:07:49 2024-06-17 15:27:51 2024-06-17 16:01:24 0:33:33 0:24:02 0:09:31 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7759587 2024-06-17 12:07:51 2024-06-17 15:27:52 2024-06-17 15:50:49 0:22:57 0:13:04 0:09:53 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7759588 2024-06-17 12:07:52 2024-06-17 15:27:52 2024-06-17 15:59:44 0:31:52 0:20:05 0:11:47 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1718639193.173996 mon.smithi012 (mon.0) 116 : cluster [WRN] Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log

fail 7759589 2024-06-17 12:07:53 2024-06-17 15:28:32 2024-06-17 15:52:50 0:24:18 0:14:15 0:10:03 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-17T15:49:32.296721+0000 mon.smithi096 (mon.0) 844 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7759590 2024-06-17 12:07:54 2024-06-17 15:29:53 2024-06-17 15:53:47 0:23:54 0:11:12 0:12:42 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
pass 7759591 2024-06-17 12:07:56 2024-06-17 15:32:54 2024-06-17 16:24:03 0:51:09 0:39:36 0:11:33 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
fail 7759592 2024-06-17 12:07:57 2024-06-17 15:34:24 2024-06-17 16:11:07 0:36:43 0:26:21 0:10:22 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-06-17T15:57:46.528879+0000 mon.a (mon.0) 486 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7759593 2024-06-17 12:07:58 2024-06-17 15:35:25 2024-06-17 16:30:33 0:55:08 0:45:49 0:09:19 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-06-17T16:00:39.000011+0000 mon.a (mon.0) 1027 : cluster [WRN] Health check failed: 17 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

dead 7759594 2024-06-17 12:07:59 2024-06-17 15:35:35 2024-06-17 15:37:29 0:01:54 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi039

pass 7759595 2024-06-17 12:08:01 2024-06-17 15:36:26 2024-06-17 16:07:35 0:31:09 0:18:35 0:12:34 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} 3
fail 7759596 2024-06-17 12:08:02 2024-06-17 15:37:46 2024-06-17 16:03:50 0:26:04 0:14:57 0:11:07 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-06-17T15:59:05.851685+0000 mon.smithi049 (mon.0) 794 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7759597 2024-06-17 12:08:03 2024-06-17 15:37:47 2024-06-17 16:34:19 0:56:32 0:40:35 0:15:57 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} 5
pass 7759598 2024-06-17 12:08:05 2024-06-17 15:54:38 2024-06-17 16:22:24 0:27:46 0:18:19 0:09:27 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} 2
pass 7759599 2024-06-17 12:08:06 2024-06-17 15:54:48 2024-06-17 16:25:08 0:30:20 0:19:53 0:10:27 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7759600 2024-06-17 12:08:07 2024-06-17 15:55:38 2024-06-17 16:43:42 0:48:04 0:32:44 0:15:20 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} 2
dead 7759601 2024-06-17 12:08:09 2024-06-17 15:59:49 2024-06-17 16:00:53 0:01:04 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi149

dead 7759602 2024-06-17 12:08:10 2024-06-17 15:59:50 2024-06-17 16:02:14 0:02:24 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi135

fail 7759603 2024-06-17 12:08:11 2024-06-17 16:01:10 2024-06-17 16:18:27 0:17:17 0:06:46 0:10:31 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

ceph version 18.2.2-1475.g07f5e007 was not installed, found 18.2.2-1440.g5a8ca914.el9.

fail 7759604 2024-06-17 12:08:13 2024-06-17 16:01:31 2024-06-17 16:44:15 0:42:44 0:31:57 0:10:47 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

reached maximum tries (301) after waiting for 300 seconds

pass 7759605 2024-06-17 12:08:14 2024-06-17 16:02:41 2024-06-17 17:07:55 1:05:14 0:55:17 0:09:57 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
dead 7759606 2024-06-17 12:08:15 2024-06-17 16:02:52 2024-06-18 04:12:51 12:09:59 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

dead 7759607 2024-06-17 12:08:16 2024-06-17 16:03:12 2024-06-17 16:04:56 0:01:44 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi049

pass 7759608 2024-06-17 12:08:18 2024-06-17 16:03:52 2024-06-17 16:29:36 0:25:44 0:15:33 0:10:11 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} 3
pass 7759609 2024-06-17 12:08:19 2024-06-17 16:04:53 2024-06-17 16:39:57 0:35:04 0:24:23 0:10:41 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} 2