Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7650506 2024-04-10 12:16:27 2024-04-10 12:18:44 2024-04-15 18:57:39 5 days, 6:38:55 1:00:35 5 days, 5:38:20 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

Command failed on smithi081 with status 127: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e7457d5c2678280d1715af178c6a4b756b1693d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8d9f24de-fb52-11ee-bc8f-c7b262605968 -- bash -c 'press F to pay respects'"

fail 7650507 2024-04-10 12:16:28 2024-04-10 12:19:15 2024-04-10 12:49:00 0:29:45 0:16:42 0:13:03 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} 3
Failure Reason:

"2024-04-10T12:42:06.118102+0000 mon.a (mon.0) 385 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7650508 2024-04-10 12:16:29 2024-04-10 12:21:36 2024-04-10 13:08:18 0:46:42 0:32:24 0:14:18 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
pass 7650509 2024-04-10 12:16:30 2024-04-10 12:23:16 2024-04-10 13:08:33 0:45:17 0:34:39 0:10:38 smithi main centos 9.stream orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
fail 7650510 2024-04-10 12:16:30 2024-04-10 12:23:17 2024-04-10 13:10:53 0:47:36 0:35:21 0:12:15 smithi main centos 9.stream orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} 1
Failure Reason:

"2024-04-10T12:45:57.598271+0000 mon.a (mon.0) 606 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7650511 2024-04-10 12:16:31 2024-04-10 12:23:37 2024-04-10 12:46:28 0:22:51 0:13:32 0:09:19 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-04-10T12:44:23.720042+0000 mon.a (mon.0) 534 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7650512 2024-04-10 12:16:32 2024-04-10 12:23:37 2024-04-10 12:48:56 0:25:19 0:14:24 0:10:55 smithi main ubuntu 22.04 orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} 2
pass 7650513 2024-04-10 12:16:33 2024-04-10 12:24:58 2024-04-10 12:56:14 0:31:16 0:19:44 0:11:32 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
fail 7650514 2024-04-10 12:16:34 2024-04-10 12:26:28 2024-04-10 13:04:18 0:37:50 0:26:52 0:10:58 smithi main centos 9.stream orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-04-10T12:43:06.034558+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7650515 2024-04-10 12:16:35 2024-04-10 12:27:19 2024-04-10 12:53:09 0:25:50 0:13:06 0:12:44 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} 2
fail 7650516 2024-04-10 12:16:36 2024-04-10 12:30:10 2024-04-10 12:57:24 0:27:14 0:17:07 0:10:07 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-04-10T12:50:38.033110+0000 mon.smithi003 (mon.0) 846 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7650517 2024-04-10 12:16:37 2024-04-10 12:30:20 2024-04-10 12:51:53 0:21:33 0:11:16 0:10:17 smithi main centos 9.stream orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} 1
pass 7650518 2024-04-10 12:16:38 2024-04-10 12:31:00 2024-04-10 12:53:14 0:22:14 0:11:43 0:10:31 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
pass 7650519 2024-04-10 12:16:39 2024-04-10 12:32:11 2024-04-10 13:34:00 1:01:49 0:50:22 0:11:27 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
pass 7650520 2024-04-10 12:16:40 2024-04-10 12:34:52 2024-04-10 13:22:33 0:47:41 0:36:29 0:11:12 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
dead 7650521 2024-04-10 12:16:41 2024-04-10 12:34:52 2024-04-10 12:56:17 0:21:25 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} 2
Failure Reason:

Error reimaging machines: reached maximum tries (101) after waiting for 600 seconds

pass 7650522 2024-04-10 12:16:42 2024-04-10 12:36:33 2024-04-16 00:35:35 5 days, 11:59:02 0:26:13 5 days, 11:32:49 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
Failure Reason:

Spec did not match any workunits: 'cephadm/foo'

pass 7650523 2024-04-10 12:16:43 2024-04-10 12:36:33 2024-04-10 13:04:30 0:27:57 0:17:24 0:10:33 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
fail 7650524 2024-04-10 12:16:44 2024-04-10 12:37:24 2024-04-10 13:09:29 0:32:05 0:21:45 0:10:20 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-04-10T13:03:59.457225+0000 mon.smithi042 (mon.0) 863 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7650525 2024-04-10 12:16:45 2024-04-10 12:37:55 2024-04-10 13:01:05 0:23:10 0:13:46 0:09:24 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7650526 2024-04-10 12:16:46 2024-04-10 12:38:05 2024-04-10 13:19:12 0:41:07 0:25:00 0:16:07 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
Failure Reason:

"2024-04-10T13:04:02.916998+0000 mon.a (mon.0) 391 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7650527 2024-04-10 12:16:47 2024-04-10 12:43:26 2024-04-10 13:22:15 0:38:49 0:26:03 0:12:46 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
Failure Reason:

"2024-04-10T13:12:04.199319+0000 mon.a (mon.0) 532 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7650528 2024-04-10 12:16:48 2024-04-10 12:46:17 2024-04-10 14:03:55 1:17:38 1:07:43 0:09:55 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7650529 2024-04-10 12:16:49 2024-04-10 12:46:57 2024-04-10 13:12:15 0:25:18 0:14:14 0:11:04 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-04-10T13:07:42.702037+0000 mon.smithi047 (mon.0) 799 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7650530 2024-04-10 12:16:50 2024-04-10 12:46:58 2024-04-10 13:27:16 0:40:18 0:26:24 0:13:54 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} 5
pass 7650531 2024-04-10 12:16:51 2024-04-10 12:51:59 2024-04-10 13:13:58 0:21:59 0:11:33 0:10:26 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} 2
pass 7650532 2024-04-10 12:16:52 2024-04-10 12:52:19 2024-04-10 13:18:25 0:26:06 0:14:16 0:11:50 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7650533 2024-04-10 12:16:53 2024-04-10 12:53:20 2024-04-10 13:52:50 0:59:30 0:49:27 0:10:03 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
pass 7650534 2024-04-10 12:16:54 2024-04-10 12:53:20 2024-04-10 13:41:03 0:47:43 0:38:54 0:08:49 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
pass 7650535 2024-04-10 12:16:55 2024-04-10 12:53:20 2024-04-10 13:20:02 0:26:42 0:14:28 0:12:14 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} 3
pass 7650536 2024-04-10 12:16:56 2024-04-10 12:54:31 2024-04-10 13:26:02 0:31:31 0:21:20 0:10:11 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
fail 7650537 2024-04-10 12:16:57 2024-04-10 12:54:31 2024-04-10 13:19:41 0:25:10 0:14:54 0:10:16 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-04-10T13:15:01.460040+0000 mon.smithi077 (mon.0) 819 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7650538 2024-04-10 12:16:58 2024-04-10 12:54:42 2024-04-10 13:40:23 0:45:41 0:34:00 0:11:41 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-04-10T13:30:00.000166+0000 mon.smithi157 (mon.0) 441 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down" in cluster log

pass 7650539 2024-04-10 12:16:59 2024-04-10 12:54:42 2024-04-10 13:26:43 0:32:01 0:20:46 0:11:15 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
fail 7650540 2024-04-10 12:17:00 2024-04-10 12:55:42 2024-04-10 13:19:12 0:23:30 0:12:00 0:11:30 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-04-10T13:13:41.153834+0000 mon.a (mon.0) 443 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7650541 2024-04-10 12:17:01 2024-04-10 12:55:53 2024-04-10 13:46:10 0:50:17 0:40:19 0:09:58 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

"2024-04-10T13:16:17.559233+0000 mon.a (mon.0) 375 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7650542 2024-04-10 12:17:02 2024-04-10 12:56:23 2024-04-10 13:25:57 0:29:34 0:17:36 0:11:58 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
Failure Reason:

"2024-04-10T13:18:52.414233+0000 mon.a (mon.0) 437 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7650543 2024-04-10 12:17:03 2024-04-10 12:56:34 2024-04-10 13:26:09 0:29:35 0:16:01 0:13:34 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
pass 7650544 2024-04-10 12:17:04 2024-04-10 13:01:15 2024-04-10 13:23:19 0:22:04 0:12:31 0:09:33 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} 1
pass 7650545 2024-04-10 12:17:05 2024-04-10 13:01:35 2024-04-10 13:27:44 0:26:09 0:13:31 0:12:38 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw} 2
pass 7650546 2024-04-10 12:17:06 2024-04-10 13:04:26 2024-04-10 13:34:24 0:29:58 0:19:27 0:10:31 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} 2
fail 7650547 2024-04-10 12:17:07 2024-04-10 13:04:36 2024-04-15 20:21:27 5 days, 7:16:51 1:12:29 5 days, 6:04:22 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

Command failed on smithi167 with status 127: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e7457d5c2678280d1715af178c6a4b756b1693d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 380d7412-fb5c-11ee-bc8f-c7b262605968 -- bash -c 'P03 was here'"

pass 7650548 2024-04-10 12:17:08 2024-04-10 13:08:27 2024-04-10 13:36:25 0:27:58 0:18:51 0:09:07 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
pass 7650549 2024-04-10 12:17:09 2024-04-10 13:08:38 2024-04-10 14:18:35 1:09:57 0:59:33 0:10:24 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
pass 7650550 2024-04-10 12:17:10 2024-04-10 13:09:58 2024-04-10 14:05:23 0:55:25 0:45:20 0:10:05 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} 2
pass 7650551 2024-04-10 12:17:11 2024-04-10 13:09:58 2024-04-10 13:33:01 0:23:03 0:12:19 0:10:44 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} 2
fail 7650552 2024-04-10 12:17:12 2024-04-10 13:09:59 2024-04-10 14:28:56 1:18:57 1:09:08 0:09:49 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7650553 2024-04-10 12:17:13 2024-04-10 13:09:59 2024-04-10 13:35:20 0:25:21 0:14:19 0:11:02 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
pass 7650554 2024-04-10 12:17:14 2024-04-10 13:10:10 2024-04-10 13:39:49 0:29:39 0:19:47 0:09:52 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} 1
pass 7650555 2024-04-10 12:17:15 2024-04-10 13:10:10 2024-04-10 13:37:32 0:27:22 0:14:34 0:12:48 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
pass 7650556 2024-04-10 12:17:16 2024-04-10 13:11:51 2024-04-10 13:42:10 0:30:19 0:18:32 0:11:47 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7650557 2024-04-10 12:17:18 2024-04-10 13:14:01 2024-04-10 13:55:46 0:41:45 0:31:43 0:10:02 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
pass 7650558 2024-04-10 12:17:19 2024-04-10 13:14:12 2024-04-10 13:52:38 0:38:26 0:23:11 0:15:15 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} 2
pass 7650559 2024-04-10 12:17:20 2024-04-10 13:17:52 2024-04-10 13:34:53 0:17:01 0:06:48 0:10:13 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
pass 7650560 2024-04-10 12:17:21 2024-04-10 13:17:53 2024-04-10 13:59:40 0:41:47 0:31:35 0:10:12 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7650561 2024-04-10 12:17:22 2024-04-10 13:18:33 2024-04-10 13:42:34 0:24:01 0:14:28 0:09:33 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} 1
Failure Reason:

"2024-04-10T13:41:25.246579+0000 mon.a (mon.0) 211 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7650562 2024-04-10 12:17:22 2024-04-10 13:20:04 2024-04-10 13:43:37 0:23:33 0:12:04 0:11:29 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli} 2
pass 7650563 2024-04-10 12:17:23 2024-04-10 13:20:04 2024-04-10 13:42:30 0:22:26 0:11:30 0:10:56 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} 2
pass 7650564 2024-04-10 12:17:24 2024-04-10 13:21:35 2024-04-10 13:48:00 0:26:25 0:14:16 0:12:09 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
fail 7650565 2024-04-10 12:17:25 2024-04-10 13:22:35 2024-04-10 13:46:26 0:23:51 0:13:02 0:10:49 smithi main centos 9.stream orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
Failure Reason:

"2024-04-10T13:40:23.077399+0000 mon.smithi089 (mon.0) 283 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7650566 2024-04-10 12:17:26 2024-04-10 13:23:26 2024-04-10 13:48:29 0:25:03 0:12:16 0:12:47 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-04-10T13:43:00.712527+0000 mon.a (mon.0) 398 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7650567 2024-04-10 12:17:28 2024-04-10 13:25:28 2024-04-10 14:20:38 0:55:10 0:45:14 0:09:56 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi094 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b00e16ae-f73f-11ee-b64a-cb9ed24678a4 -e sha1=2e7457d5c2678280d1715af178c6a4b756b1693d -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''

pass 7650568 2024-04-10 12:17:29 2024-04-10 13:25:28 2024-04-10 13:50:57 0:25:29 0:16:03 0:09:26 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7650569 2024-04-10 12:17:30 2024-04-10 13:25:29 2024-04-10 13:48:23 0:22:54 0:14:08 0:08:46 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
pass 7650570 2024-04-10 12:17:31 2024-04-10 13:25:29 2024-04-10 13:48:40 0:23:11 0:14:32 0:08:39 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
fail 7650571 2024-04-10 12:17:32 2024-04-10 13:25:40 2024-04-10 13:48:30 0:22:50 0:14:22 0:08:28 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} 2
Failure Reason:

"2024-04-10T13:46:31.888567+0000 mon.a (mon.0) 51 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7650572 2024-04-10 12:17:33 2024-04-10 13:25:40 2024-04-10 13:57:07 0:31:27 0:21:56 0:09:31 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-04-10T13:53:38.777300+0000 mon.smithi099 (mon.0) 966 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7650573 2024-04-10 12:17:34 2024-04-10 13:25:40 2024-04-10 14:14:57 0:49:17 0:39:02 0:10:15 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7650574 2024-04-10 12:17:35 2024-04-10 13:25:41 2024-04-10 13:50:38 0:24:57 0:15:13 0:09:44 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
pass 7650575 2024-04-10 12:17:36 2024-04-10 13:25:41 2024-04-10 14:27:43 1:02:02 0:51:15 0:10:47 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
pass 7650576 2024-04-10 12:17:37 2024-04-10 13:26:11 2024-04-10 14:00:31 0:34:20 0:24:01 0:10:19 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys} 2
fail 7650577 2024-04-10 12:17:38 2024-04-10 13:27:22 2024-04-10 14:01:19 0:33:57 0:22:51 0:11:06 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} 3
Failure Reason:

"2024-04-10T13:58:39.052981+0000 mon.a (mon.0) 741 : cluster [WRN] Health check failed: 1 stray host(s) with 1 daemon(s) not managed by cephadm (CEPHADM_STRAY_HOST)" in cluster log

fail 7650578 2024-04-10 12:17:39 2024-04-10 13:27:22 2024-04-10 13:50:59 0:23:37 0:13:44 0:09:53 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-04-10T13:48:56.903564+0000 mon.a (mon.0) 537 : cluster [WRN] Health check failed: cephadm background work is paused (CEPHADM_PAUSED)" in cluster log

pass 7650579 2024-04-10 12:17:40 2024-04-10 13:27:53 2024-04-10 13:51:35 0:23:42 0:13:33 0:10:09 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7650580 2024-04-10 12:17:41 2024-04-10 13:28:33 2024-04-10 13:55:08 0:26:35 0:13:49 0:12:46 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} 2
fail 7650581 2024-04-10 12:17:42 2024-04-10 13:34:35 2024-04-10 14:04:45 0:30:10 0:18:11 0:11:59 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-04-10T13:57:23.807235+0000 mon.smithi029 (mon.0) 868 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

dead 7650582 2024-04-10 12:17:43 2024-04-10 13:35:25 2024-04-11 01:43:57 12:08:32 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
Failure Reason:

hit max job timeout

pass 7650583 2024-04-10 12:17:44 2024-04-10 13:35:26 2024-04-10 14:10:14 0:34:48 0:24:28 0:10:20 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7650584 2024-04-10 12:17:45 2024-04-10 13:36:26 2024-04-10 14:20:00 0:43:34 0:32:06 0:11:28 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7650585 2024-04-10 12:17:46 2024-04-10 13:36:36 2024-04-10 14:03:19 0:26:43 0:14:54 0:11:49 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-04-10T13:58:47.933080+0000 mon.smithi066 (mon.0) 843 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7650586 2024-04-10 12:17:47 2024-04-10 13:37:37 2024-04-10 14:02:17 0:24:40 0:12:06 0:12:34 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
pass 7650587 2024-04-10 12:17:48 2024-04-10 13:41:08 2024-04-10 14:29:10 0:48:02 0:38:44 0:09:18 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
pass 7650588 2024-04-10 12:17:49 2024-04-10 13:41:08 2024-04-10 14:10:54 0:29:46 0:19:19 0:10:27 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7650589 2024-04-10 12:17:50 2024-04-10 13:41:09 2024-04-10 14:18:35 0:37:26 0:25:37 0:11:49 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-04-10T14:04:06.270738+0000 mon.a (mon.0) 380 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7650590 2024-04-10 12:17:51 2024-04-10 13:42:19 2024-04-10 14:40:01 0:57:42 0:48:56 0:08:46 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
pass 7650591 2024-04-10 12:17:52 2024-04-10 13:42:20 2024-04-10 14:38:16 0:55:56 0:45:28 0:10:28 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} 2
pass 7650592 2024-04-10 12:17:53 2024-04-10 13:42:40 2024-04-10 14:13:23 0:30:43 0:19:47 0:10:56 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} 3
fail 7650593 2024-04-10 12:17:54 2024-04-10 13:43:40 2024-04-10 14:13:06 0:29:26 0:14:27 0:14:59 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-04-10T14:08:39.234457+0000 mon.smithi154 (mon.0) 781 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

pass 7650594 2024-04-10 12:17:55 2024-04-10 13:47:51 2024-04-10 14:36:43 0:48:52 0:37:44 0:11:08 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} 5
pass 7650595 2024-04-10 12:17:56 2024-04-10 13:48:32 2024-04-10 14:16:31 0:27:59 0:18:33 0:09:26 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} 2
pass 7650596 2024-04-10 12:17:57 2024-04-10 13:48:42 2024-04-10 14:20:36 0:31:54 0:20:06 0:11:48 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
fail 7650597 2024-04-10 12:17:58 2024-04-10 13:50:43 2024-04-10 14:24:37 0:33:54 0:20:39 0:13:15 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} 3
Failure Reason:

"2024-04-10T14:17:21.860294+0000 mon.a (mon.0) 441 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7650598 2024-04-10 12:17:59 2024-04-10 13:51:44 2024-04-10 14:41:47 0:50:03 0:39:18 0:10:45 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7650599 2024-04-10 12:18:00 2024-04-10 13:52:44 2024-04-10 14:17:40 0:24:56 0:14:25 0:10:31 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
fail 7650600 2024-04-10 12:18:01 2024-04-10 13:52:55 2024-04-16 12:38:19 5 days, 22:45:24 0:17:53 5 days, 22:27:31 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

Command failed on smithi079 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2e7457d5c2678280d1715af178c6a4b756b1693d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 530936bc-fbec-11ee-bc8f-c7b262605968 -- bash -c \'set -e\nset -x\nceph orch ps\nHOST=$(hostname -s)\nOSD=$(ceph orch ps $HOST | grep osd | head -n 1 | awk \'"\'"\'{print $1}\'"\'"\')\necho "host $HOST, osd $OSD"\nceph orch daemon stop $OSD\nwhile ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\nceph auth export $OSD > k\nceph orch daemon rm $OSD --force\nceph orch ps --refresh\nwhile ceph orch ps | grep $OSD ; do sleep 5 ; done\nceph auth add $OSD -i k\nceph cephadm osd activate $HOST\nwhile ! ceph orch ps | grep $OSD | grep running ; do sleep 5 ; done\npress F to pay respects\n\''

pass 7650601 2024-04-10 12:18:02 2024-04-10 13:55:15 2024-04-10 14:25:38 0:30:23 0:18:36 0:11:47 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
pass 7650602 2024-04-10 12:18:03 2024-04-10 13:55:56 2024-04-10 14:57:25 1:01:29 0:50:41 0:10:48 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
pass 7650603 2024-04-10 12:18:04 2024-04-10 13:56:36 2024-04-10 14:29:30 0:32:54 0:23:06 0:09:48 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} 2