Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7739716 2024-06-03 20:10:42 2024-06-10 10:58:48 2024-06-10 11:18:23 0:19:35 0:07:35 0:12:00 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_res_dom} 2
Failure Reason:

"2024-06-10T11:16:00.808086+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739717 2024-06-03 20:10:43 2024-06-10 11:17:15 2024-06-10 11:19:59 0:02:44 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi153

fail 7739718 2024-06-03 20:10:44 2024-06-10 11:18:56 2024-06-10 11:40:45 0:21:49 0:12:06 0:09:43 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-06-10T11:38:26.773585+0000 mon.a (mon.0) 101 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739719 2024-06-03 20:10:45 2024-06-10 11:18:56 2024-06-10 11:37:55 0:18:59 0:09:07 0:09:52 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
Failure Reason:

Command failed on smithi114 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6e13b2aa-271d-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739720 2024-06-03 20:10:46 2024-06-10 11:18:56 2024-06-10 11:38:00 0:19:04 0:09:53 0:09:11 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
Failure Reason:

"2024-06-10T11:36:36.101621+0000 mon.a (mon.0) 105 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739721 2024-06-03 20:10:47 2024-06-10 11:18:57 2024-06-10 12:08:15 0:49:18 0:33:03 0:16:15 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

"2024-06-10T11:45:06.272504+0000 mon.smithi046 (mon.0) 109 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739722 2024-06-03 20:10:48 2024-06-10 11:24:28 2024-06-10 11:45:45 0:21:17 0:09:22 0:11:55 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_python} 2
Failure Reason:

"2024-06-10T11:43:15.437779+0000 mon.a (mon.0) 109 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739723 2024-06-03 20:10:49 2024-06-10 11:26:29 2024-06-10 11:56:07 0:29:38 0:19:57 0:09:41 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1718019943.7617955 mgr.smithi176.mvggcj (mgr.14236) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7739724 2024-06-03 20:10:50 2024-06-10 11:27:09 2024-06-10 12:11:27 0:44:18 0:34:02 0:10:16 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

Command failed on smithi112 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5dd798ec-271e-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7739725 2024-06-03 20:10:51 2024-06-10 11:28:40 2024-06-10 11:56:18 0:27:38 0:13:21 0:14:17 smithi main ubuntu 22.04 orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignore_mgr_down ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
Failure Reason:

"2024-06-10T11:53:43.815796+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7739726 2024-06-03 20:10:52 2024-06-10 11:32:51 2024-06-10 11:51:48 0:18:57 0:09:29 0:09:28 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_adoption} 1
fail 7739727 2024-06-03 20:10:53 2024-06-10 11:32:51 2024-06-10 11:52:44 0:19:53 0:08:52 0:11:01 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} 2
Failure Reason:

Test failure: test_device_ls (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli)

fail 7739728 2024-06-03 20:10:54 2024-06-10 11:33:42 2024-06-10 12:04:08 0:30:26 0:16:37 0:13:49 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

"2024-06-10T11:53:01.620428+0000 mon.smithi148 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739729 2024-06-03 20:10:54 2024-06-10 11:37:02 2024-06-10 11:55:01 0:17:59 0:07:43 0:10:16 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-06-10T11:52:45.958020+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739730 2024-06-03 20:10:56 2024-06-10 11:37:43 2024-06-10 11:56:48 0:19:05 0:09:44 0:09:21 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} 2
Failure Reason:

"2024-06-10T11:56:10.527249+0000 mon.a (mon.0) 104 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739731 2024-06-03 20:10:57 2024-06-10 11:37:55 2024-06-10 12:03:02 0:25:07 0:16:22 0:08:45 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-06-10T11:53:01.139246+0000 mon.smithi114 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739732 2024-06-03 20:10:58 2024-06-10 11:38:05 2024-06-10 11:54:42 0:16:37 0:07:27 0:09:10 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} 1
Failure Reason:

"2024-06-10T11:53:07.867794+0000 mon.smithi007 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739733 2024-06-03 20:10:58 2024-06-10 11:38:05 2024-06-10 11:56:06 0:18:01 0:07:55 0:10:06 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-06-10T11:54:07.156814+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739734 2024-06-03 20:10:59 2024-06-10 11:38:56 2024-06-10 12:11:46 0:32:50 0:22:55 0:09:55 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

Command failed on smithi018 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ce89b952-271f-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7739735 2024-06-03 20:11:00 2024-06-10 11:38:56 2024-06-10 12:01:24 0:22:28 0:11:01 0:11:27 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/connectivity task/test_monitoring_stack_basic} 3
Failure Reason:

"2024-06-10T11:58:50.529129+0000 mon.a (mon.0) 109 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739736 2024-06-03 20:11:01 2024-06-10 11:39:27 2024-06-10 12:05:07 0:25:40 0:15:47 0:09:53 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-06-10T11:54:46.332908+0000 mon.smithi069 (mon.0) 122 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739737 2024-06-03 20:11:02 2024-06-10 11:39:57 2024-06-10 12:22:03 0:42:06 0:31:37 0:10:29 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-06-10T11:59:43.337852+0000 mon.smithi012 (mon.0) 116 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739738 2024-06-03 20:11:03 2024-06-10 11:40:08 2024-06-10 12:05:32 0:25:24 0:16:24 0:09:00 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-06-10T11:55:24.623407+0000 mon.smithi026 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739739 2024-06-03 20:11:04 2024-06-10 11:40:38 2024-06-10 11:41:52 0:01:14 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi156

fail 7739740 2024-06-03 20:11:05 2024-06-10 11:40:48 2024-06-10 12:07:43 0:26:55 0:14:24 0:12:31 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/classic task/test_rgw_multisite} 3
Failure Reason:

Command failed on smithi129 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 12b365a0-2721-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

dead 7739741 2024-06-03 20:11:06 2024-06-10 11:41:59 2024-06-10 11:43:13 0:01:14 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi073

fail 7739742 2024-06-03 20:11:07 2024-06-10 11:42:09 2024-06-10 12:02:02 0:19:53 0:10:00 0:09:53 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-06-10T12:00:27.325788+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739743 2024-06-03 20:11:08 2024-06-10 11:43:10 2024-06-10 12:22:15 0:39:05 0:26:11 0:12:54 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi033 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0c78155a-2721-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7739744 2024-06-03 20:11:09 2024-06-10 11:46:06 2024-06-10 12:04:48 0:18:42 0:09:18 0:09:24 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_cephadm_timeout} 1
Failure Reason:

Command failed on smithi078 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1cab1288-2721-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739745 2024-06-03 20:11:10 2024-06-10 11:46:06 2024-06-10 12:13:16 0:27:10 0:16:15 0:10:55 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-06-10T12:03:43.180446+0000 mon.smithi156 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739746 2024-06-03 20:11:11 2024-06-10 11:48:07 2024-06-10 12:09:01 0:20:54 0:11:48 0:09:06 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async start tasks/rotate-keys} 2
Failure Reason:

"2024-06-10T12:07:56.751829+0000 mon.a (mon.0) 105 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739747 2024-06-03 20:11:12 2024-06-10 11:48:07 2024-06-10 12:08:18 0:20:11 0:07:29 0:12:42 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
Failure Reason:

Command failed on smithi059 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 77d0e9ee-2721-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'

fail 7739748 2024-06-03 20:11:12 2024-06-10 11:48:48 2024-06-10 12:10:53 0:22:05 0:11:26 0:10:39 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/connectivity task/test_set_mon_crush_locations} 3
Failure Reason:

"2024-06-10T12:08:38.270507+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739749 2024-06-03 20:11:13 2024-06-10 11:50:09 2024-06-10 12:31:47 0:41:38 0:33:00 0:08:38 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-06-10T12:08:43.726211+0000 mon.smithi107 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739750 2024-06-03 20:11:14 2024-06-10 11:50:09 2024-06-10 12:33:05 0:42:56 0:31:38 0:11:18 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-06-10T12:10:10.179062+0000 mon.smithi045 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739751 2024-06-03 20:11:15 2024-06-10 11:51:09 2024-06-10 12:13:43 0:22:34 0:11:28 0:11:06 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_basic} 2
Failure Reason:

"2024-06-10T12:11:31.182639+0000 mon.a (mon.0) 104 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739752 2024-06-03 20:11:16 2024-06-10 11:51:50 2024-06-10 12:21:47 0:29:57 0:18:50 0:11:07 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-06-10T12:11:02.542096+0000 mon.smithi098 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739753 2024-06-03 20:11:17 2024-06-10 11:52:20 2024-06-10 12:11:57 0:19:37 0:08:20 0:11:17 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

Command failed on smithi084 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d8e6c604-2721-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'

dead 7739754 2024-06-03 20:11:18 2024-06-10 11:52:51 2024-06-10 11:55:05 0:02:14 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi038

fail 7739755 2024-06-03 20:11:19 2024-06-10 11:54:01 2024-06-10 12:14:20 0:20:19 0:09:41 0:10:38 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_ca_signed_key} 2
Failure Reason:

"2024-06-10T12:13:08.664039+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739756 2024-06-03 20:11:20 2024-06-10 11:55:02 2024-06-10 12:22:08 0:27:06 0:16:31 0:10:35 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-06-10T12:10:42.085537+0000 mon.smithi007 (mon.0) 122 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739757 2024-06-03 20:11:21 2024-06-10 11:55:02 2024-06-10 12:19:58 0:24:56 0:13:55 0:11:01 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-06-10T12:17:29.596714+0000 mon.a (mon.0) 100 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739758 2024-06-03 20:11:22 2024-06-10 11:56:13 2024-06-10 12:25:18 0:29:05 0:20:26 0:08:39 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1718021615.947517 mon.smithi052 (mon.0) 116 : cluster [WRN] Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log

fail 7739759 2024-06-03 20:11:23 2024-06-10 11:56:13 2024-06-10 12:17:38 0:21:25 0:12:21 0:09:04 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli} 1
Failure Reason:

"2024-06-10T12:15:46.548785+0000 mon.a (mon.0) 100 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739760 2024-06-03 20:11:24 2024-06-10 11:56:13 2024-06-10 12:23:41 0:27:28 0:16:25 0:11:03 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-06-10T12:12:35.414971+0000 mon.smithi040 (mon.0) 118 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739761 2024-06-03 20:11:25 2024-06-10 11:56:54 2024-06-10 12:14:29 0:17:35 0:07:16 0:10:19 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/host rook/1.7.2} 1
Failure Reason:

Command failed on smithi077 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7739762 2024-06-03 20:11:26 2024-06-10 11:56:54 2024-06-10 12:18:45 0:21:51 0:07:44 0:14:07 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_domain} 2
Failure Reason:

"2024-06-10T12:18:04.923744+0000 mon.a (mon.0) 107 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739763 2024-06-03 20:11:27 2024-06-10 12:04:08 2024-06-10 12:45:35 0:41:27 0:31:58 0:09:29 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-10T12:23:10.772588+0000 mon.smithi038 (mon.0) 109 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739764 2024-06-03 20:11:28 2024-06-10 12:04:09 2024-06-10 12:28:25 0:24:16 0:13:46 0:10:30 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_api_tests} 2
Failure Reason:

"2024-06-10T12:25:46.428721+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7739765 2024-06-03 20:11:29 2024-06-10 12:04:49 2024-06-10 12:28:09 0:23:20 0:13:57 0:09:23 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm} 1
fail 7739766 2024-06-03 20:11:30 2024-06-10 12:05:10 2024-06-10 12:30:31 0:25:21 0:15:51 0:09:30 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-06-10T12:20:00.640328+0000 mon.smithi104 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739767 2024-06-03 20:11:31 2024-06-10 12:05:40 2024-06-10 12:32:53 0:27:13 0:15:59 0:11:14 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-10T12:22:46.597033+0000 mon.smithi144 (mon.0) 120 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739768 2024-06-03 20:11:32 2024-06-10 12:07:51 2024-06-10 12:24:38 0:16:47 0:07:47 0:09:00 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-06-10T12:22:25.709687+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739769 2024-06-03 20:11:33 2024-06-10 12:07:51 2024-06-10 12:09:25 0:01:34 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi163

fail 7739770 2024-06-03 20:11:34 2024-06-10 12:08:21 2024-06-10 12:25:34 0:17:13 0:07:00 0:10:13 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_res_basic} 2
Failure Reason:

"2024-06-10T12:24:10.302363+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739771 2024-06-03 20:11:35 2024-06-10 12:08:22 2024-06-10 12:26:27 0:18:05 0:08:42 0:09:23 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_cephadm_repos} 1
Failure Reason:

Command failed (workunit test cephadm/test_repos.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=087aaa2e55fee49d0c9c03c269dcf326d62c113a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

fail 7739772 2024-06-03 20:11:36 2024-06-10 12:09:02 2024-06-10 12:50:43 0:41:41 0:32:15 0:09:26 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-06-10T12:27:59.284597+0000 mon.smithi073 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739773 2024-06-03 20:11:37 2024-06-10 12:09:03 2024-06-10 12:30:04 0:21:01 0:09:41 0:11:20 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

"2024-06-10T12:27:49.888581+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739774 2024-06-03 20:11:38 2024-06-10 12:11:03 2024-06-10 12:45:42 0:34:39 0:25:56 0:08:43 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi002 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 80fa0192-2724-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7739775 2024-06-03 20:11:38 2024-06-10 12:11:04 2024-06-10 12:32:46 0:21:42 0:11:32 0:10:10 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli_mon} 5
Failure Reason:

"2024-06-10T12:30:36.045274+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7739776 2024-06-03 20:11:39 2024-06-10 12:11:54 2024-06-10 12:35:37 0:23:43 0:14:21 0:09:22 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} 2
fail 7739777 2024-06-03 20:11:40 2024-06-10 12:12:04 2024-06-10 12:37:11 0:25:07 0:15:57 0:09:10 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-06-10T12:27:22.915585+0000 mon.smithi132 (mon.0) 120 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739778 2024-06-03 20:11:41 2024-06-10 12:12:05 2024-06-10 12:35:16 0:23:11 0:11:25 0:11:46 smithi main ubuntu 22.04 orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} 1
Failure Reason:

"2024-06-10T12:32:42.041006+0000 mon.smithi156 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739779 2024-06-03 20:11:42 2024-06-10 12:13:25 2024-06-10 12:33:00 0:19:35 0:08:05 0:11:30 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

Command failed on smithi016 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid cde4a9b2-2724-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'

fail 7739780 2024-06-03 20:11:43 2024-06-10 12:13:46 2024-06-10 12:33:36 0:19:50 0:09:18 0:10:32 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_extra_daemon_features} 2
Failure Reason:

Command failed on smithi105 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1a0d05c8-2725-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739781 2024-06-03 20:11:44 2024-06-10 12:14:26 2024-06-10 12:42:13 0:27:47 0:18:02 0:09:45 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-06-10T12:30:56.745849+0000 mon.smithi077 (mon.0) 121 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

fail 7739782 2024-06-03 20:11:45 2024-06-10 12:15:07 2024-06-10 13:01:14 0:46:07 0:33:03 0:13:04 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-06-10T12:36:45.660883+0000 mon.smithi086 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739783 2024-06-03 20:11:46 2024-06-10 12:17:27 2024-06-10 12:39:15 0:21:48 0:12:24 0:09:24 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_dom} 2
Failure Reason:

"2024-06-10T12:37:31.946643+0000 mon.a (mon.0) 100 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739784 2024-06-03 20:11:47 2024-06-10 12:17:48 2024-06-10 12:19:52 0:02:04 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rados_python} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi179

fail 7739785 2024-06-03 20:11:48 2024-06-10 12:18:48 2024-06-10 13:37:12 1:18:24 1:04:38 0:13:46 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

Command failed on smithi059 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 764944fe-2726-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\''

fail 7739786 2024-06-03 20:11:49 2024-06-10 12:18:59 2024-06-10 13:03:54 0:44:55 0:31:39 0:13:16 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-06-10T12:40:27.664457+0000 mon.smithi119 (mon.0) 116 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739787 2024-06-03 20:11:50 2024-06-10 12:19:59 2024-06-10 12:22:53 0:02:54 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_host_drain} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi098

dead 7739788 2024-06-03 20:11:51 2024-06-10 12:21:50 2024-06-10 12:23:14 0:01:24 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi007

fail 7739789 2024-06-03 20:11:52 2024-06-10 12:22:10 2024-06-10 12:53:26 0:31:16 0:20:55 0:10:21 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-06-10T12:39:54.126090+0000 mon.smithi012 (mon.0) 121 : cluster [WRN] Health check failed: 1 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH)" in cluster log

dead 7739790 2024-06-03 20:11:53 2024-06-10 12:22:11 2024-06-10 12:23:14 0:01:03 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_adoption} 1
Failure Reason:

Error reimaging machines: Failed to power on smithi154

fail 7739791 2024-06-03 20:11:54 2024-06-10 12:22:11 2024-06-10 13:06:21 0:44:10 0:33:00 0:11:10 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

"2024-06-10T12:42:34.155452+0000 mon.smithi088 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739792 2024-06-03 20:11:55 2024-06-10 12:22:11 2024-06-10 12:48:17 0:26:06 0:13:53 0:12:13 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} 2
Failure Reason:

"2024-06-10T12:45:29.106568+0000 mon.a (mon.0) 101 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739793 2024-06-03 20:11:56 2024-06-10 12:22:22 2024-06-10 12:45:15 0:22:53 0:11:55 0:10:58 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
Failure Reason:

Command failed on smithi040 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 881dc948-2726-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'

fail 7739794 2024-06-03 20:11:57 2024-06-10 12:23:42 2024-06-10 12:49:38 0:25:56 0:16:36 0:09:20 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-06-10T12:39:49.683020+0000 mon.smithi026 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739795 2024-06-03 20:11:58 2024-06-10 12:24:43 2024-06-10 12:44:27 0:19:44 0:10:04 0:09:40 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-06-10T12:43:06.478611+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739796 2024-06-03 20:11:59 2024-06-10 12:25:23 2024-06-10 13:07:28 0:42:05 0:32:05 0:10:00 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-06-10T12:45:10.461429+0000 mon.smithi046 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739797 2024-06-03 20:12:00 2024-06-10 12:25:43 2024-06-10 12:45:24 0:19:41 0:08:52 0:10:49 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
Failure Reason:

Command failed on smithi145 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c70571ce-2726-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739798 2024-06-03 20:12:01 2024-06-10 12:26:34 2024-06-10 12:53:26 0:26:52 0:16:32 0:10:20 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-06-10T12:43:48.131977+0000 mon.smithi069 (mon.0) 122 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739799 2024-06-03 20:12:02 2024-06-10 12:28:35 2024-06-10 12:49:08 0:20:33 0:08:10 0:12:23 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-06-10T12:46:43.936021+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739800 2024-06-03 20:12:03 2024-06-10 12:30:05 2024-06-10 12:49:07 0:19:02 0:10:23 0:08:39 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rotate-keys} 2
Failure Reason:

"2024-06-10T12:47:36.060256+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739801 2024-06-03 20:12:04 2024-06-10 12:30:05 2024-06-10 12:55:31 0:25:26 0:16:05 0:09:21 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-06-10T12:45:13.529480+0000 mon.smithi104 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739802 2024-06-03 20:12:04 2024-06-10 12:30:36 2024-06-10 12:48:46 0:18:10 0:07:56 0:10:14 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} 2
Failure Reason:

"2024-06-10T12:47:41.481642+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739803 2024-06-03 20:12:05 2024-06-10 12:31:56 2024-06-10 12:57:52 0:25:56 0:16:33 0:09:23 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-06-10T12:47:58.371284+0000 mon.smithi089 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739804 2024-06-03 20:12:06 2024-06-10 12:32:57 2024-06-10 12:54:08 0:21:11 0:10:33 0:10:38 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_monitoring_stack_basic} 3
Failure Reason:

"2024-06-10T12:51:26.822228+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739805 2024-06-03 20:12:07 2024-06-10 12:32:57 2024-06-10 13:03:42 0:30:45 0:20:50 0:09:55 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1718023924.4081168 mgr.smithi112.tqbjld (mgr.14244) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7739806 2024-06-03 20:12:08 2024-06-10 12:32:58 2024-06-10 12:56:30 0:23:32 0:12:29 0:11:03 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_cephadm_timeout} 1
Failure Reason:

Command failed on smithi107 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1127d43a-2728-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739807 2024-06-03 20:12:09 2024-06-10 12:32:58 2024-06-10 13:14:45 0:41:47 0:31:42 0:10:05 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-06-10T12:52:17.008428+0000 mon.smithi016 (mon.0) 113 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739808 2024-06-03 20:12:10 2024-06-10 12:33:08 2024-06-10 12:51:01 0:17:53 0:07:36 0:10:17 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/calico rook/master} 3
Failure Reason:

Command failed on smithi045 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

dead 7739809 2024-06-03 20:12:11 2024-06-10 12:33:09 2024-06-10 12:34:43 0:01:34 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi203

fail 7739810 2024-06-03 20:12:12 2024-06-10 12:33:39 2024-06-10 13:09:48 0:36:09 0:24:01 0:12:08 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi098 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bdb64002-2727-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7739811 2024-06-03 20:12:13 2024-06-10 12:34:30 2024-06-10 12:59:37 0:25:07 0:16:02 0:09:05 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-06-10T12:50:10.925559+0000 mon.smithi156 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739812 2024-06-03 20:12:14 2024-06-10 12:34:30 2024-06-10 12:55:30 0:21:00 0:10:28 0:10:32 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_rgw_multisite} 3
Failure Reason:

Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1e54d310-2728-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739813 2024-06-03 20:12:15 2024-06-10 12:34:30 2024-06-10 13:01:31 0:27:01 0:16:19 0:10:42 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-06-10T12:50:09.556848+0000 mon.smithi162 (mon.0) 120 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739814 2024-06-03 20:12:15 2024-06-10 12:34:31 2024-06-10 12:52:13 0:17:42 0:07:17 0:10:25 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_basic} 2
Failure Reason:

"2024-06-10T12:50:32.984391+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739815 2024-06-03 20:12:16 2024-06-10 12:34:41 2024-06-10 12:52:12 0:17:31 0:07:35 0:09:56 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7d9fc14-2727-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'

fail 7739816 2024-06-03 20:12:17 2024-06-10 12:35:22 2024-06-10 13:00:41 0:25:19 0:16:03 0:09:16 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-06-10T12:51:12.496774+0000 mon.smithi084 (mon.0) 122 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739817 2024-06-03 20:12:18 2024-06-10 12:35:42 2024-06-10 13:21:08 0:45:26 0:32:46 0:12:40 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-06-10T12:56:41.721983+0000 mon.smithi132 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739818 2024-06-03 20:12:19 2024-06-10 12:37:13 2024-06-10 13:02:56 0:25:43 0:13:41 0:12:02 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-06-10T12:59:42.373619+0000 mon.a (mon.0) 104 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739819 2024-06-03 20:12:20 2024-06-10 12:39:23 2024-06-10 13:08:19 0:28:56 0:14:52 0:14:04 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_set_mon_crush_locations} 3
Failure Reason:

"2024-06-10T13:04:03.646225+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739820 2024-06-03 20:12:21 2024-06-10 12:41:34 2024-06-10 13:13:15 0:31:41 0:21:18 0:10:23 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-06-10T13:01:49.764339+0000 mgr.smithi135.nypoey (mgr.14242) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7739821 2024-06-03 20:12:22 2024-06-10 12:42:14 2024-06-10 13:27:31 0:45:17 0:34:07 0:11:10 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

Command failed on smithi052 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd023ca6-2728-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7739822 2024-06-03 20:12:23 2024-06-10 12:44:35 2024-06-10 13:08:43 0:24:08 0:13:04 0:11:04 smithi main ubuntu 22.04 orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignore_mgr_down ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
Failure Reason:

"2024-06-10T13:05:53.193128+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739823 2024-06-03 20:12:24 2024-06-10 12:45:16 2024-06-10 13:04:12 0:18:56 0:09:11 0:09:45 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_orch_cli} 1
Failure Reason:

"2024-06-10T13:02:15.405801+0000 mon.a (mon.0) 107 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7739824 2024-06-03 20:12:25 2024-06-10 12:45:16 2024-06-10 13:09:15 0:23:59 0:14:14 0:09:45 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} 2
fail 7739825 2024-06-03 20:12:26 2024-06-10 12:45:36 2024-06-10 13:27:15 0:41:39 0:32:27 0:09:12 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-06-10T13:04:29.950860+0000 mon.smithi002 (mon.0) 109 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739826 2024-06-03 20:12:26 2024-06-10 12:45:47 2024-06-10 13:03:44 0:17:57 0:07:53 0:10:04 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-06-10T13:02:09.452968+0000 mon.a (mon.0) 105 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739827 2024-06-03 20:12:27 2024-06-10 12:46:47 2024-06-10 13:10:12 0:23:25 0:11:55 0:11:30 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_domain} 2
Failure Reason:

"2024-06-10T13:08:37.470664+0000 mon.a (mon.0) 100 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739828 2024-06-03 20:12:28 2024-06-10 12:48:18 2024-06-10 13:13:48 0:25:30 0:15:47 0:09:43 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-10T13:03:58.601461+0000 mon.smithi173 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739829 2024-06-03 20:12:29 2024-06-10 12:48:48 2024-06-10 13:10:39 0:21:51 0:11:41 0:10:10 smithi main ubuntu 22.04 orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} 1
Failure Reason:

"2024-06-10T13:08:06.432305+0000 mon.smithi142 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739830 2024-06-03 20:12:30 2024-06-10 12:49:09 2024-06-10 13:06:10 0:17:01 0:07:54 0:09:07 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-06-10T13:04:48.467708+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739831 2024-06-03 20:12:31 2024-06-10 12:49:19 2024-06-10 13:10:42 0:21:23 0:10:50 0:10:33 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
Failure Reason:

"2024-06-10T13:07:51.499390+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739832 2024-06-03 20:12:32 2024-06-10 12:51:12 2024-06-10 13:19:58 0:28:46 0:17:43 0:11:03 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-06-10T13:09:14.180179+0000 mon.smithi037 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739833 2024-06-03 20:12:33 2024-06-10 12:52:12 2024-06-10 13:16:35 0:24:23 0:13:45 0:10:38 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_ca_signed_key} 2
Failure Reason:

Command failed on smithi157 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fac45ddc-272a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739834 2024-06-03 20:12:34 2024-06-10 12:52:23 2024-06-10 13:35:10 0:42:47 0:32:28 0:10:19 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-10T13:12:21.017598+0000 mon.smithi012 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739835 2024-06-03 20:12:35 2024-06-10 12:53:33 2024-06-10 13:20:40 0:27:07 0:16:42 0:10:25 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-06-10T13:09:22.272510+0000 mon.smithi029 (mon.0) 120 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739836 2024-06-03 20:12:36 2024-06-10 12:53:34 2024-06-10 13:11:18 0:17:44 0:07:00 0:10:44 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_res_basic} 2
Failure Reason:

"2024-06-10T13:09:42.173228+0000 mon.a (mon.0) 113 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739837 2024-06-03 20:12:37 2024-06-10 12:54:14 2024-06-10 12:56:38 0:02:24 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi104

pass 7739838 2024-06-03 20:12:38 2024-06-10 12:55:35 2024-06-10 13:18:32 0:22:57 0:13:51 0:09:06 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm} 1
fail 7739839 2024-06-03 20:12:38 2024-06-10 12:55:35 2024-06-10 14:05:05 1:09:30 0:59:56 0:09:34 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f6cc225a-272a-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\''

fail 7739840 2024-06-03 20:12:39 2024-06-10 12:55:35 2024-06-10 13:14:35 0:19:00 0:07:24 0:11:36 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
Failure Reason:

"2024-06-10T13:12:53.406353+0000 mon.a (mon.0) 107 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739841 2024-06-03 20:12:40 2024-06-10 12:56:36 2024-06-10 13:16:54 0:20:18 0:08:49 0:11:29 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rados_python} 2
Failure Reason:

"2024-06-10T13:14:15.585174+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739842 2024-06-03 20:12:41 2024-06-10 14:07:44 2024-06-10 14:08:48 0:01:04 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi189

dead 7739843 2024-06-03 20:12:42 2024-06-10 14:07:44 2024-06-10 14:12:49 0:05:05 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli_mon} 5
Failure Reason:

Error reimaging machines: Failed to power on smithi165

fail 7739844 2024-06-03 20:12:43 2024-06-10 14:11:45 2024-06-10 14:39:11 0:27:26 0:16:27 0:10:59 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-06-10T14:29:10.259057+0000 mon.smithi037 (mon.0) 120 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739845 2024-06-03 20:12:44 2024-06-10 14:14:06 2024-06-10 14:34:05 0:19:59 0:09:48 0:10:11 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-06-10T14:32:19.866388+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739846 2024-06-03 20:12:45 2024-06-10 14:17:59 2024-06-10 14:20:53 0:02:54 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi050

pass 7739847 2024-06-03 20:12:46 2024-06-10 14:19:50 2024-06-10 14:34:43 0:14:53 0:06:02 0:08:51 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} 1
fail 7739848 2024-06-03 20:12:47 2024-06-10 14:19:50 2024-06-10 14:46:30 0:26:40 0:16:10 0:10:30 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-06-10T14:36:38.962123+0000 mon.smithi079 (mon.0) 120 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739849 2024-06-03 20:12:48 2024-06-10 14:21:30 2024-06-10 14:22:54 0:01:24 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_res_dom} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi121

fail 7739850 2024-06-03 20:12:49 2024-06-10 14:21:51 2024-06-10 14:46:54 0:25:03 0:15:51 0:09:12 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-06-10T14:37:12.240834+0000 mon.smithi114 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739851 2024-06-03 20:12:50 2024-06-10 14:21:51 2024-06-10 14:40:18 0:18:27 0:08:00 0:10:27 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

Command failed on smithi040 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid da2aa246-2736-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'

fail 7739852 2024-06-03 20:12:51 2024-06-10 14:23:02 2024-06-10 14:48:08 0:25:06 0:15:51 0:09:15 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

"2024-06-10T14:37:51.847876+0000 mon.smithi089 (mon.0) 120 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739853 2024-06-03 20:12:52 2024-06-10 14:23:22 2024-06-10 14:47:01 0:23:39 0:14:05 0:09:34 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} 2
Failure Reason:

"2024-06-10T14:44:26.067868+0000 mon.a (mon.0) 105 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739854 2024-06-03 20:12:53 2024-06-10 14:23:33 2024-06-10 15:08:31 0:44:58 0:32:06 0:12:52 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi121 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 749a7c02-2737-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

fail 7739855 2024-06-03 20:12:54 2024-06-10 14:23:33 2024-06-10 14:41:06 0:17:33 0:07:55 0:09:38 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/flannel rook/1.7.2} 1
Failure Reason:

Command failed on smithi143 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7739856 2024-06-03 20:12:55 2024-06-10 14:23:34 2024-06-10 14:45:10 0:21:36 0:11:59 0:09:37 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_adoption} 1
fail 7739857 2024-06-03 20:12:56 2024-06-10 14:23:34 2024-06-10 14:50:30 0:26:56 0:16:41 0:10:15 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

"2024-06-10T14:38:59.165633+0000 mon.smithi145 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739858 2024-06-03 20:12:57 2024-06-10 14:23:35 2024-06-10 14:44:44 0:21:09 0:09:46 0:11:23 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} 2
Failure Reason:

"2024-06-10T14:43:14.181745+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739859 2024-06-03 20:12:58 2024-06-10 14:23:35 2024-06-10 15:05:07 0:41:32 0:32:35 0:08:57 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-06-10T14:42:40.960462+0000 mon.smithi078 (mon.0) 115 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739860 2024-06-03 20:12:59 2024-06-10 14:23:35 2024-06-10 14:24:49 0:01:14 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi151

fail 7739861 2024-06-03 20:13:00 2024-06-10 14:23:46 2024-06-10 14:51:01 0:27:15 0:15:53 0:11:22 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-06-10T14:39:42.048709+0000 mon.smithi154 (mon.0) 122 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739862 2024-06-03 20:13:01 2024-06-10 14:23:46 2024-06-10 14:28:40 0:04:54 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi113

fail 7739863 2024-06-03 20:13:02 2024-06-10 14:27:37 2024-06-10 14:50:00 0:22:23 0:11:07 0:11:16 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} 3
Failure Reason:

Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b58eb7e-2738-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739864 2024-06-03 20:13:03 2024-06-10 14:28:47 2024-06-10 14:51:00 0:22:13 0:12:01 0:10:12 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

Command failed on smithi038 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20b58144-2738-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'

fail 7739865 2024-06-03 20:13:04 2024-06-10 14:29:18 2024-06-10 14:58:14 0:28:56 0:16:52 0:12:04 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-06-10T14:47:10.843553+0000 mon.smithi012 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739866 2024-06-03 20:13:05 2024-06-10 14:31:18 2024-06-10 15:24:15 0:52:57 0:38:50 0:14:07 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

Command failed on smithi157 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f4d5aa6-2738-11ef-bc9b-c7b262605968 -e sha1=65533a907fc9add1863609703fdbbea67993c8ef -- bash -c \'ceph versions | jq -e \'"\'"\'.osd | length == 2\'"\'"\'\''

dead 7739867 2024-06-03 20:13:06 2024-06-10 14:34:09 2024-06-10 14:35:33 0:01:24 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi053

fail 7739868 2024-06-03 20:13:07 2024-06-10 14:34:30 2024-06-10 14:56:33 0:22:03 0:12:04 0:09:59 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} 2
Failure Reason:

"2024-06-10T14:54:54.379556+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739869 2024-06-03 20:13:08 2024-06-10 14:34:50 2024-06-10 15:21:19 0:46:29 0:33:37 0:12:52 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-06-10T14:57:52.456031+0000 mon.smithi016 (mon.0) 109 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739870 2024-06-03 20:13:09 2024-06-10 14:35:31 2024-06-10 14:36:55 0:01:24 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
Failure Reason:

Error reimaging machines: Failed to power on smithi116

fail 7739871 2024-06-03 20:13:10 2024-06-10 14:35:51 2024-06-10 15:10:25 0:34:34 0:20:51 0:13:43 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1718031517.3345072 mgr.smithi151.vjemfo (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7739872 2024-06-03 20:13:11 2024-06-10 14:39:12 2024-06-10 14:58:22 0:19:10 0:09:29 0:09:41 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_cephadm_timeout} 1
Failure Reason:

Command failed on smithi188 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5b04712e-2739-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

pass 7739873 2024-06-03 20:13:12 2024-06-10 14:39:12 2024-06-10 15:03:23 0:24:11 0:14:13 0:09:58 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} 2
fail 7739874 2024-06-03 20:13:13 2024-06-10 14:39:13 2024-06-10 15:04:06 0:24:53 0:16:10 0:08:43 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-06-10T14:54:15.364675+0000 mon.smithi002 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739875 2024-06-03 20:13:14 2024-06-10 14:39:13 2024-06-10 14:56:04 0:16:51 0:07:03 0:09:48 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} 1
Failure Reason:

"2024-06-10T14:53:38.815552+0000 mon.smithi116 (mon.0) 124 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739876 2024-06-03 20:13:15 2024-06-10 14:39:13 2024-06-10 14:59:44 0:20:31 0:09:23 0:11:08 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

Command failed on smithi040 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5a1d006e-2739-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'

fail 7739877 2024-06-03 20:13:16 2024-06-10 14:40:24 2024-06-10 15:07:09 0:26:45 0:16:37 0:10:08 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-06-10T14:55:40.298872+0000 mon.smithi053 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739878 2024-06-03 20:13:17 2024-06-10 14:40:24 2024-06-10 15:07:23 0:26:59 0:16:23 0:10:36 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-06-10T14:55:44.355064+0000 mon.smithi001 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739879 2024-06-03 20:13:17 2024-06-10 14:40:45 2024-06-10 15:00:47 0:20:02 0:06:47 0:13:15 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_basic} 2
Failure Reason:

"2024-06-10T14:59:12.580401+0000 mon.a (mon.0) 109 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739880 2024-06-03 20:13:18 2024-06-10 14:43:45 2024-06-10 15:08:17 0:24:32 0:13:45 0:10:47 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
Failure Reason:

"2024-06-10T15:06:08.606888+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739881 2024-06-03 20:13:19 2024-06-10 14:44:36 2024-06-10 14:45:50 0:01:14 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi050

fail 7739882 2024-06-03 20:13:20 2024-06-10 14:44:46 2024-06-10 15:11:03 0:26:17 0:14:36 0:11:41 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
Failure Reason:

Command failed on smithi044 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9438068-273a-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739883 2024-06-03 20:13:21 2024-06-10 14:54:41 2024-06-10 15:38:02 0:43:21 0:31:48 0:11:33 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-06-10T15:14:39.814178+0000 mon.smithi045 (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739884 2024-06-03 20:13:22 2024-06-10 14:54:42 2024-06-10 15:19:44 0:25:02 0:16:04 0:08:58 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-06-10T15:09:34.345042+0000 mon.smithi046 (mon.0) 120 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739885 2024-06-03 20:13:23 2024-06-10 14:54:42 2024-06-10 15:13:34 0:18:52 0:07:44 0:11:08 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
Failure Reason:

"2024-06-10T15:11:27.243483+0000 mon.a (mon.0) 110 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739886 2024-06-03 20:13:24 2024-06-10 14:56:43 2024-06-10 15:19:36 0:22:53 0:10:29 0:12:24 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} 3
Failure Reason:

"2024-06-10T15:16:34.385619+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739887 2024-06-03 20:13:25 2024-06-10 14:58:24 2024-06-10 15:30:03 0:31:39 0:20:16 0:11:23 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-06-10T15:18:00.697945+0000 mgr.smithi152.feassg (mgr.14234) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7739888 2024-06-03 20:13:26 2024-06-10 14:58:44 2024-06-10 15:17:43 0:18:59 0:08:58 0:10:01 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli} 1
Failure Reason:

"2024-06-10T15:15:22.553721+0000 mon.a (mon.0) 107 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739889 2024-06-03 20:13:27 2024-06-10 14:58:44 2024-06-10 15:25:58 0:27:14 0:16:17 0:10:57 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-06-10T15:14:27.554376+0000 mon.smithi097 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739890 2024-06-03 20:13:28 2024-06-10 14:59:05 2024-06-10 15:15:49 0:16:44 0:07:40 0:09:04 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_domain} 2
Failure Reason:

"2024-06-10T15:14:36.814410+0000 mon.a (mon.0) 113 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739891 2024-06-03 20:13:29 2024-06-10 14:59:05 2024-06-10 15:24:41 0:25:36 0:16:20 0:09:16 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-06-10T15:14:58.894420+0000 mon.smithi040 (mon.0) 120 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739892 2024-06-03 20:13:30 2024-06-10 14:59:46 2024-06-10 15:44:52 0:45:06 0:31:42 0:13:24 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-06-10T15:21:44.207968+0000 mon.smithi143 (mon.0) 116 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

dead 7739893 2024-06-03 20:13:31 2024-06-10 15:00:56 2024-06-11 03:13:10 12:12:14 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

hit max job timeout

fail 7739894 2024-06-03 20:13:32 2024-06-10 15:03:27 2024-06-10 15:25:08 0:21:41 0:10:53 0:10:48 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
Failure Reason:

Command failed on smithi002 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f125fc-273c-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4'

fail 7739895 2024-06-03 20:13:33 2024-06-10 15:04:07 2024-06-10 15:24:08 0:20:01 0:10:31 0:09:30 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-06-10T15:22:53.709599+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739896 2024-06-03 20:13:34 2024-06-10 15:05:08 2024-06-10 15:32:04 0:26:56 0:16:24 0:10:32 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-06-10T15:22:27.113443+0000 mon.smithi053 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739897 2024-06-03 20:13:35 2024-06-10 15:07:19 2024-06-10 15:25:37 0:18:18 0:08:12 0:10:06 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-06-10T15:23:39.164934+0000 mon.a (mon.0) 108 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739898 2024-06-03 20:13:36 2024-06-10 15:08:19 2024-06-10 15:34:27 0:26:08 0:14:40 0:11:28 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rados_python} 2
Failure Reason:

"2024-06-10T15:30:40.678178+0000 mon.a (mon.0) 100 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739899 2024-06-03 20:13:37 2024-06-10 15:08:40 2024-06-10 15:50:31 0:41:51 0:31:27 0:10:24 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-06-10T15:28:52.663334+0000 mon.smithi144 (mon.0) 116 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739900 2024-06-03 20:13:37 2024-06-10 15:08:50 2024-06-10 15:34:16 0:25:26 0:12:47 0:12:39 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_basic} 2
Failure Reason:

"2024-06-10T15:31:24.119160+0000 mon.a (mon.0) 100 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739901 2024-06-03 20:13:38 2024-06-10 15:10:11 2024-06-10 15:37:13 0:27:02 0:17:15 0:09:47 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-06-10T15:25:51.071109+0000 mon.smithi044 (mon.0) 118 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739902 2024-06-03 20:13:39 2024-06-10 15:10:11 2024-06-10 15:30:06 0:19:55 0:07:31 0:12:24 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/host rook/master} 3
Failure Reason:

Command failed on smithi088 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7739903 2024-06-03 20:13:40 2024-06-10 15:10:11 2024-06-10 15:29:21 0:19:10 0:09:37 0:09:33 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} 2
Failure Reason:

"2024-06-10T15:27:16.925212+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739904 2024-06-03 20:13:41 2024-06-10 15:10:22 2024-06-10 15:40:27 0:30:05 0:20:29 0:09:36 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1718033401.0792239 mgr.smithi018.kzdccy (mgr.14234) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7739905 2024-06-03 20:13:42 2024-06-10 15:10:42 2024-06-10 15:43:21 0:32:39 0:17:19 0:15:20 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_orch_cli_mon} 5
Failure Reason:

"2024-06-10T15:37:53.114669+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

fail 7739906 2024-06-03 20:13:45 2024-06-10 15:19:38 2024-06-10 15:36:40 0:17:02 0:07:45 0:09:17 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:65533a907fc9add1863609703fdbbea67993c8ef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bb2c5648-273e-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'

fail 7739907 2024-06-03 20:13:46 2024-06-10 15:19:49 2024-06-10 15:44:48 0:24:59 0:16:03 0:08:56 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-06-10T15:35:14.158694+0000 mon.smithi026 (mon.0) 122 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log