Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7706082 2024-05-14 16:07:50 2024-05-14 22:28:03 2024-05-14 23:04:21 0:36:18 0:23:20 0:12:58 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-05-14T23:00:00.000157+0000 mon.smithi044 (mon.0) 812 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

fail 7706083 2024-05-14 16:07:51 2024-05-14 22:30:44 2024-05-14 23:00:22 0:29:38 0:19:27 0:10:11 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_timeout} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm_timeout.py) on smithi187 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm_timeout.py'

fail 7706084 2024-05-14 16:07:52 2024-05-14 22:30:44 2024-05-14 22:49:42 0:18:58 0:07:05 0:11:53 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi097 with status 5: 'sudo systemctl stop ceph-eabbf39c-1243-11ef-bc98-c7b262605968@mon.smithi097'

fail 7706085 2024-05-14 16:07:54 2024-05-14 22:32:35 2024-05-14 22:47:51 0:15:16 0:05:00 0:10:16 smithi main centos 9.stream orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

Command failed on smithi003 with status 5: 'sudo systemctl stop ceph-adffcca8-1243-11ef-bc98-c7b262605968@mon.smithi003'

pass 7706086 2024-05-14 16:07:55 2024-05-14 22:32:45 2024-05-14 23:26:44 0:53:59 0:45:01 0:08:58 smithi main ubuntu 22.04 orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
pass 7706087 2024-05-14 16:07:56 2024-05-14 22:33:06 2024-05-14 22:54:14 0:21:08 0:11:30 0:09:38 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli} 2
fail 7706088 2024-05-14 16:07:57 2024-05-14 22:33:06 2024-05-14 23:04:33 0:31:27 0:21:40 0:09:47 smithi main centos 9.stream orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

Command failed on smithi045 with status 1: 'CEPH_REF=master CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/virtualenv/bin/cram -v -- /home/ubuntu/cephtest/archive/cram.client.0/*.t'

pass 7706089 2024-05-14 16:07:58 2024-05-14 22:33:46 2024-05-14 22:59:09 0:25:23 0:14:58 0:10:25 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
fail 7706090 2024-05-14 16:07:59 2024-05-14 22:36:07 2024-05-14 23:04:27 0:28:20 0:16:57 0:11:23 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-05-14T22:53:10.616328+0000 mon.smithi100 (mon.0) 271 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recently crashed" in cluster log

fail 7706091 2024-05-14 16:08:01 2024-05-14 22:37:28 2024-05-14 23:04:06 0:26:38 0:15:37 0:11:01 smithi main centos 9.stream orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} 1
Failure Reason:

timeout expired in wait_until_healthy

fail 7706092 2024-05-14 16:08:02 2024-05-14 22:39:08 2024-05-14 23:00:48 0:21:40 0:10:51 0:10:49 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-05-14T22:55:53.012461+0000 mon.a (mon.0) 209 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7706093 2024-05-14 16:08:03 2024-05-14 22:39:39 2024-05-14 23:21:24 0:41:45 0:30:22 0:11:23 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
fail 7706094 2024-05-14 16:08:04 2024-05-14 22:40:29 2024-05-14 23:46:15 1:05:46 0:52:18 0:13:28 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-05-14T23:10:00.000236+0000 mon.a (mon.0) 1277 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled" in cluster log

pass 7706095 2024-05-14 16:08:05 2024-05-14 22:41:40 2024-05-14 23:04:57 0:23:17 0:13:13 0:10:04 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_extra_daemon_features} 2
fail 7706096 2024-05-14 16:08:06 2024-05-14 22:41:40 2024-05-14 23:09:52 0:28:12 0:17:14 0:10:58 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-05-14T22:59:31.677363+0000 mon.smithi112 (mon.0) 278 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

fail 7706097 2024-05-14 16:08:07 2024-05-14 22:42:31 2024-05-14 23:18:05 0:35:34 0:23:08 0:12:26 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-05-14T23:10:00.000238+0000 mon.smithi107 (mon.0) 761 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

fail 7706098 2024-05-14 16:08:08 2024-05-14 22:44:21 2024-05-14 23:16:14 0:31:53 0:22:35 0:09:18 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-05-14T23:10:00.000221+0000 mon.smithi081 (mon.0) 767 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

pass 7706099 2024-05-14 16:08:10 2024-05-14 22:44:32 2024-05-14 23:10:00 0:25:28 0:16:16 0:09:12 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain} 3
fail 7706100 2024-05-14 16:08:11 2024-05-14 22:44:42 2024-05-14 23:04:05 0:19:23 0:08:14 0:11:09 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi086 with status 5: 'sudo systemctl stop ceph-ba7d7212-1245-11ef-bc98-c7b262605968@mon.smithi086'

fail 7706101 2024-05-14 16:08:12 2024-05-14 22:44:53 2024-05-14 23:13:06 0:28:13 0:17:18 0:10:55 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-05-14T23:02:49.452594+0000 mon.smithi007 (mon.0) 271 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recently crashed" in cluster log

fail 7706102 2024-05-14 16:08:13 2024-05-14 22:46:13 2024-05-14 23:02:41 0:16:28 0:05:23 0:11:05 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

Command failed on smithi139 with status 5: 'sudo systemctl stop ceph-b92899aa-1245-11ef-bc98-c7b262605968@mon.a'

pass 7706103 2024-05-14 16:08:14 2024-05-14 22:47:34 2024-05-14 23:12:02 0:24:28 0:13:24 0:11:04 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
fail 7706104 2024-05-14 16:08:15 2024-05-14 22:49:04 2024-05-14 23:16:10 0:27:06 0:17:24 0:09:42 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-05-14T23:05:17.146821+0000 mon.smithi003 (mon.0) 278 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

fail 7706105 2024-05-14 16:08:16 2024-05-14 22:49:15 2024-05-14 23:34:12 0:44:57 0:35:00 0:09:57 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-05-14T23:20:00.000173+0000 mon.a (mon.0) 1579 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

fail 7706106 2024-05-14 16:08:17 2024-05-14 22:49:15 2024-05-14 23:24:24 0:35:09 0:22:42 0:12:27 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} 2
Failure Reason:

"2024-05-14T23:20:00.000168+0000 mon.a (mon.0) 1362 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log

fail 7706107 2024-05-14 16:08:18 2024-05-14 22:51:06 2024-05-14 23:20:42 0:29:36 0:17:04 0:12:32 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-05-14T23:10:07.668964+0000 mon.smithi087 (mon.0) 271 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

fail 7706108 2024-05-14 16:08:19 2024-05-14 22:53:16 2024-05-14 23:26:05 0:32:49 0:22:11 0:10:38 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-05-14T23:20:00.000216+0000 mon.smithi143 (mon.0) 763 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

pass 7706109 2024-05-14 16:08:20 2024-05-14 22:54:17 2024-05-14 23:28:11 0:33:54 0:19:36 0:14:18 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
pass 7706110 2024-05-14 16:08:21 2024-05-14 22:59:08 2024-05-14 23:34:53 0:35:45 0:25:11 0:10:34 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
fail 7706111 2024-05-14 16:08:22 2024-05-14 22:59:18 2024-05-14 23:28:56 0:29:38 0:17:29 0:12:09 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-05-14T23:18:57.014160+0000 mon.smithi103 (mon.0) 269 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recently crashed" in cluster log

fail 7706112 2024-05-14 16:08:23 2024-05-14 23:01:19 2024-05-14 23:27:50 0:26:31 0:12:11 0:14:20 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-05-14T23:21:37.335630+0000 mon.a (mon.0) 266 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7706113 2024-05-14 16:08:24 2024-05-14 23:04:50 2024-05-14 23:24:38 0:19:48 0:07:35 0:12:13 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi102 with status 5: 'sudo systemctl stop ceph-d2195f8c-1248-11ef-bc98-c7b262605968@mon.smithi102'

pass 7706114 2024-05-14 16:08:25 2024-05-14 23:04:50 2024-05-14 23:27:36 0:22:46 0:12:55 0:09:51 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_orch_cli} 1
fail 7706115 2024-05-14 16:08:26 2024-05-14 23:04:50 2024-05-14 23:33:47 0:28:57 0:18:36 0:10:21 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

"2024-05-14T23:22:27.642341+0000 mon.smithi067 (mon.0) 279 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

fail 7706116 2024-05-14 16:08:27 2024-05-14 23:04:51 2024-05-14 23:55:35 0:50:44 0:38:45 0:11:59 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_orch_cli_mon} 5
Failure Reason:

"2024-05-14T23:31:14.095810+0000 mon.a (mon.0) 522 : cluster 3 [WRN] CEPHADM_STRAY_DAEMON: 2 stray daemon(s) not managed by cephadm" in cluster log

fail 7706117 2024-05-14 16:08:28 2024-05-14 23:04:51 2024-05-14 23:31:39 0:26:48 0:17:46 0:09:02 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-05-14T23:21:58.633638+0000 mon.smithi136 (mon.0) 271 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recently crashed" in cluster log

fail 7706118 2024-05-14 16:08:29 2024-05-14 23:04:51 2024-05-14 23:36:47 0:31:56 0:22:03 0:09:53 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-05-14T23:30:00.000211+0000 mon.smithi162 (mon.0) 757 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

fail 7706119 2024-05-14 16:08:30 2024-05-14 23:04:52 2024-05-15 00:12:44 1:07:52 0:57:58 0:09:54 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

"2024-05-14T23:26:39.059428+0000 mon.a (mon.0) 198 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7706120 2024-05-14 16:08:32 2024-05-14 23:04:52 2024-05-14 23:55:26 0:50:34 0:38:43 0:11:51 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
fail 7706121 2024-05-14 16:08:33 2024-05-14 23:05:03 2024-05-14 23:32:31 0:27:28 0:17:57 0:09:31 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
Failure Reason:

"2024-05-14T23:22:36.415559+0000 mon.smithi053 (mon.0) 270 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

pass 7706122 2024-05-14 16:08:34 2024-05-14 23:05:03 2024-05-14 23:32:50 0:27:47 0:14:07 0:13:40 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} 3
fail 7706123 2024-05-14 16:08:35 2024-05-14 23:07:24 2024-05-14 23:24:37 0:17:13 0:07:21 0:09:52 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi183 with status 5: 'sudo systemctl stop ceph-e18c591a-1248-11ef-bc98-c7b262605968@mon.smithi183'

fail 7706124 2024-05-14 16:08:36 2024-05-14 23:07:44 2024-05-14 23:24:13 0:16:29 0:06:44 0:09:45 smithi main ubuntu 20.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

Command failed on smithi117 with status 5: 'sudo systemctl stop ceph-e12a9af4-1248-11ef-bc98-c7b262605968@mon.a'

fail 7706125 2024-05-14 16:08:37 2024-05-14 23:08:15 2024-05-14 23:35:49 0:27:34 0:17:40 0:09:54 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-05-14T23:25:20.571699+0000 mon.smithi113 (mon.0) 278 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

pass 7706126 2024-05-14 16:08:38 2024-05-14 23:08:25 2024-05-14 23:35:24 0:26:59 0:15:51 0:11:08 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
fail 7706127 2024-05-14 16:08:39 2024-05-14 23:10:05 2024-05-14 23:45:33 0:35:28 0:23:29 0:11:59 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-05-14T23:40:00.000214+0000 mon.smithi165 (mon.0) 784 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

fail 7706128 2024-05-14 16:08:40 2024-05-14 23:11:46 2024-05-14 23:45:46 0:34:00 0:22:41 0:11:19 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-05-14T23:40:00.000165+0000 mon.smithi005 (mon.0) 780 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

pass 7706129 2024-05-14 16:08:41 2024-05-14 23:12:06 2024-05-14 23:35:15 0:23:09 0:11:47 0:11:22 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli} 2
pass 7706130 2024-05-14 16:08:42 2024-05-14 23:14:07 2024-05-14 23:39:44 0:25:37 0:15:40 0:09:57 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
fail 7706131 2024-05-14 16:08:43 2024-05-14 23:14:48 2024-05-14 23:42:37 0:27:49 0:17:45 0:10:04 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
Failure Reason:

"2024-05-14T23:32:20.197874+0000 mon.smithi028 (mon.0) 279 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

fail 7706132 2024-05-14 16:08:44 2024-05-14 23:15:08 2024-05-14 23:41:52 0:26:44 0:15:51 0:10:53 smithi main centos 9.stream orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
Failure Reason:

timeout expired in wait_until_healthy

pass 7706133 2024-05-14 16:08:45 2024-05-14 23:16:08 2024-05-14 23:41:17 0:25:09 0:12:03 0:13:06 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
fail 7706134 2024-05-14 16:08:46 2024-05-14 23:20:29 2024-05-15 00:25:29 1:05:00 0:55:37 0:09:23 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-05-15T00:00:00.001744+0000 mon.a (mon.0) 1601 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

fail 7706135 2024-05-14 16:08:47 2024-05-14 23:20:30 2024-05-15 00:31:56 1:11:26 1:01:37 0:09:49 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

"2024-05-14T23:41:22.387951+0000 mon.a (mon.0) 201 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7706136 2024-05-14 16:08:48 2024-05-14 23:20:30 2024-05-14 23:42:00 0:21:30 0:12:38 0:08:52 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_adoption} 1
fail 7706137 2024-05-14 16:08:49 2024-05-14 23:20:30 2024-05-14 23:49:46 0:29:16 0:17:25 0:11:51 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-05-14T23:38:09.784523+0000 mon.smithi026 (mon.0) 273 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

fail 7706138 2024-05-14 16:08:50 2024-05-14 23:20:31 2024-05-14 23:39:45 0:19:14 0:07:42 0:11:32 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi088 with status 5: 'sudo systemctl stop ceph-cbe793ac-124a-11ef-bc98-c7b262605968@mon.smithi088'

fail 7706139 2024-05-14 16:08:51 2024-05-14 23:20:31 2024-05-14 23:41:32 0:21:01 0:11:48 0:09:13 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} 2
Failure Reason:

"2024-05-14T23:40:00.000202+0000 mon.a (mon.0) 398 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)" in cluster log

fail 7706140 2024-05-14 16:08:52 2024-05-14 23:20:32 2024-05-14 23:54:16 0:33:44 0:22:37 0:11:07 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-05-14T23:50:00.000204+0000 mon.smithi001 (mon.0) 838 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

fail 7706141 2024-05-14 16:08:53 2024-05-14 23:20:42 2024-05-14 23:53:52 0:33:10 0:17:21 0:15:49 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-05-14T23:43:43.875393+0000 mon.smithi064 (mon.0) 278 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed" in cluster log

fail 7706142 2024-05-14 16:08:54 2024-05-14 23:26:53 2024-05-14 23:50:41 0:23:48 0:13:10 0:10:38 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi139 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7706143 2024-05-14 16:08:54 2024-05-14 23:27:44 2024-05-14 23:55:17 0:27:33 0:17:31 0:10:02 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-05-14T23:44:23.864532+0000 mon.smithi012 (mon.0) 274 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed" in cluster log

fail 7706144 2024-05-14 16:08:56 2024-05-14 23:28:14 2024-05-14 23:55:28 0:27:14 0:17:45 0:09:29 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-05-14T23:45:47.794579+0000 mon.smithi059 (mon.0) 276 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

pass 7706145 2024-05-14 16:08:57 2024-05-14 23:28:25 2024-05-15 00:10:55 0:42:30 0:31:44 0:10:46 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
fail 7706146 2024-05-14 16:08:58 2024-05-14 23:29:15 2024-05-15 00:06:25 0:37:10 0:23:32 0:13:38 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} 2
Failure Reason:

"2024-05-14T23:51:30.386183+0000 mon.a (mon.0) 192 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7706147 2024-05-14 16:08:58 2024-05-14 23:32:57 2024-05-14 23:52:39 0:19:42 0:09:34 0:10:08 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} 1
Failure Reason:

Command failed (workunit test cephadm/test_repos.sh) on smithi149 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1fb5d10608dccee73edd6ef8a2c7b804a097c324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

fail 7706148 2024-05-14 16:09:00 2024-05-14 23:32:57 2024-05-14 23:51:54 0:18:57 0:09:05 0:09:52 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi122 with status 5: 'sudo systemctl stop ceph-6ef62d78-124c-11ef-bc98-c7b262605968@mon.smithi122'

fail 7706149 2024-05-14 16:09:01 2024-05-14 23:32:57 2024-05-14 23:48:05 0:15:08 0:05:23 0:09:45 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi116 with status 5: 'sudo systemctl stop ceph-19e5891e-124c-11ef-bc98-c7b262605968@mon.a'

fail 7706150 2024-05-14 16:09:02 2024-05-14 23:33:08 2024-05-15 00:07:19 0:34:11 0:23:15 0:10:56 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-05-15T00:00:00.000188+0000 mon.smithi137 (mon.0) 770 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

pass 7706151 2024-05-14 16:09:03 2024-05-14 23:33:38 2024-05-14 23:59:56 0:26:18 0:15:20 0:10:58 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_timeout} 1
fail 7706152 2024-05-14 16:09:04 2024-05-14 23:34:59 2024-05-15 00:02:21 0:27:22 0:18:07 0:09:15 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-05-14T23:52:06.318623+0000 mon.smithi002 (mon.0) 272 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

pass 7706153 2024-05-14 16:09:05 2024-05-14 23:35:20 2024-05-15 00:00:49 0:25:29 0:16:22 0:09:07 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
fail 7706154 2024-05-14 16:09:06 2024-05-14 23:35:30 2024-05-15 00:04:40 0:29:10 0:19:31 0:09:39 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-05-14T23:54:03.592951+0000 mon.smithi033 (mon.0) 276 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

fail 7706155 2024-05-14 16:09:07 2024-05-14 23:35:30 2024-05-14 23:59:15 0:23:45 0:12:49 0:10:56 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-05-14T23:53:32.449697+0000 mon.a (mon.0) 208 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7706156 2024-05-14 16:09:08 2024-05-14 23:36:21 2024-05-15 00:01:55 0:25:34 0:14:15 0:11:19 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} 2
fail 7706157 2024-05-14 16:09:09 2024-05-14 23:36:22 2024-05-15 00:04:12 0:27:50 0:18:13 0:09:37 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-05-14T23:54:27.223274+0000 mon.smithi103 (mon.0) 271 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 2 mgr modules have recently crashed" in cluster log

fail 7706158 2024-05-14 16:09:10 2024-05-14 23:36:22 2024-05-14 23:53:10 0:16:48 0:07:35 0:09:13 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi117 with status 5: 'sudo systemctl stop ceph-eb02a252-124c-11ef-bc98-c7b262605968@mon.smithi117'

fail 7706159 2024-05-14 16:09:11 2024-05-14 23:36:22 2024-05-15 00:10:58 0:34:36 0:22:06 0:12:30 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-05-15T00:00:00.000180+0000 mon.smithi160 (mon.0) 661 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

dead 7706160 2024-05-14 16:09:12 2024-05-14 23:36:23 2024-05-14 23:41:38 0:05:15 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
Failure Reason:

Error reimaging machines: Expected smithi155's OS to be ubuntu 22.04 but found centos 9

dead 7706161 2024-05-14 16:09:13 2024-05-14 23:36:23 2024-05-14 23:43:35 0:07:12 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys} 2
Failure Reason:

SSH connection to smithi155 was lost: 'sudo yum install -y kernel'

pass 7706162 2024-05-14 16:09:14 2024-05-14 23:36:24 2024-05-15 00:11:03 0:34:39 0:23:03 0:11:36 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} 3
fail 7706163 2024-05-14 16:09:15 2024-05-14 23:36:24 2024-05-15 00:07:57 0:31:33 0:22:10 0:09:23 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-05-15T00:00:00.000213+0000 mon.smithi086 (mon.0) 744 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log

fail 7706164 2024-05-14 16:09:16 2024-05-14 23:36:24 2024-05-15 00:05:41 0:29:17 0:18:38 0:10:39 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-05-14T23:54:15.749132+0000 mon.smithi053 (mon.0) 279 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed" in cluster log

pass 7706165 2024-05-14 16:09:17 2024-05-14 23:36:25 2024-05-14 23:59:09 0:22:44 0:14:03 0:08:41 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
fail 7706166 2024-05-14 16:09:18 2024-05-14 23:36:25 2024-05-15 00:03:08 0:26:43 0:18:28 0:08:15 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

"2024-05-14T23:53:19.922522+0000 mon.smithi073 (mon.0) 269 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 1 mgr modules have recently crashed" in cluster log

fail 7706167 2024-05-14 16:09:19 2024-05-14 23:36:26 2024-05-15 00:08:50 0:32:24 0:22:09 0:10:15 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

"2024-05-15T00:00:00.000254+0000 mon.smithi162 (mon.0) 677 : cluster 3 [WRN] RECENT_MGR_MODULE_CRASH: 4 mgr modules have recently crashed" in cluster log