Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7581159 2024-03-04 15:07:44 2024-03-04 15:08:52 2024-03-04 15:33:41 0:24:49 0:14:47 0:10:02 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7581160 2024-03-04 15:07:45 2024-03-04 15:08:53 2024-03-04 15:52:12 0:43:19 0:28:47 0:14:32 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
fail 7581161 2024-03-04 15:07:46 2024-03-04 15:08:53 2024-03-04 16:40:48 1:31:55 1:17:58 0:13:57 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-04T15:41:36.470109+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7581162 2024-03-04 15:07:47 2024-03-04 15:08:54 2024-03-04 15:54:14 0:45:20 0:31:34 0:13:46 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
fail 7581163 2024-03-04 15:07:48 2024-03-04 15:08:54 2024-03-04 16:16:00 1:07:06 0:54:55 0:12:11 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7581164 2024-03-04 15:07:49 2024-03-04 15:08:54 2024-03-04 15:59:55 0:51:01 0:40:12 0:10:49 smithi main centos 9.stream orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
pass 7581165 2024-03-04 15:07:50 2024-03-04 15:08:55 2024-03-04 16:16:45 1:07:50 0:54:36 0:13:14 smithi main ubuntu 22.04 orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
pass 7581166 2024-03-04 15:07:51 2024-03-04 15:08:55 2024-03-04 15:36:50 0:27:55 0:14:29 0:13:26 smithi main ubuntu 22.04 orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} 2
pass 7581167 2024-03-04 15:07:52 2024-03-04 15:08:56 2024-03-04 15:57:52 0:48:56 0:34:03 0:14:53 smithi main centos 9.stream orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
pass 7581168 2024-03-04 15:07:53 2024-03-04 15:08:56 2024-03-04 15:47:02 0:38:06 0:23:19 0:14:47 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
pass 7581169 2024-03-04 15:07:54 2024-03-04 15:08:56 2024-03-04 15:39:05 0:30:09 0:19:42 0:10:27 smithi main ubuntu 22.04 orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} 1
fail 7581170 2024-03-04 15:07:55 2024-03-04 15:08:57 2024-03-04 15:30:46 0:21:49 0:11:43 0:10:06 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-04T15:26:19.907133+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

dead 7581171 2024-03-04 15:07:56 2024-03-04 15:08:57 2024-03-04 15:17:20 0:08:23 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} 5
Failure Reason:

Error reimaging machines: list index out of range

dead 7581172 2024-03-04 15:07:57 2024-03-04 15:08:58 2024-03-04 15:09:00 0:00:02 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
Failure Reason:

Error reimaging machines: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/268/task

fail 7581173 2024-03-04 15:07:57 2024-03-04 15:08:58 2024-03-04 15:18:25 0:09:27 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} 3
Failure Reason:

Command failed on smithi103 with status 1: 'sudo yum install -y kernel'

fail 7581174 2024-03-04 15:07:58 2024-03-04 15:08:59 2024-03-04 15:26:26 0:17:27 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

Failed to reconnect to smithi103

pass 7581175 2024-03-04 15:07:59 2024-03-04 15:08:59 2024-03-04 15:50:29 0:41:30 0:27:55 0:13:35 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7581176 2024-03-04 15:08:00 2024-03-04 15:09:00 2024-03-04 15:51:27 0:42:27 0:27:57 0:14:30 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
pass 7581177 2024-03-04 15:08:01 2024-03-04 15:09:00 2024-03-04 16:03:47 0:54:47 0:40:59 0:13:48 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7581178 2024-03-04 15:08:02 2024-03-04 15:09:00 2024-03-04 16:02:17 0:53:17 0:42:10 0:11:07 smithi main ubuntu 20.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

"2024-03-04T15:28:07.509637+0000 mon.a (mon.0) 200 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

dead 7581179 2024-03-04 15:08:03 2024-03-04 15:09:01 2024-03-04 15:18:58 0:09:57 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
Failure Reason:

Error reimaging machines: list index out of range

pass 7581180 2024-03-04 15:08:04 2024-03-04 15:09:01 2024-03-04 15:36:10 0:27:09 0:15:54 0:11:15 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7581181 2024-03-04 15:08:05 2024-03-04 15:09:02 2024-03-04 15:41:10 0:32:08 0:17:19 0:14:49 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} 1
pass 7581182 2024-03-04 15:08:06 2024-03-04 15:09:02 2024-03-04 15:55:02 0:46:00 0:30:33 0:15:27 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7581183 2024-03-04 15:08:07 2024-03-04 15:09:02 2024-03-04 15:38:03 0:29:01 0:14:59 0:14:02 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
fail 7581184 2024-03-04 15:08:08 2024-03-04 15:09:03 2024-03-04 15:57:13 0:48:10 0:35:54 0:12:16 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-04T15:50:00.000136+0000 mon.a (mon.0) 1799 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

fail 7581185 2024-03-04 15:08:09 2024-03-04 15:09:03 2024-03-04 16:29:41 1:20:38 1:06:48 0:13:50 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-04T16:00:00.000195+0000 mon.a (mon.0) 2570 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled" in cluster log

pass 7581186 2024-03-04 15:08:10 2024-03-04 15:09:04 2024-03-04 15:40:54 0:31:50 0:17:25 0:14:25 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} 2
pass 7581187 2024-03-04 15:08:11 2024-03-04 15:09:04 2024-03-04 15:48:49 0:39:45 0:25:23 0:14:22 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
fail 7581188 2024-03-04 15:08:11 2024-03-04 15:09:04 2024-03-04 16:14:27 1:05:23 0:56:06 0:09:17 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7581189 2024-03-04 15:08:12 2024-03-04 15:09:05 2024-03-04 15:44:55 0:35:50 0:24:14 0:11:36 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
fail 7581190 2024-03-04 15:08:13 2024-03-04 15:09:05 2024-03-04 15:45:58 0:36:53 0:24:10 0:12:43 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi090 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

pass 7581191 2024-03-04 15:08:14 2024-03-04 15:09:06 2024-03-04 15:45:30 0:36:24 0:22:52 0:13:32 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
pass 7581192 2024-03-04 15:08:15 2024-03-04 15:09:06 2024-03-04 15:42:28 0:33:22 0:18:46 0:14:36 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
pass 7581193 2024-03-04 15:08:16 2024-03-04 15:09:07 2024-03-04 15:36:22 0:27:15 0:16:12 0:11:03 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
pass 7581194 2024-03-04 15:08:17 2024-03-04 15:09:07 2024-03-04 15:50:23 0:41:16 0:28:39 0:12:37 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} 2
pass 7581195 2024-03-04 15:08:18 2024-03-04 15:09:07 2024-03-04 15:25:46 0:16:39 0:06:54 0:09:45 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
fail 7581196 2024-03-04 15:08:19 2024-03-04 15:09:08 2024-03-04 16:01:48 0:52:40 0:39:06 0:13:34 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1709566765.979432 mon.smithi012 (mon.0) 330 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7581197 2024-03-04 15:08:20 2024-03-04 15:09:08 2024-03-04 15:44:31 0:35:23 0:21:10 0:14:13 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} 1
dead 7581198 2024-03-04 15:08:21 2024-03-04 15:09:09 2024-03-04 15:15:17 0:06:08 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Error reimaging machines: Expected smithi170's OS to be centos 9 but found ubuntu 22.04

dead 7581199 2024-03-04 15:08:22 2024-03-04 15:09:09 2024-03-04 15:15:17 0:06:08 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

Error reimaging machines: Expected smithi170's OS to be centos 9 but found ubuntu 22.04

dead 7581200 2024-03-04 15:08:23 2024-03-04 15:09:09 2024-03-04 15:19:04 0:09:55 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

Error reimaging machines: list index out of range

fail 7581201 2024-03-04 15:08:24 2024-03-04 15:09:10 2024-03-04 16:06:49 0:57:39 0:45:49 0:11:50 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-04T15:50:00.000151+0000 mon.a (mon.0) 1181 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

pass 7581202 2024-03-04 15:08:25 2024-03-04 15:09:10 2024-03-04 15:33:21 0:24:11 0:14:21 0:09:50 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} 2
pass 7581203 2024-03-04 15:08:26 2024-03-04 15:09:11 2024-03-04 15:36:18 0:27:07 0:15:16 0:11:51 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
pass 7581204 2024-03-04 15:08:27 2024-03-04 15:09:11 2024-03-04 15:37:34 0:28:23 0:16:11 0:12:12 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
pass 7581205 2024-03-04 15:08:28 2024-03-04 15:09:11 2024-03-04 15:37:01 0:27:50 0:16:17 0:11:33 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
fail 7581206 2024-03-04 15:08:29 2024-03-04 15:09:12 2024-03-04 15:20:32 0:11:20 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi073 with status 1: 'sudo yum install -y kernel'

fail 7581207 2024-03-04 15:08:30 2024-03-04 15:09:12 2024-03-04 15:27:59 0:18:47 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rotate-keys} 2
Failure Reason:

Failed to reconnect to smithi073

pass 7581208 2024-03-04 15:08:31 2024-03-04 15:09:13 2024-03-04 15:50:39 0:41:26 0:29:37 0:11:49 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} 3
fail 7581209 2024-03-04 15:08:32 2024-03-04 15:09:13 2024-03-04 15:24:48 0:15:35 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} 1
Failure Reason:

failed to install new kernel version within timeout

pass 7581210 2024-03-04 15:08:33 2024-03-04 15:09:14 2024-03-04 15:52:04 0:42:50 0:28:42 0:14:08 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
pass 7581211 2024-03-04 15:08:34 2024-03-04 15:09:14 2024-03-04 15:44:14 0:35:00 0:24:44 0:10:16 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
pass 7581212 2024-03-04 15:08:35 2024-03-04 15:09:14 2024-03-04 15:42:54 0:33:40 0:19:57 0:13:43 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7581213 2024-03-04 15:08:36 2024-03-04 15:09:15 2024-03-04 15:33:50 0:24:35 0:12:10 0:12:25 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli} 2
pass 7581214 2024-03-04 15:08:37 2024-03-04 15:09:15 2024-03-04 15:46:40 0:37:25 0:22:53 0:14:32 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7581215 2024-03-04 15:08:38 2024-03-04 15:09:15 2024-03-04 15:34:44 0:25:29 0:13:37 0:11:52 smithi main centos 9.stream orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} 1
fail 7581216 2024-03-04 15:08:39 2024-03-04 15:09:16 2024-03-04 15:44:34 0:35:18 0:21:47 0:13:31 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-04T15:38:57.904963+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7581217 2024-03-04 15:08:40 2024-03-04 15:09:16 2024-03-04 15:56:20 0:47:04 0:35:18 0:11:46 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7581218 2024-03-04 15:08:41 2024-03-04 15:09:17 2024-03-04 16:00:28 0:51:11 0:38:28 0:12:43 smithi main ubuntu 20.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-03-04T15:30:30.119779+0000 mon.a (mon.0) 169 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7581219 2024-03-04 15:08:42 2024-03-04 15:09:17 2024-03-04 15:45:56 0:36:39 0:14:11 0:22:28 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

Failed to reconnect to smithi170

dead 7581220 2024-03-04 15:08:43 2024-03-04 15:15:29 2024-03-04 15:40:54 0:25:25 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

SSH connection to smithi165 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

fail 7581221 2024-03-04 15:08:43 2024-03-04 15:17:29 2024-03-04 16:15:24 0:57:55 0:47:10 0:10:45 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-04T15:41:41.931377+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7581222 2024-03-04 15:08:44 2024-03-04 15:17:30 2024-03-04 15:32:45 0:15:15 0:04:13 0:11:02 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} 3
Failure Reason:

['Failed to manage policy for boolean nagios_run_sudo: [Errno 11] Resource temporarily unavailable']

pass 7581223 2024-03-04 15:08:45 2024-03-04 15:19:11 2024-03-04 15:53:09 0:33:58 0:24:00 0:09:58 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
pass 7581224 2024-03-04 15:08:46 2024-03-04 15:19:21 2024-03-04 15:48:38 0:29:17 0:14:14 0:15:03 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7581225 2024-03-04 15:08:47 2024-03-04 15:25:53 2024-03-04 16:20:11 0:54:18 0:36:54 0:17:24 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} 5
pass 7581226 2024-03-04 15:08:48 2024-03-04 15:31:35 2024-03-04 16:03:17 0:31:42 0:20:53 0:10:49 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} 3
pass 7581227 2024-03-04 15:08:49 2024-03-04 15:31:36 2024-03-04 15:57:54 0:26:18 0:15:53 0:10:25 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
pass 7581228 2024-03-04 15:08:50 2024-03-04 15:32:26 2024-03-04 15:56:26 0:24:00 0:14:41 0:09:19 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
fail 7581229 2024-03-04 15:08:51 2024-03-04 15:33:27 2024-03-04 16:20:44 0:47:17 0:38:16 0:09:01 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7581230 2024-03-04 15:08:52 2024-03-04 15:33:47 2024-03-04 16:00:53 0:27:06 0:17:28 0:09:38 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
fail 7581231 2024-03-04 15:08:53 2024-03-04 15:33:58 2024-03-04 16:06:28 0:32:30 0:22:06 0:10:24 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python} 2
Failure Reason:

"2024-03-04T15:52:18.835520+0000 mon.a (mon.0) 177 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7581232 2024-03-04 15:08:54 2024-03-04 16:01:27 934 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} 3
pass 7581233 2024-03-04 15:08:55 2024-03-04 15:36:19 2024-03-04 16:10:09 0:33:50 0:24:01 0:09:49 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7581234 2024-03-04 15:08:56 2024-03-04 15:36:30 2024-03-04 16:02:19 0:25:49 0:14:14 0:11:35 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7581235 2024-03-04 15:08:57 2024-03-04 15:37:00 2024-03-04 16:24:31 0:47:31 0:37:26 0:10:05 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7581236 2024-03-04 15:08:58 2024-03-04 15:37:11 2024-03-04 15:56:13 0:19:02 0:10:12 0:08:50 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_adoption} 1
pass 7581237 2024-03-04 15:08:59 2024-03-04 16:02:36 924 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
fail 7581238 2024-03-04 15:09:00 2024-03-04 15:37:41 2024-03-04 16:00:14 0:22:33 0:11:35 0:10:58 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-04T15:55:05.452976+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7581239 2024-03-04 15:09:01 2024-03-04 16:52:12 3705 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-04T16:10:00.001602+0000 mon.a (mon.0) 1063 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

fail 7581240 2024-03-04 15:09:02 2024-03-04 15:41:03 2024-03-04 16:21:38 0:40:35 0:28:38 0:11:57 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-03-04T15:57:30.221569+0000 mon.a (mon.0) 165 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7581241 2024-03-04 15:09:03 2024-03-04 15:42:34 2024-03-04 16:03:31 0:20:57 0:11:18 0:09:39 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} 2
pass 7581242 2024-03-04 15:09:04 2024-03-04 15:42:34 2024-03-04 16:07:22 0:24:48 0:14:14 0:10:34 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7581243 2024-03-04 15:09:05 2024-03-04 15:44:15 2024-03-04 16:17:55 0:33:40 0:23:22 0:10:18 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rotate-keys} 2
fail 7581244 2024-03-04 15:09:06 2024-03-04 15:44:35 2024-03-04 16:10:36 0:26:01 0:13:12 0:12:49 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi178 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

pass 7581245 2024-03-04 15:09:07 2024-03-04 15:44:56 2024-03-04 16:16:05 0:31:09 0:19:56 0:11:13 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
fail 7581246 2024-03-04 15:09:09 2024-03-04 19:36:42 13002 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi134 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

pass 7581247 2024-03-04 15:09:10 2024-03-04 15:46:47 2024-03-04 16:19:06 0:32:19 0:21:14 0:11:05 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7581248 2024-03-04 15:09:11 2024-03-04 15:46:58 2024-03-04 16:23:00 0:36:02 0:25:55 0:10:07 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
pass 7581249 2024-03-04 15:09:12 2024-03-04 16:10:29 858 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
fail 7581250 2024-03-04 15:09:13 2024-03-04 15:46:59 2024-03-04 16:06:29 0:19:30 0:08:49 0:10:41 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} 1
Failure Reason:

Command failed (workunit test cephadm/test_repos.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9ca0b5de35ed230e84387abdd8e619ed8e4f3282 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

pass 7581251 2024-03-04 15:09:14 2024-03-04 15:46:59 2024-03-04 16:12:06 0:25:07 0:14:24 0:10:43 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_cephadm_timeout} 1
pass 7581252 2024-03-04 15:09:15 2024-03-04 15:46:59 2024-03-04 16:13:50 0:26:51 0:18:10 0:08:41 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
fail 7581253 2024-03-04 15:09:16 2024-03-04 15:47:00 2024-03-04 16:27:49 0:40:49 0:30:48 0:10:01 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-04T16:20:00.000104+0000 mon.a (mon.0) 1812 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log