Pulpito

teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi

User Scheduled Started Updated Runtime Suite Branch Machine Type Revision Pass Fail Dead
teuthology 2024-03-11 20:08:13 2024-03-12 02:25:06 2024-03-12 15:06:27 12:41:21 orch main smithi 2b0101e 116 53 4

Average wait time: 0:11:30

See other runs of suite 'orch' on branch 'main'?

See other runs on branch 'main'?

See other runs scheduled on 2024-03-11?

See detail view?

Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7591724 2024-03-11 20:09:32 2024-03-11 20:11:40 2024-03-11 20:37:06 0:25:26 0:15:39 0:09:47 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
fail 7591725 2024-03-11 20:09:33 2024-03-11 20:11:40 2024-03-11 20:39:21 0:27:41 0:17:19 0:10:22 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi096 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7591726 2024-03-11 20:09:34 2024-03-11 20:11:41 2024-03-11 21:14:48 1:03:07 0:51:45 0:11:22 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-11T20:40:00.000210+0000 mon.a (mon.0) 1881 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log

pass 7591727 2024-03-11 20:09:35 2024-03-11 20:11:41 2024-03-11 20:40:37 0:28:56 0:18:57 0:09:59 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
pass 7591728 2024-03-11 20:09:36 2024-03-11 20:11:41 2024-03-11 20:47:49 0:36:08 0:23:26 0:12:42 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rotate-keys} 2
pass 7591729 2024-03-11 20:09:37 2024-03-11 20:12:52 2024-03-11 20:44:38 0:31:46 0:22:17 0:09:29 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} 2
pass 7591730 2024-03-11 20:09:38 2024-03-11 20:30:09 407 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
fail 7591731 2024-03-11 20:09:39 2024-03-11 20:12:53 2024-03-11 20:59:10 0:46:17 0:35:24 0:10:53 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-03-11T20:50:00.000265+0000 mon.smithi063 (mon.0) 488 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 38/228 objects degraded (16.667%), 15 pgs degraded" in cluster log

fail 7591732 2024-03-11 20:09:40 2024-03-11 20:13:13 2024-03-11 21:00:16 0:47:03 0:34:47 0:12:16 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-03-11T20:33:38.064181+0000 mon.smithi083 (mon.0) 497 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

pass 7591733 2024-03-11 20:09:41 2024-03-11 20:13:13 2024-03-11 21:10:42 0:57:29 0:46:56 0:10:33 smithi main ubuntu 22.04 orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
pass 7591734 2024-03-11 20:09:42 2024-03-11 20:13:14 2024-03-11 20:32:05 0:18:51 0:10:02 0:08:49 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_adoption} 1
pass 7591735 2024-03-11 20:09:43 2024-03-11 20:13:14 2024-03-11 20:37:03 0:23:49 0:12:05 0:11:44 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} 2
pass 7591736 2024-03-11 20:09:44 2024-03-11 20:13:14 2024-03-11 20:36:32 0:23:18 0:13:14 0:10:04 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} 2
fail 7591737 2024-03-11 20:09:45 2024-03-11 20:13:15 2024-03-11 20:48:40 0:35:25 0:25:27 0:09:58 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-03-11T20:29:56.950776+0000 mon.a (mon.0) 188 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7591738 2024-03-11 20:09:47 2024-03-11 20:13:15 2024-03-11 20:35:58 0:22:43 0:13:13 0:09:30 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final} 2
pass 7591739 2024-03-11 20:09:48 2024-03-11 20:13:16 2024-03-11 20:40:47 0:27:31 0:18:31 0:09:00 smithi main ubuntu 22.04 orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} 1
fail 7591740 2024-03-11 20:09:49 2024-03-11 20:13:16 2024-03-11 20:34:22 0:21:06 0:11:26 0:09:40 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-11T20:29:49.739247+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

fail 7591741 2024-03-11 20:09:50 2024-03-11 20:13:16 2024-03-11 21:10:36 0:57:20 0:43:33 0:13:47 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi008 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 05bd216a-dfe6-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''

pass 7591742 2024-03-11 20:09:51 2024-03-11 20:15:27 2024-03-11 20:42:50 0:27:23 0:14:36 0:12:47 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
pass 7591743 2024-03-11 20:09:52 2024-03-11 20:17:08 2024-03-11 20:46:21 0:29:13 0:18:26 0:10:47 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} 2
pass 7591744 2024-03-11 20:09:53 2024-03-11 20:17:18 2024-03-11 20:49:37 0:32:19 0:20:19 0:12:00 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
fail 7591745 2024-03-11 20:09:54 2024-03-11 20:18:29 2024-03-11 21:30:33 1:12:04 0:59:52 0:12:12 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-11T21:00:00.000279+0000 mon.a (mon.0) 1176 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

pass 7591746 2024-03-11 20:09:55 2024-03-11 20:19:40 2024-03-11 20:56:11 0:36:31 0:25:19 0:11:12 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
pass 7591747 2024-03-11 20:09:56 2024-03-11 20:20:30 2024-03-11 20:46:32 0:26:02 0:14:41 0:11:21 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
pass 7591748 2024-03-11 20:09:57 2024-03-11 20:21:41 2024-03-11 20:51:24 0:29:43 0:13:42 0:16:01 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
fail 7591749 2024-03-11 20:09:58 2024-03-11 20:26:22 2024-03-11 21:30:16 1:03:54 0:51:48 0:12:06 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-11T20:47:20.463493+0000 mon.a (mon.0) 175 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7591750 2024-03-11 20:09:59 2024-03-11 20:26:22 2024-03-11 21:03:15 0:36:53 0:23:19 0:13:34 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} 3
fail 7591751 2024-03-11 20:10:00 2024-03-11 20:28:13 2024-03-11 21:17:32 0:49:19 0:37:36 0:11:43 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7591752 2024-03-11 20:10:01 2024-03-11 20:28:33 2024-03-11 20:53:34 0:25:01 0:14:00 0:11:01 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_cephadm_timeout} 1
pass 7591753 2024-03-11 20:10:02 2024-03-11 20:28:34 2024-03-11 20:54:26 0:25:52 0:14:48 0:11:04 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7591754 2024-03-11 20:10:03 2024-03-11 20:28:34 2024-03-11 21:00:04 0:31:30 0:21:48 0:09:42 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7591755 2024-03-11 20:10:04 2024-03-11 20:28:34 2024-03-11 20:51:31 0:22:57 0:13:36 0:09:21 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7591756 2024-03-11 20:10:05 2024-03-11 20:28:35 2024-03-11 20:53:30 0:24:55 0:15:42 0:09:13 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
pass 7591757 2024-03-11 20:10:06 2024-03-11 20:28:35 2024-03-11 20:52:42 0:24:07 0:12:13 0:11:54 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
pass 7591758 2024-03-11 20:10:07 2024-03-11 20:28:35 2024-03-11 20:59:12 0:30:37 0:20:20 0:10:17 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
fail 7591759 2024-03-11 20:10:08 2024-03-11 20:28:36 2024-03-11 21:09:32 0:40:56 0:30:23 0:10:33 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-11T21:00:00.000141+0000 mon.a (mon.0) 1565 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

fail 7591760 2024-03-11 20:10:09 2024-03-11 20:28:36 2024-03-11 20:46:08 0:17:32 0:07:30 0:10:02 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/calico rook/1.7.2} 3
Failure Reason:

Command failed on smithi052 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

fail 7591761 2024-03-11 20:10:10 2024-03-11 20:28:37 2024-03-11 21:17:39 0:49:02 0:38:00 0:11:02 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-03-11T20:47:42.583795+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7591762 2024-03-11 20:10:11 2024-03-11 20:28:37 2024-03-11 20:54:44 0:26:07 0:15:59 0:10:08 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
pass 7591763 2024-03-11 20:10:12 2024-03-11 20:28:57 2024-03-11 21:03:17 0:34:20 0:19:58 0:14:22 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} 3
pass 7591764 2024-03-11 20:10:13 2024-03-11 20:30:08 2024-03-11 21:15:25 0:45:17 0:33:02 0:12:15 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
pass 7591765 2024-03-11 20:10:14 2024-03-11 20:30:39 2024-03-11 21:00:31 0:29:52 0:19:36 0:10:16 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_orch_cli} 1
pass 7591766 2024-03-11 20:10:15 2024-03-11 20:30:39 2024-03-11 21:07:33 0:36:54 0:24:34 0:12:20 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
fail 7591767 2024-03-11 20:10:16 2024-03-11 20:33:50 2024-03-11 21:17:25 0:43:35 0:31:58 0:11:37 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} 2
Failure Reason:

"2024-03-11T20:56:41.184995+0000 mon.a (mon.0) 182 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7591768 2024-03-11 20:10:17 2024-03-11 20:36:00 2024-03-11 21:01:49 0:25:49 0:14:44 0:11:05 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7591769 2024-03-11 20:10:18 2024-03-11 20:36:41 2024-03-11 21:04:38 0:27:57 0:17:03 0:10:54 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7591770 2024-03-11 20:10:19 2024-03-11 20:37:11 2024-03-11 21:00:51 0:23:40 0:13:37 0:10:03 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7591771 2024-03-11 20:10:20 2024-03-11 20:37:12 2024-03-11 21:09:57 0:32:45 0:19:22 0:13:23 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} 3
pass 7591772 2024-03-11 20:10:21 2024-03-11 20:40:02 2024-03-11 21:05:45 0:25:43 0:14:43 0:11:00 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
fail 7591773 2024-03-11 20:10:22 2024-03-11 20:40:43 2024-03-11 21:32:29 0:51:46 0:36:17 0:15:29 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-11T21:10:00.000160+0000 mon.a (mon.0) 875 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

pass 7591774 2024-03-11 20:10:23 2024-03-11 20:42:54 2024-03-11 21:13:47 0:30:53 0:20:07 0:10:46 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7591775 2024-03-11 20:10:24 2024-03-11 20:44:14 2024-03-11 21:09:05 0:24:51 0:15:42 0:09:09 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} 3
fail 7591776 2024-03-11 20:10:25 2024-03-11 20:44:14 2024-03-11 21:33:08 0:48:54 0:38:00 0:10:54 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7591777 2024-03-11 20:10:26 2024-03-11 20:44:15 2024-03-11 21:19:32 0:35:17 0:23:35 0:11:42 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_orch_cli_mon} 5
pass 7591778 2024-03-11 20:10:27 2024-03-11 20:44:15 2024-03-11 21:05:01 0:20:46 0:11:31 0:09:15 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} 2
pass 7591779 2024-03-11 20:10:28 2024-03-11 20:44:16 2024-03-11 21:07:04 0:22:48 0:14:18 0:08:30 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7591780 2024-03-11 20:10:29 2024-03-11 20:44:16 2024-03-11 21:07:07 0:22:51 0:12:46 0:10:05 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
fail 7591781 2024-03-11 20:10:30 2024-03-11 20:44:16 2024-03-11 21:08:09 0:23:53 0:11:36 0:12:17 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-11T21:01:49.482049+0000 mon.a (mon.0) 245 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7591782 2024-03-11 20:10:31 2024-03-11 20:44:47 2024-03-11 21:11:08 0:26:21 0:15:14 0:11:07 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7591783 2024-03-11 20:10:32 2024-03-11 20:45:47 2024-03-11 21:11:33 0:25:46 0:15:11 0:10:35 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
pass 7591784 2024-03-11 20:10:33 2024-03-11 20:46:28 2024-03-11 21:21:42 0:35:14 0:23:57 0:11:17 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rotate-keys} 2
fail 7591785 2024-03-11 20:10:34 2024-03-11 20:46:38 2024-03-11 21:26:56 0:40:18 0:28:55 0:11:23 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-03-11T21:06:29.207460+0000 mon.a (mon.0) 469 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

pass 7591786 2024-03-11 20:10:35 2024-03-11 20:47:59 2024-03-11 21:12:03 0:24:04 0:11:20 0:12:44 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} 2
pass 7591787 2024-03-11 20:10:36 2024-03-11 20:49:39 2024-03-11 21:27:14 0:37:35 0:26:38 0:10:57 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
pass 7591788 2024-03-11 20:10:37 2024-03-11 20:51:30 2024-03-11 21:20:47 0:29:17 0:15:55 0:13:22 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
fail 7591789 2024-03-11 20:10:38 2024-03-11 20:52:50 2024-03-11 21:54:56 1:02:06 0:50:42 0:11:24 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-11T21:30:00.303896+0000 mon.a (mon.0) 1365 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log

pass 7591790 2024-03-11 20:10:39 2024-03-11 20:52:51 2024-03-11 21:20:44 0:27:53 0:15:52 0:12:01 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final} 2
fail 7591791 2024-03-11 20:10:40 2024-03-11 20:53:31 2024-03-11 21:16:50 0:23:19 0:13:17 0:10:02 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi119 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

pass 7591792 2024-03-11 20:10:41 2024-03-11 20:53:42 2024-03-11 21:39:41 0:45:59 0:33:26 0:12:33 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
pass 7591793 2024-03-11 20:10:42 2024-03-11 20:54:32 2024-03-11 21:16:34 0:22:02 0:10:31 0:11:31 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_adoption} 1
pass 7591794 2024-03-11 20:10:43 2024-03-11 20:54:53 2024-03-11 21:24:01 0:29:08 0:18:29 0:10:39 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7591795 2024-03-11 20:10:44 2024-03-11 20:56:13 2024-03-11 21:22:42 0:26:29 0:14:29 0:12:00 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final} 2
pass 7591796 2024-03-11 20:10:45 2024-03-11 20:57:44 2024-03-11 21:29:38 0:31:54 0:21:04 0:10:50 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
fail 7591797 2024-03-11 20:10:46 2024-03-11 20:57:44 2024-03-11 22:02:51 1:05:07 0:52:57 0:12:10 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-11T21:30:00.000181+0000 mon.a (mon.0) 2380 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled" in cluster log

fail 7591798 2024-03-11 20:10:47 2024-03-11 20:59:15 2024-03-11 21:17:05 0:17:50 0:08:30 0:09:20 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} 1
Failure Reason:

Command failed (workunit test cephadm/test_repos.sh) on smithi122 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

pass 7591799 2024-03-11 20:10:48 2024-03-11 20:59:35 2024-03-11 21:24:36 0:25:01 0:14:52 0:10:09 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
fail 7591800 2024-03-11 20:10:49 2024-03-11 20:59:35 2024-03-11 21:22:42 0:23:07 0:11:39 0:11:28 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-11T21:17:08.454036+0000 mon.a (mon.0) 193 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7591801 2024-03-11 20:10:50 2024-03-11 20:59:36 2024-03-11 21:28:46 0:29:10 0:17:49 0:11:21 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
fail 7591802 2024-03-11 20:10:51 2024-03-11 20:59:36 2024-03-11 21:16:59 0:17:23 0:07:32 0:09:51 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/flannel rook/master} 1
Failure Reason:

Command failed on smithi003 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7591803 2024-03-11 20:10:52 2024-03-11 20:59:36 2024-03-11 21:23:11 0:23:35 0:14:16 0:09:19 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
fail 7591804 2024-03-11 20:10:53 2024-03-11 20:59:37 2024-03-11 22:06:41 1:07:04 0:56:11 0:10:53 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-11T21:30:00.002162+0000 mon.a (mon.0) 921 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

pass 7591805 2024-03-11 20:10:54 2024-03-11 20:59:37 2024-03-11 21:24:37 0:25:00 0:15:30 0:09:30 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_extra_daemon_features} 2
fail 7591806 2024-03-11 20:10:55 2024-03-11 20:59:38 2024-03-11 21:48:36 0:48:58 0:38:02 0:10:56 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7591807 2024-03-11 20:10:56 2024-03-11 21:00:58 2024-03-11 21:31:28 0:30:30 0:19:38 0:10:52 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_cephadm_timeout} 1
pass 7591808 2024-03-11 20:10:57 2024-03-11 21:01:59 2024-03-11 21:35:06 0:33:07 0:21:41 0:11:26 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7591809 2024-03-11 20:10:58 2024-03-11 21:03:19 2024-03-11 21:39:11 0:35:52 0:25:26 0:10:26 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
fail 7591810 2024-03-11 20:10:59 2024-03-11 21:03:19 2024-03-11 22:18:50 1:15:31 1:03:21 0:12:10 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

"2024-03-11T21:26:05.260760+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7591811 2024-03-11 20:11:00 2024-03-11 21:03:20 2024-03-11 21:36:08 0:32:48 0:19:02 0:13:46 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7591812 2024-03-11 20:11:01 2024-03-11 21:04:40 2024-03-11 21:34:16 0:29:36 0:17:43 0:11:53 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain} 3
pass 7591813 2024-03-11 20:11:02 2024-03-11 21:05:11 2024-03-11 21:34:49 0:29:38 0:18:58 0:10:40 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
pass 7591814 2024-03-11 20:11:03 2024-03-11 21:05:52 2024-03-11 21:34:15 0:28:23 0:16:40 0:11:43 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
fail 7591815 2024-03-11 20:11:04 2024-03-11 21:07:12 2024-03-11 21:53:27 0:46:15 0:36:48 0:09:27 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} 2
Failure Reason:

"2024-03-11T21:33:01.944154+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7591816 2024-03-11 20:11:05 2024-03-11 21:07:43 2024-03-11 21:40:43 0:33:00 0:22:04 0:10:56 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
fail 7591817 2024-03-11 20:11:06 2024-03-12 02:25:06 2024-03-12 03:08:43 0:43:37 0:30:42 0:12:55 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-12T03:00:00.000133+0000 mon.a (mon.0) 1750 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

pass 7591818 2024-03-11 20:11:07 2024-03-12 02:25:06 2024-03-12 02:50:24 0:25:18 0:14:33 0:10:45 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7591819 2024-03-11 20:11:08 2024-03-12 02:25:06 2024-03-12 03:06:14 0:41:08 0:31:40 0:09:28 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7591820 2024-03-11 20:11:09 2024-03-12 02:25:17 2024-03-12 03:19:35 0:54:18 0:34:12 0:20:06 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-03-12T02:53:52.503774+0000 mon.smithi003 (mon.0) 491 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

pass 7591821 2024-03-11 20:11:10 2024-03-12 02:28:48 2024-03-12 03:15:43 0:46:55 0:36:24 0:10:31 smithi main centos 9.stream orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mon osd} overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/nfs} 1
pass 7591822 2024-03-11 20:11:11 2024-03-12 02:28:48 2024-03-12 02:52:00 0:23:12 0:13:42 0:09:30 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} 1
pass 7591823 2024-03-11 20:11:12 2024-03-12 02:28:58 2024-03-12 02:52:11 0:23:13 0:11:33 0:11:40 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} 2
pass 7591824 2024-03-11 20:11:13 2024-03-12 02:29:09 2024-03-12 03:04:53 0:35:44 0:24:54 0:10:50 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
pass 7591825 2024-03-11 20:11:14 2024-03-12 02:29:49 2024-03-12 02:57:42 0:27:53 0:17:00 0:10:53 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
pass 7591826 2024-03-11 20:11:15 2024-03-12 02:30:40 2024-03-12 02:51:42 0:21:02 0:11:35 0:09:27 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} 1
fail 7591827 2024-03-11 20:11:16 2024-03-12 02:30:40 2024-03-12 02:57:27 0:26:47 0:11:59 0:14:48 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-03-12T02:52:23.500567+0000 mon.a (mon.0) 190 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7591828 2024-03-11 20:11:17 2024-03-12 02:34:21 2024-03-12 03:03:39 0:29:18 0:17:22 0:11:56 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7591829 2024-03-11 20:11:18 2024-03-12 02:34:31 2024-03-12 03:04:25 0:29:54 0:19:32 0:10:22 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7591830 2024-03-11 20:11:19 2024-03-12 02:34:42 2024-03-12 03:04:10 0:29:28 0:19:31 0:09:57 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
fail 7591831 2024-03-11 20:11:20 2024-03-12 02:34:52 2024-03-12 03:07:05 0:32:13 0:21:50 0:10:23 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-03-12T03:00:00.000207+0000 mon.smithi106 (mon.0) 725 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s)" in cluster log

pass 7591832 2024-03-11 20:11:21 2024-03-12 02:34:52 2024-03-12 03:01:20 0:26:28 0:15:02 0:11:26 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7591833 2024-03-11 20:11:22 2024-03-12 02:36:03 2024-03-12 03:07:41 0:31:38 0:14:41 0:16:57 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7591834 2024-03-11 20:11:24 2024-03-12 02:36:13 2024-03-12 03:12:53 0:36:40 0:24:10 0:12:30 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} 2
fail 7591835 2024-03-11 20:11:25 2024-03-12 02:37:44 2024-03-12 03:26:12 0:48:28 0:33:24 0:15:04 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

"2024-03-12T02:58:40.818683+0000 mon.a (mon.0) 197 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7591836 2024-03-11 20:11:26 2024-03-12 02:37:44 2024-03-12 03:38:17 1:00:33 0:49:24 0:11:09 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
Failure Reason:

"2024-03-12T03:00:16.177735+0000 mon.a (mon.0) 180 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7591837 2024-03-11 20:11:27 2024-03-12 02:38:35 2024-03-12 03:13:16 0:34:41 0:18:06 0:16:35 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_rgw_multisite} 3
fail 7591838 2024-03-11 20:11:28 2024-03-12 02:40:15 2024-03-12 03:29:45 0:49:30 0:38:07 0:11:23 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

dead 7591839 2024-03-11 20:11:29 2024-03-12 02:40:26 2024-03-12 02:41:41 0:01:15 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} 5
Failure Reason:

Error reimaging machines: Failed to power on smithi028

dead 7591840 2024-03-11 20:11:30 2024-03-12 02:40:36 2024-03-12 14:49:24 12:08:48 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

hit max job timeout

pass 7591841 2024-03-11 20:11:31 2024-03-12 02:40:37 2024-03-12 03:05:20 0:24:43 0:14:39 0:10:04 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7591842 2024-03-11 20:11:32 2024-03-12 02:40:37 2024-03-12 03:10:07 0:29:30 0:20:26 0:09:04 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
pass 7591843 2024-03-11 20:11:33 2024-03-12 02:40:38 2024-03-12 03:13:20 0:32:42 0:21:59 0:10:43 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
fail 7591844 2024-03-11 20:11:34 2024-03-12 02:41:48 2024-03-12 02:59:41 0:17:53 0:07:30 0:10:23 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/host rook/1.7.2} 3
Failure Reason:

Command failed on smithi001 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7591845 2024-03-11 20:11:35 2024-03-12 02:41:59 2024-03-12 03:13:12 0:31:13 0:17:52 0:13:21 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
fail 7591846 2024-03-11 20:11:36 2024-03-12 02:41:59 2024-03-12 03:05:28 0:23:29 0:12:13 0:11:16 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-12T02:59:19.880343+0000 mon.a (mon.0) 248 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

dead 7591847 2024-03-11 20:11:37 2024-03-12 02:42:20 2024-03-12 14:52:39 12:10:19 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

hit max job timeout

fail 7591848 2024-03-11 20:11:38 2024-03-12 02:44:40 2024-03-12 03:32:50 0:48:10 0:37:37 0:10:33 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7591849 2024-03-11 20:11:39 2024-03-12 02:45:21 2024-03-12 03:07:29 0:22:08 0:12:09 0:09:59 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} 1
pass 7591850 2024-03-11 20:11:40 2024-03-12 02:45:51 2024-03-12 03:12:17 0:26:26 0:12:52 0:13:34 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7591851 2024-03-11 20:11:41 2024-03-12 02:48:42 2024-03-12 03:14:15 0:25:33 0:14:54 0:10:39 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7591852 2024-03-11 20:11:42 2024-03-12 02:49:12 2024-03-12 03:17:42 0:28:30 0:19:11 0:09:19 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
fail 7591853 2024-03-11 20:11:43 2024-03-12 04:06:36 3890 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-03-12T03:30:00.000296+0000 mon.a (mon.0) 2422 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

fail 7591854 2024-03-11 20:11:44 2024-03-12 02:50:33 2024-03-12 03:53:50 1:03:17 0:52:20 0:10:57 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

"2024-03-12T03:20:00.000255+0000 mon.a (mon.0) 2080 : cluster 3 [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets" in cluster log

fail 7591855 2024-03-11 20:11:45 2024-03-12 02:50:43 2024-03-12 03:13:47 0:23:04 0:11:09 0:11:55 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_ca_signed_key} 2
Failure Reason:

Command failed on smithi077 with status 5: 'sudo systemctl stop ceph-f9e0f334-e01d-11ee-95c8-87774f69a715@mon.a'

pass 7591856 2024-03-11 20:11:46 2024-03-12 02:52:04 2024-03-12 03:25:17 0:33:13 0:16:01 0:17:12 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
pass 7591857 2024-03-11 20:11:47 2024-03-12 02:52:14 2024-03-12 03:18:08 0:25:54 0:14:31 0:11:23 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
fail 7591858 2024-03-11 20:11:48 2024-03-12 02:52:55 2024-03-12 03:23:49 0:30:54 0:15:18 0:15:36 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7591859 2024-03-11 20:11:49 2024-03-12 02:52:55 2024-03-12 03:59:40 1:06:45 0:56:34 0:10:11 smithi main ubuntu 20.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-03-12T03:11:14.956338+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7591860 2024-03-11 20:11:50 2024-03-12 02:53:06 2024-03-12 03:23:44 0:30:38 0:20:04 0:10:34 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-03-12T03:20:00.000275+0000 mon.smithi083 (mon.0) 818 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 2 failed cephadm daemon(s)" in cluster log

pass 7591861 2024-03-11 20:11:51 2024-03-12 02:54:16 2024-03-12 03:24:16 0:30:00 0:19:25 0:10:35 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
fail 7591862 2024-03-11 20:11:52 2024-03-12 02:54:17 2024-03-12 04:18:24 1:24:07 1:10:33 0:13:34 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi118 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d8e7844-e01e-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

pass 7591863 2024-03-11 20:11:53 2024-03-12 02:54:37 2024-03-12 03:26:13 0:31:36 0:22:22 0:09:14 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} 1
pass 7591864 2024-03-11 20:11:54 2024-03-12 02:55:27 2024-03-12 03:24:36 0:29:09 0:16:21 0:12:48 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} 2
pass 7591865 2024-03-11 20:11:55 2024-03-12 02:55:28 2024-03-12 03:23:52 0:28:24 0:17:26 0:10:58 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7591866 2024-03-11 20:11:56 2024-03-12 02:55:28 2024-03-12 03:20:51 0:25:23 0:13:41 0:11:42 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
dead 7591867 2024-03-11 20:11:57 2024-03-12 02:57:49 2024-03-12 15:06:27 12:08:38 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

hit max job timeout

pass 7591868 2024-03-11 20:11:58 2024-03-12 02:58:09 2024-03-12 03:19:45 0:21:36 0:08:30 0:13:06 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos} 1
fail 7591869 2024-03-11 20:11:59 2024-03-12 03:00:20 2024-03-12 04:20:52 1:20:32 1:07:55 0:12:37 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-12T03:40:00.000170+0000 mon.a (mon.0) 1089 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

fail 7591870 2024-03-11 20:12:00 2024-03-12 03:01:30 2024-03-12 03:34:46 0:33:16 0:22:42 0:10:34 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} 2
Failure Reason:

"2024-03-12T03:30:00.000144+0000 mon.a (mon.0) 1190 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 2 pool(s) do not have an application enabled" in cluster log

pass 7591871 2024-03-11 20:12:01 2024-03-12 03:01:51 2024-03-12 03:32:49 0:30:58 0:17:48 0:13:10 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7591872 2024-03-11 20:12:02 2024-03-12 03:03:41 2024-03-12 03:40:03 0:36:22 0:25:25 0:10:57 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7591873 2024-03-11 20:12:03 2024-03-12 03:04:12 2024-03-12 03:36:11 0:31:59 0:21:33 0:10:26 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
pass 7591874 2024-03-11 20:12:04 2024-03-12 03:04:12 2024-03-12 03:34:12 0:30:00 0:19:33 0:10:27 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_extra_daemon_features} 2
pass 7591875 2024-03-11 20:12:05 2024-03-12 03:04:33 2024-03-12 03:38:21 0:33:48 0:17:47 0:16:01 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7591876 2024-03-11 20:12:06 2024-03-12 03:05:03 2024-03-12 03:34:05 0:29:02 0:17:45 0:11:17 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
fail 7591877 2024-03-11 20:12:07 2024-03-12 03:05:03 2024-03-12 03:56:33 0:51:30 0:38:23 0:13:07 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7591878 2024-03-11 20:12:08 2024-03-12 03:05:24 2024-03-12 03:32:16 0:26:52 0:16:35 0:10:17 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} 1
pass 7591879 2024-03-11 20:12:09 2024-03-12 03:05:24 2024-03-12 03:45:28 0:40:04 0:26:54 0:13:10 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
pass 7591880 2024-03-11 20:12:10 2024-03-12 03:06:25 2024-03-12 03:43:45 0:37:20 0:25:10 0:12:10 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_host_drain} 3
pass 7591881 2024-03-11 20:12:11 2024-03-12 03:07:45 2024-03-12 04:01:36 0:53:51 0:38:56 0:14:55 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
pass 7591882 2024-03-11 20:12:12 2024-03-12 03:10:16 2024-03-12 03:45:51 0:35:35 0:23:29 0:12:06 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
fail 7591883 2024-03-11 20:12:13 2024-03-12 03:11:37 2024-03-12 04:12:44 1:01:07 0:48:56 0:12:11 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-03-12T03:57:51.392089+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7591884 2024-03-11 20:12:14 2024-03-12 03:11:37 2024-03-12 04:01:46 0:50:09 0:40:34 0:09:35 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} 2
Failure Reason:

"2024-03-12T03:35:06.045695+0000 mon.a (mon.0) 181 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7591885 2024-03-11 20:12:15 2024-03-12 03:11:38 2024-03-12 03:42:20 0:30:42 0:16:19 0:14:23 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
fail 7591886 2024-03-11 20:12:16 2024-03-12 03:11:38 2024-03-12 03:29:08 0:17:30 0:07:18 0:10:12 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/1-node k8s/1.21 net/calico rook/master} 1
Failure Reason:

Command failed on smithi081 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7591887 2024-03-11 20:12:17 2024-03-12 03:11:38 2024-03-12 03:40:49 0:29:11 0:16:54 0:12:17 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
fail 7591888 2024-03-11 20:12:18 2024-03-12 03:11:39 2024-03-12 03:38:33 0:26:54 0:16:40 0:10:14 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-03-12T03:33:48.852760+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log

pass 7591889 2024-03-11 20:12:19 2024-03-12 03:11:39 2024-03-12 03:42:07 0:30:28 0:16:02 0:14:26 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7591890 2024-03-11 20:12:20 2024-03-12 03:11:40 2024-03-12 03:42:30 0:30:50 0:17:52 0:12:58 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7591891 2024-03-11 20:12:21 2024-03-12 03:11:40 2024-03-12 03:41:30 0:29:50 0:15:10 0:14:40 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
fail 7591892 2024-03-11 20:12:22 2024-03-12 03:11:40 2024-03-12 07:12:59 4:01:19 3:47:07 0:14:12 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi115 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1c349451176cc5b4ebfb24b22eaaa754e05cff6c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

pass 7591893 2024-03-11 20:12:23 2024-03-12 03:11:41 2024-03-12 04:01:26 0:49:45 0:37:14 0:12:31 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} 5
pass 7591894 2024-03-11 20:12:24 2024-03-12 03:12:51 2024-03-12 03:43:08 0:30:17 0:19:51 0:10:26 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7591895 2024-03-11 20:12:25 2024-03-12 03:13:22 2024-03-12 03:49:03 0:35:41 0:25:08 0:10:33 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_monitoring_stack_basic} 3
pass 7591896 2024-03-11 20:12:26 2024-03-12 03:13:22 2024-03-12 03:44:54 0:31:32 0:21:57 0:09:35 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2