Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7907671 2024-09-16 20:09:43 2024-09-17 09:07:39 2024-09-17 09:29:26 0:21:47 0:10:57 0:10:50 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_domain} 2
pass 7907672 2024-09-16 20:09:44 2024-09-17 09:07:59 2024-09-17 09:31:09 0:23:10 0:13:28 0:09:42 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7907673 2024-09-16 20:09:45 2024-09-17 09:08:20 2024-09-17 09:32:12 0:23:52 0:10:56 0:12:56 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} 2
pass 7907674 2024-09-16 20:09:47 2024-09-17 10:14:24 2024-09-17 10:57:50 0:43:26 0:32:46 0:10:40 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7907675 2024-09-16 20:09:48 2024-09-17 10:14:45 2024-09-17 10:33:04 0:18:19 0:08:04 0:10:15 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/3-node k8s/1.21 net/host rook/master} 3
Failure Reason:

Command failed on smithi060 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7907676 2024-09-16 20:09:49 2024-09-17 10:15:25 2024-09-17 11:10:19 0:54:54 0:40:37 0:14:17 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_orch_cli_mon} 5
pass 7907677 2024-09-16 20:09:51 2024-09-17 10:18:26 2024-09-17 10:48:07 0:29:41 0:19:47 0:09:54 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7907678 2024-09-16 20:09:52 2024-09-17 10:18:36 2024-09-17 11:14:39 0:56:03 0:46:36 0:09:27 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-squid 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
pass 7907679 2024-09-16 20:09:53 2024-09-17 10:18:47 2024-09-17 10:47:13 0:28:26 0:18:12 0:10:14 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_basic} 2
pass 7907680 2024-09-16 20:09:54 2024-09-17 10:19:17 2024-09-17 10:40:26 0:21:09 0:12:37 0:08:32 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
pass 7907681 2024-09-16 20:09:56 2024-09-17 10:19:18 2024-09-17 10:42:21 0:23:03 0:14:15 0:08:48 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
pass 7907682 2024-09-16 20:09:57 2024-09-17 10:19:28 2024-09-17 10:42:51 0:23:23 0:13:23 0:10:00 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7907683 2024-09-16 20:09:58 2024-09-17 10:19:29 2024-09-17 10:48:56 0:29:27 0:20:05 0:09:22 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} 1
dead 7907684 2024-09-16 20:10:00 2024-09-17 10:19:39 2024-09-17 18:28:46 8:09:07 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

pass 7907685 2024-09-16 20:10:01 2024-09-17 10:20:19 2024-09-17 10:42:08 0:21:49 0:10:47 0:11:02 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_res_dom} 2
pass 7907686 2024-09-16 20:10:02 2024-09-17 10:20:20 2024-09-17 10:55:00 0:34:40 0:23:05 0:11:35 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async start tasks/rotate-keys} 2
pass 7907687 2024-09-16 20:10:03 2024-09-17 10:22:10 2024-09-17 10:54:12 0:32:02 0:22:34 0:09:28 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw 3-final} 2
pass 7907688 2024-09-16 20:10:04 2024-09-17 10:22:41 2024-09-17 10:37:49 0:15:08 0:05:28 0:09:40 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
pass 7907689 2024-09-16 20:10:06 2024-09-17 10:22:51 2024-09-17 11:06:27 0:43:36 0:33:13 0:10:23 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
pass 7907690 2024-09-16 20:10:07 2024-09-17 10:23:22 2024-09-17 11:07:35 0:44:13 0:32:33 0:11:40 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
pass 7907691 2024-09-16 20:10:08 2024-09-17 10:24:32 2024-09-17 11:38:16 1:13:44 1:03:32 0:10:12 smithi main ubuntu 22.04 orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignore_mgr_down ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
pass 7907692 2024-09-16 20:10:10 2024-09-17 10:24:33 2024-09-17 10:44:22 0:19:49 0:09:05 0:10:44 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_adoption} 1
pass 7907693 2024-09-16 20:10:11 2024-09-17 10:25:23 2024-09-17 10:51:48 0:26:25 0:15:37 0:10:48 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} 2
fail 7907694 2024-09-16 20:10:12 2024-09-17 10:26:04 2024-09-17 10:52:13 0:26:09 0:15:28 0:10:41 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

reached maximum tries (120) after waiting for 120 seconds

fail 7907695 2024-09-16 20:10:13 2024-09-17 10:27:14 2024-09-17 11:01:04 0:33:50 0:22:28 0:11:22 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-09-17T10:43:29.929044+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7907696 2024-09-16 20:10:15 2024-09-17 10:28:05 2024-09-17 10:49:33 0:21:28 0:11:32 0:09:56 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_basic} 2
pass 7907697 2024-09-16 20:10:16 2024-09-17 10:28:15 2024-09-17 10:49:16 0:21:01 0:11:50 0:09:11 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/basic 3-final} 2
fail 7907698 2024-09-16 20:10:17 2024-09-17 10:28:16 2024-09-17 10:47:04 0:18:48 0:10:27 0:08:21 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} 1
Failure Reason:

"2024-09-17T10:44:29.814540+0000 mon.smithi045 (mon.0) 296 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7907699 2024-09-16 20:10:19 2024-09-17 10:28:16 2024-09-17 10:51:32 0:23:16 0:11:25 0:11:51 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
fail 7907700 2024-09-16 20:10:20 2024-09-17 10:28:16 2024-09-17 11:23:23 0:55:07 0:43:59 0:11:08 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi028 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8fc148b0-74e1-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''

pass 7907701 2024-09-16 20:10:21 2024-09-17 10:30:27 2024-09-17 10:51:35 0:21:08 0:12:41 0:08:27 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
pass 7907702 2024-09-16 20:10:22 2024-09-17 10:30:38 2024-09-17 10:56:13 0:25:35 0:12:43 0:12:52 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} 2
pass 7907703 2024-09-16 20:10:24 2024-09-17 10:34:10 2024-09-17 10:53:04 0:18:54 0:10:22 0:08:32 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_domain} 2
pass 7907704 2024-09-16 20:10:25 2024-09-17 10:34:10 2024-09-17 11:04:09 0:29:59 0:20:09 0:09:50 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
pass 7907705 2024-09-16 20:10:26 2024-09-17 10:34:10 2024-09-17 11:42:44 1:08:34 0:58:38 0:09:56 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} 2
pass 7907706 2024-09-16 20:10:27 2024-09-17 10:35:01 2024-09-17 10:56:05 0:21:04 0:11:45 0:09:19 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7907707 2024-09-16 20:10:29 2024-09-17 10:37:35 2024-09-17 11:13:50 0:36:15 0:25:16 0:10:59 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
pass 7907708 2024-09-16 20:10:30 2024-09-17 10:38:05 2024-09-17 11:02:21 0:24:16 0:13:02 0:11:14 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
pass 7907709 2024-09-16 20:10:31 2024-09-17 10:39:26 2024-09-17 11:10:30 0:31:04 0:18:48 0:12:16 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_basic} 2
pass 7907710 2024-09-16 20:10:32 2024-09-17 10:40:06 2024-09-17 11:23:58 0:43:52 0:33:11 0:10:41 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_api_tests} 2
pass 7907711 2024-09-16 20:10:34 2024-09-17 10:40:47 2024-09-17 11:17:09 0:36:22 0:23:45 0:12:37 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} 3
fail 7907712 2024-09-16 20:10:35 2024-09-17 10:42:18 2024-09-17 11:29:58 0:47:40 0:37:27 0:10:13 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 7907713 2024-09-16 20:10:36 2024-09-17 10:42:38 2024-09-17 11:03:36 0:20:58 0:12:37 0:08:21 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_cephadm_timeout} 1
pass 7907714 2024-09-16 20:10:37 2024-09-17 10:42:38 2024-09-17 11:06:22 0:23:44 0:12:59 0:10:45 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7907715 2024-09-16 20:10:38 2024-09-17 10:43:09 2024-09-17 11:18:38 0:35:29 0:26:47 0:08:42 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-squid 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
pass 7907716 2024-09-16 20:10:40 2024-09-17 10:43:39 2024-09-17 11:16:12 0:32:33 0:21:43 0:10:50 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7907717 2024-09-16 20:10:41 2024-09-17 10:43:39 2024-09-17 11:16:40 0:33:01 0:20:54 0:12:07 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
pass 7907718 2024-09-16 20:10:42 2024-09-17 10:47:00 2024-09-17 11:13:44 0:26:44 0:15:22 0:11:22 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_ctdb_res_basic} 4
pass 7907719 2024-09-16 20:10:43 2024-09-17 10:48:21 2024-09-17 11:11:51 0:23:30 0:12:44 0:10:46 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7907720 2024-09-16 20:10:44 2024-09-17 10:49:11 2024-09-17 11:12:32 0:23:21 0:14:20 0:09:01 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
pass 7907721 2024-09-16 20:10:46 2024-09-17 10:49:32 2024-09-17 11:11:24 0:21:52 0:11:05 0:10:47 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
pass 7907722 2024-09-16 20:10:47 2024-09-17 10:50:02 2024-09-17 11:27:46 0:37:44 0:27:43 0:10:01 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
pass 7907723 2024-09-16 20:10:48 2024-09-17 10:51:03 2024-09-17 11:14:29 0:23:26 0:13:55 0:09:31 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
dead 7907724 2024-09-16 20:10:49 2024-09-17 10:51:43 2024-09-17 11:11:28 0:19:45 0:07:29 0:12:16 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/calico rook/1.7.2} 3
Failure Reason:

['ubuntu@smithi179.front.sepia.ceph.com: Permission denied (publickey).']

pass 7907725 2024-09-16 20:10:50 2024-09-17 10:51:54 2024-09-17 11:17:29 0:25:35 0:15:48 0:09:47 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_ctdb_res_dom} 4
fail 7907726 2024-09-16 20:10:52 2024-09-17 10:52:24 2024-09-17 11:20:42 0:28:18 0:17:50 0:10:28 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} 3
Failure Reason:

Command failed on smithi089 with status 5: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3b223a86-74e5-11ef-bceb-c7b262605968 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v1/status\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts | jq -e \'"\'"\'.data | .[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n\''

pass 7907727 2024-09-16 20:10:53 2024-09-17 10:53:25 2024-09-17 11:37:25 0:44:00 0:32:24 0:11:36 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
pass 7907728 2024-09-16 20:10:54 2024-09-17 10:54:25 2024-09-17 11:29:57 0:35:32 0:25:11 0:10:21 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_orch_cli} 1
pass 7907729 2024-09-16 20:10:55 2024-09-17 10:54:26 2024-09-17 11:15:45 0:21:19 0:12:16 0:09:03 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7907730 2024-09-16 20:10:57 2024-09-17 10:54:56 2024-09-17 11:28:44 0:33:48 0:23:55 0:09:53 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
pass 7907731 2024-09-16 20:10:58 2024-09-17 10:55:16 2024-09-17 12:23:37 1:28:21 1:17:44 0:10:37 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-reef 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
pass 7907732 2024-09-16 20:10:59 2024-09-17 10:55:57 2024-09-17 11:39:44 0:43:47 0:33:15 0:10:32 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rados_python} 2
pass 7907733 2024-09-16 20:11:00 2024-09-17 10:56:17 2024-09-17 11:19:19 0:23:02 0:13:27 0:09:35 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7907734 2024-09-16 20:11:02 2024-09-17 10:56:28 2024-09-17 11:35:04 0:38:36 0:27:04 0:11:32 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_ctdb_res_ips} 4
pass 7907735 2024-09-16 20:11:03 2024-09-17 10:57:19 2024-09-17 11:22:38 0:25:19 0:14:53 0:10:26 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
dead 7907736 2024-09-16 20:11:04 2024-09-17 10:57:29 2024-09-17 19:08:50 8:11:21 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} 3
Failure Reason:

hit max job timeout

pass 7907737 2024-09-16 20:11:05 2024-09-17 10:59:20 2024-09-17 11:22:30 0:23:10 0:13:24 0:09:46 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
pass 7907738 2024-09-16 20:11:07 2024-09-17 10:59:30 2024-09-17 11:24:21 0:24:51 0:13:09 0:11:42 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7907739 2024-09-16 20:11:08 2024-09-17 11:01:21 2024-09-17 11:22:25 0:21:04 0:11:33 0:09:31 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_domain} 2
pass 7907740 2024-09-16 20:11:09 2024-09-17 11:01:21 2024-09-17 11:48:24 0:47:03 0:36:31 0:10:32 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
pass 7907741 2024-09-16 20:11:10 2024-09-17 11:01:22 2024-09-17 11:33:51 0:32:29 0:21:32 0:10:57 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7907742 2024-09-16 20:11:12 2024-09-17 11:01:52 2024-09-17 11:28:26 0:26:34 0:15:13 0:11:21 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} 3
fail 7907743 2024-09-16 20:11:13 2024-09-17 11:03:33 2024-09-17 11:51:25 0:47:52 0:36:54 0:10:58 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 7907744 2024-09-16 20:11:14 2024-09-17 11:04:23 2024-09-17 12:10:12 1:05:49 0:52:25 0:13:24 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_orch_cli_mon} 5
pass 7907745 2024-09-16 20:11:15 2024-09-17 11:06:44 2024-09-17 11:33:16 0:26:32 0:15:38 0:10:54 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/on orchestrator_cli} 2
pass 7907746 2024-09-16 20:11:17 2024-09-17 11:07:45 2024-09-17 11:31:48 0:24:03 0:12:42 0:11:21 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7907747 2024-09-16 20:11:18 2024-09-17 11:10:35 2024-09-17 11:31:33 0:20:58 0:11:34 0:09:24 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} 1
pass 7907748 2024-09-16 20:11:19 2024-09-17 11:10:36 2024-09-17 11:32:28 0:21:52 0:10:29 0:11:23 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
pass 7907749 2024-09-16 20:11:20 2024-09-17 11:10:46 2024-09-17 12:05:53 0:55:07 0:44:19 0:10:48 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-squid 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
pass 7907750 2024-09-16 20:11:21 2024-09-17 11:11:47 2024-09-17 11:30:45 0:18:58 0:10:09 0:08:49 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_res_basic} 2
pass 7907751 2024-09-16 20:11:23 2024-09-17 11:11:47 2024-09-17 11:34:12 0:22:25 0:12:35 0:09:50 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
pass 7907752 2024-09-16 20:11:24 2024-09-17 11:11:47 2024-09-17 11:45:13 0:33:26 0:22:56 0:10:30 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/root mon_election/classic msgr/async-v2only start tasks/rotate-keys} 2
pass 7907753 2024-09-16 20:11:25 2024-09-17 11:12:08 2024-09-17 11:42:24 0:30:16 0:20:45 0:09:31 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
fail 7907754 2024-09-16 20:11:26 2024-09-17 11:12:48 2024-09-17 11:35:12 0:22:24 0:10:37 0:11:47 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} 2
Failure Reason:

"2024-09-17T11:33:22.143558+0000 mon.a (mon.0) 343 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

dead 7907755 2024-09-16 20:11:27 2024-09-17 11:13:59 2024-09-17 19:23:04 8:09:05 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

pass 7907756 2024-09-16 20:11:29 2024-09-17 11:13:59 2024-09-17 11:43:49 0:29:50 0:19:22 0:10:28 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_dom} 2
pass 7907757 2024-09-16 20:11:30 2024-09-17 11:14:10 2024-09-17 11:37:39 0:23:29 0:14:13 0:09:16 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7907758 2024-09-16 20:11:31 2024-09-17 11:14:50 2024-09-17 11:38:00 0:23:10 0:14:03 0:09:07 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw 3-final} 2
pass 7907759 2024-09-16 20:11:32 2024-09-17 11:15:00 2024-09-17 11:38:59 0:23:59 0:13:36 0:10:23 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} 1
pass 7907760 2024-09-16 20:11:33 2024-09-17 11:16:01 2024-09-17 12:00:14 0:44:13 0:31:39 0:12:34 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
pass 7907761 2024-09-16 20:11:35 2024-09-17 11:16:21 2024-09-17 11:35:15 0:18:54 0:08:54 0:10:00 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_adoption} 1
fail 7907762 2024-09-16 20:11:36 2024-09-17 11:16:22 2024-09-17 11:56:28 0:40:06 0:30:00 0:10:06 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

reached maximum tries (120) after waiting for 120 seconds

pass 7907763 2024-09-16 20:11:37 2024-09-17 11:16:52 2024-09-17 11:48:11 0:31:19 0:19:36 0:11:43 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} 2
pass 7907764 2024-09-16 20:11:39 2024-09-17 11:17:23 2024-09-17 11:38:44 0:21:21 0:11:49 0:09:32 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/basic 3-final} 2
fail 7907765 2024-09-16 20:11:40 2024-09-17 11:17:43 2024-09-17 12:13:33 0:55:50 0:44:17 0:11:33 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

Command failed on smithi005 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 53f7b4ca-74e8-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''

pass 7907766 2024-09-16 20:11:41 2024-09-17 11:17:43 2024-09-17 11:48:02 0:30:19 0:20:46 0:09:33 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
pass 7907767 2024-09-16 20:11:42 2024-09-17 11:17:54 2024-09-17 11:39:13 0:21:19 0:10:38 0:10:41 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} 2
pass 7907768 2024-09-16 20:11:43 2024-09-17 11:18:14 2024-09-17 12:04:42 0:46:28 0:33:21 0:13:07 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} 2
fail 7907769 2024-09-16 20:11:45 2024-09-17 11:18:14 2024-09-17 11:36:15 0:18:01 0:09:09 0:08:52 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} 1
Failure Reason:

Command failed (workunit test cephadm/test_repos.sh) on smithi134 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2b698643d05f73e79c2d2888228947a84a4ac0bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

pass 7907770 2024-09-16 20:11:46 2024-09-17 11:18:55 2024-09-17 11:44:27 0:25:32 0:12:31 0:13:01 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
pass 7907771 2024-09-16 20:11:47 2024-09-17 11:19:35 2024-09-17 11:40:00 0:20:25 0:10:05 0:10:20 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
pass 7907772 2024-09-16 20:11:48 2024-09-17 11:20:56 2024-09-17 11:43:36 0:22:40 0:11:33 0:11:07 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7907773 2024-09-16 20:11:49 2024-09-17 11:22:36 2024-09-17 11:47:50 0:25:14 0:12:54 0:12:20 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
fail 7907774 2024-09-16 20:11:51 2024-09-17 11:22:47 2024-09-17 11:40:32 0:17:45 0:07:54 0:09:51 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/flannel rook/master} 1
Failure Reason:

Command failed on smithi159 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7907775 2024-09-16 20:11:52 2024-09-17 11:22:47 2024-09-17 12:22:01 0:59:14 0:50:29 0:08:45 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
pass 7907776 2024-09-16 20:11:53 2024-09-17 11:22:48 2024-09-17 11:42:32 0:19:44 0:10:03 0:09:41 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_basic} 2
pass 7907777 2024-09-16 20:11:54 2024-09-17 11:23:28 2024-09-17 11:45:11 0:21:43 0:11:56 0:09:47 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_extra_daemon_features} 2
fail 7907778 2024-09-16 20:11:56 2024-09-17 11:23:38 2024-09-17 12:10:06 0:46:28 0:38:01 0:08:27 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 7907779 2024-09-16 20:11:57 2024-09-17 11:23:39 2024-09-17 11:53:10 0:29:31 0:20:41 0:08:50 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_cephadm_timeout} 1
pass 7907780 2024-09-16 20:11:58 2024-09-17 11:23:39 2024-09-17 11:56:52 0:33:13 0:22:38 0:10:35 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7907781 2024-09-16 20:11:59 2024-09-17 11:24:10 2024-09-17 12:01:40 0:37:30 0:27:33 0:09:57 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-squid 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
pass 7907782 2024-09-16 20:12:00 2024-09-17 11:24:30 2024-09-17 12:01:36 0:37:06 0:24:56 0:12:10 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7907783 2024-09-16 20:12:01 2024-09-17 11:28:01 2024-09-17 11:53:29 0:25:28 0:14:15 0:11:13 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7907784 2024-09-16 20:12:03 2024-09-17 11:28:41 2024-09-17 11:51:41 0:23:00 0:12:44 0:10:16 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
pass 7907785 2024-09-16 20:12:04 2024-09-17 11:28:42 2024-09-17 12:06:19 0:37:37 0:26:20 0:11:17 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_ctdb_res_basic} 4
pass 7907786 2024-09-16 20:12:05 2024-09-17 11:30:12 2024-09-17 11:56:03 0:25:51 0:15:00 0:10:51 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain} 3
pass 7907787 2024-09-16 20:12:07 2024-09-17 11:31:03 2024-09-17 11:55:12 0:24:09 0:13:37 0:10:32 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
pass 7907788 2024-09-16 20:12:08 2024-09-17 11:32:04 2024-09-17 12:15:02 0:42:58 0:32:29 0:10:29 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/classic msgr/async start tasks/rados_python} 2
pass 7907789 2024-09-16 20:12:09 2024-09-17 11:32:44 2024-09-17 12:04:15 0:31:31 0:21:37 0:09:54 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
fail 7907790 2024-09-16 20:12:10 2024-09-17 11:32:44 2024-09-17 11:59:28 0:26:44 0:15:43 0:11:01 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_ctdb_res_dom} 4
Failure Reason:

"2024-09-17T11:57:54.236182+0000 mon.a (mon.0) 791 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7907791 2024-09-16 20:12:11 2024-09-17 11:34:05 2024-09-17 12:16:28 0:42:23 0:28:28 0:13:55 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
pass 7907792 2024-09-16 20:12:13 2024-09-17 11:34:25 2024-09-17 11:56:09 0:21:44 0:12:17 0:09:27 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7907793 2024-09-16 20:12:14 2024-09-17 11:35:16 2024-09-17 12:16:28 0:41:12 0:31:20 0:09:52 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
pass 7907794 2024-09-16 20:12:15 2024-09-17 11:35:26 2024-09-17 12:18:28 0:43:02 0:32:57 0:10:05 smithi main centos 9.stream orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
pass 7907795 2024-09-16 20:12:16 2024-09-17 11:35:27 2024-09-17 12:18:24 0:42:57 0:33:54 0:09:03 smithi main centos 9.stream orch/cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignore_mgr_down ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} 1
pass 7907796 2024-09-16 20:12:17 2024-09-17 11:35:27 2024-09-17 11:58:27 0:23:00 0:12:51 0:10:09 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} 1
pass 7907797 2024-09-16 20:12:19 2024-09-17 11:35:27 2024-09-17 12:01:13 0:25:46 0:14:48 0:10:58 smithi main ubuntu 22.04 orch/cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} 2
pass 7907798 2024-09-16 20:12:20 2024-09-17 11:37:39 2024-09-17 12:09:21 0:31:42 0:20:51 0:10:51 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
fail 7907799 2024-09-16 20:12:21 2024-09-17 11:37:49 2024-09-17 12:11:21 0:33:32 0:22:29 0:11:03 smithi main centos 9.stream orch/cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-09-17T11:53:59.021130+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7907800 2024-09-16 20:12:22 2024-09-17 11:38:19 2024-09-17 12:04:00 0:25:41 0:15:22 0:10:19 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
pass 7907801 2024-09-16 20:12:24 2024-09-17 11:39:00 2024-09-17 12:08:36 0:29:36 0:20:00 0:09:36 smithi main ubuntu 22.04 orch/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} 1
pass 7907802 2024-09-16 20:12:25 2024-09-17 11:39:01 2024-09-17 12:00:16 0:21:15 0:10:30 0:10:45 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
pass 7907803 2024-09-16 20:12:26 2024-09-17 11:39:12 2024-09-17 13:04:52 1:25:40 1:16:26 0:09:14 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-reef 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
fail 7907804 2024-09-16 20:12:27 2024-09-17 11:39:22 2024-09-17 12:07:18 0:27:56 0:16:33 0:11:23 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_ctdb_res_ips} 4
Failure Reason:

SELinux denials found on ubuntu@smithi077.front.sepia.ceph.com: ['type=AVC msg=audit(1726574577.536:10852): avc: denied { nlmsg_read } for pid=60353 comm="ss" scontext=system_u:system_r:container_t:s0:c26,c385 tcontext=system_u:system_r:container_t:s0:c26,c385 tclass=netlink_tcpdiag_socket permissive=1']

pass 7907805 2024-09-16 20:12:28 2024-09-17 11:40:13 2024-09-17 12:05:56 0:25:43 0:15:20 0:10:23 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
fail 7907806 2024-09-16 20:12:30 2024-09-17 11:40:43 2024-09-17 12:09:49 0:29:06 0:17:32 0:11:34 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
Failure Reason:

Command failed on smithi145 with status 5: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1682d544-74ec-11ef-bceb-c7b262605968 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v1/status\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts | jq -e \'"\'"\'.data | .[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n\''

pass 7907807 2024-09-16 20:12:31 2024-09-17 11:42:44 2024-09-17 12:13:04 0:30:20 0:20:45 0:09:35 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
pass 7907808 2024-09-16 20:12:32 2024-09-17 11:42:44 2024-09-17 12:07:50 0:25:06 0:14:31 0:10:35 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7907809 2024-09-16 20:12:33 2024-09-17 11:42:55 2024-09-17 12:06:03 0:23:08 0:12:56 0:10:12 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7907810 2024-09-16 20:12:34 2024-09-17 11:43:05 2024-09-17 12:12:25 0:29:20 0:20:23 0:08:57 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_domain} 2
pass 7907811 2024-09-16 20:12:36 2024-09-17 11:43:05 2024-09-17 12:06:50 0:23:45 0:13:34 0:10:11 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7907812 2024-09-16 20:12:37 2024-09-17 11:43:46 2024-09-17 12:17:05 0:33:19 0:22:29 0:10:50 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/connectivity msgr/async-v1only start tasks/rotate-keys} 2
pass 7907813 2024-09-16 20:12:38 2024-09-17 11:44:06 2024-09-17 12:42:05 0:57:59 0:48:17 0:09:42 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} 2
fail 7907814 2024-09-16 20:12:39 2024-09-17 11:44:37 2024-09-17 12:06:58 0:22:21 0:12:15 0:10:06 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_rgw_multisite} 3
Failure Reason:

Command failed on smithi017 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7498493279982dfe87f00616198a5967475f6169 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 890abb68-74ec-11ef-bceb-c7b262605968 -- bash -c \'set -e\nset -x\nwhile true; do TOKEN=$(ceph rgw realm tokens | jq -r \'"\'"\'.[0].token\'"\'"\'); echo $TOKEN; if [ "$TOKEN" != "master zone has no endpoint" ]; then break; fi; sleep 5; done\nTOKENS=$(ceph rgw realm tokens)\necho $TOKENS | jq --exit-status \'"\'"\'.[0].realm == "myrealm1"\'"\'"\'\necho $TOKENS | jq --exit-status \'"\'"\'.[0].token\'"\'"\'\nTOKEN_JSON=$(ceph rgw realm tokens | jq -r \'"\'"\'.[0].token\'"\'"\' | base64 --decode)\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.realm_name == "myrealm1"\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.endpoint | test("http://.+:\\\\d+")\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.realm_id | test("^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.access_key\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.secret\'"\'"\'\n\''

fail 7907815 2024-09-16 20:12:40 2024-09-17 11:45:27 2024-09-17 12:35:03 0:49:36 0:38:03 0:11:33 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 7907816 2024-09-16 20:12:42 2024-09-17 11:48:08 2024-09-17 12:22:32 0:34:24 0:22:29 0:11:55 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} 5
pass 7907817 2024-09-16 20:12:43 2024-09-17 11:48:29 2024-09-17 12:11:39 0:23:10 0:12:43 0:10:27 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7907818 2024-09-16 20:12:44 2024-09-17 11:48:39 2024-09-17 12:44:00 0:55:21 0:43:27 0:11:54 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-squid 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
pass 7907819 2024-09-16 20:12:45 2024-09-17 11:49:50 2024-09-17 12:10:39 0:20:49 0:09:49 0:11:00 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_res_basic} 2
pass 7907820 2024-09-16 20:12:47 2024-09-17 11:51:40 2024-09-17 12:21:31 0:29:51 0:20:06 0:09:45 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
pass 7907821 2024-09-16 20:12:48 2024-09-17 11:52:01 2024-09-17 12:16:39 0:24:38 0:12:54 0:11:44 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7907822 2024-09-16 20:12:49 2024-09-17 11:53:41 2024-09-17 12:29:16 0:35:35 0:22:33 0:13:02 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
dead 7907823 2024-09-16 20:12:50 2024-09-17 11:55:22 2024-09-17 20:04:56 8:09:34 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

fail 7907824 2024-09-16 20:12:51 2024-09-17 11:56:12 2024-09-17 12:14:17 0:18:05 0:07:54 0:10:11 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none cluster/3-node k8s/1.21 net/host rook/1.7.2} 3
Failure Reason:

Command failed on smithi079 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7907825 2024-09-16 20:12:53 2024-09-17 11:56:43 2024-09-17 12:18:28 0:21:45 0:10:47 0:10:58 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
pass 7907826 2024-09-16 20:12:54 2024-09-17 11:57:13 2024-09-17 12:20:03 0:22:50 0:11:06 0:11:44 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_res_dom} 2
pass 7907827 2024-09-16 20:12:55 2024-09-17 11:59:04 2024-09-17 12:22:27 0:23:23 0:14:12 0:09:11 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
fail 7907828 2024-09-16 20:12:56 2024-09-17 11:59:04 2024-09-17 12:46:41 0:47:37 0:37:54 0:09:43 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 7907829 2024-09-16 20:12:57 2024-09-17 11:59:45 2024-09-17 12:23:25 0:23:40 0:13:23 0:10:17 smithi main ubuntu 22.04 orch/cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} 1
fail 7907830 2024-09-16 20:12:59 2024-09-17 11:59:45 2024-09-17 12:23:20 0:23:35 0:14:28 0:09:07 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

reached maximum tries (120) after waiting for 120 seconds

pass 7907831 2024-09-16 20:13:00 2024-09-17 12:00:16 2024-09-17 12:21:20 0:21:04 0:12:09 0:08:55 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} 2
pass 7907832 2024-09-16 20:13:01 2024-09-17 12:00:16 2024-09-17 12:26:16 0:26:00 0:13:46 0:12:14 smithi main centos 9.stream orch/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7907833 2024-09-16 20:13:02 2024-09-17 12:00:16 2024-09-17 12:28:53 0:28:37 0:19:07 0:09:30 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
pass 7907834 2024-09-16 20:13:03 2024-09-17 12:01:27 2024-09-17 12:39:03 0:37:36 0:27:30 0:10:06 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
pass 7907835 2024-09-16 20:13:05 2024-09-17 12:01:57 2024-09-17 12:45:13 0:43:16 0:32:54 0:10:22 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} 2
pass 7907836 2024-09-16 20:13:06 2024-09-17 12:01:58 2024-09-17 12:31:58 0:30:00 0:17:31 0:12:29 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_ca_signed_key} 2
pass 7907837 2024-09-16 20:13:07 2024-09-17 12:04:19 2024-09-17 12:25:18 0:20:59 0:11:56 0:09:03 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
pass 7907838 2024-09-16 20:13:08 2024-09-17 12:04:29 2024-09-17 12:36:05 0:31:36 0:18:55 0:12:41 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} 2
pass 7907839 2024-09-16 20:13:09 2024-09-17 12:04:59 2024-09-17 12:29:28 0:24:29 0:12:33 0:11:56 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
pass 7907840 2024-09-16 20:13:10 2024-09-17 12:06:10 2024-09-17 12:29:23 0:23:13 0:13:46 0:09:27 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm} 1
pass 7907841 2024-09-16 20:13:12 2024-09-17 12:06:10 2024-09-17 12:35:54 0:29:44 0:19:16 0:10:28 smithi main ubuntu 22.04 orch/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7907842 2024-09-16 20:13:13 2024-09-17 12:06:21 2024-09-17 12:38:11 0:31:50 0:21:43 0:10:07 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
pass 7907843 2024-09-16 20:13:14 2024-09-17 12:06:31 2024-09-17 12:27:44 0:21:13 0:10:16 0:10:57 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_basic} 2
fail 7907844 2024-09-16 20:13:15 2024-09-17 12:06:31 2024-09-17 15:35:01 3:28:30 3:18:22 0:10:08 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi167 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=2b698643d05f73e79c2d2888228947a84a4ac0bd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

pass 7907845 2024-09-16 20:13:17 2024-09-17 12:07:12 2024-09-17 12:29:48 0:22:36 0:12:55 0:09:41 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} 1
pass 7907846 2024-09-16 20:13:18 2024-09-17 12:07:12 2024-09-17 12:28:10 0:20:58 0:12:21 0:08:37 smithi main centos 9.stream orch/cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli} 2
pass 7907847 2024-09-16 20:13:19 2024-09-17 12:07:12 2024-09-17 12:30:37 0:23:25 0:12:46 0:10:39 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7907848 2024-09-16 20:13:20 2024-09-17 12:07:33 2024-09-17 12:28:31 0:20:58 0:11:47 0:09:11 smithi main centos 9.stream orch/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
pass 7907849 2024-09-16 20:13:21 2024-09-17 12:07:33 2024-09-17 12:27:04 0:19:31 0:10:20 0:09:11 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
fail 7907850 2024-09-16 20:13:23 2024-09-17 12:08:04 2024-09-17 12:59:51 0:51:47 0:40:42 0:11:05 smithi main centos 9.stream orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-squid 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

Command failed on smithi084 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 49535086-74ef-11ef-bceb-c7b262605968 -e sha1=7498493279982dfe87f00616198a5967475f6169 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''

pass 7907851 2024-09-16 20:13:24 2024-09-17 12:08:54 2024-09-17 12:24:09 0:15:15 0:05:39 0:09:36 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos} 1
pass 7907852 2024-09-16 20:13:25 2024-09-17 12:24:21 2024-09-17 13:32:29 1:08:08 0:58:28 0:09:40 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} 2
pass 7907853 2024-09-16 20:13:26 2024-09-17 12:25:32 2024-09-17 12:58:34 0:33:02 0:21:18 0:11:44 smithi main centos 9.stream orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python} 2
pass 7907854 2024-09-16 20:13:27 2024-09-17 12:26:32 2024-09-17 12:50:26 0:23:54 0:13:25 0:10:29 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7907855 2024-09-16 20:13:28 2024-09-17 12:27:23 2024-09-17 12:51:16 0:23:53 0:12:19 0:11:34 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
fail 7907856 2024-09-16 20:13:30 2024-09-17 12:28:03 2024-09-17 12:53:41 0:25:38 0:14:54 0:10:44 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_ctdb_res_basic} 4
Failure Reason:

"2024-09-17T12:51:18.246519+0000 mon.a (mon.0) 825 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

pass 7907857 2024-09-16 20:13:31 2024-09-17 12:28:44 2024-09-17 13:04:38 0:35:54 0:25:48 0:10:06 smithi main ubuntu 22.04 orch/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7907858 2024-09-16 20:13:32 2024-09-17 12:29:04 2024-09-17 13:01:21 0:32:17 0:21:55 0:10:22 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
pass 7907859 2024-09-16 20:13:33 2024-09-17 12:29:35 2024-09-17 12:59:26 0:29:51 0:19:43 0:10:08 smithi main ubuntu 22.04 orch/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_extra_daemon_features} 2
pass 7907860 2024-09-16 20:13:35 2024-09-17 12:29:35 2024-09-17 12:53:04 0:23:29 0:13:27 0:10:02 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
pass 7907861 2024-09-16 20:13:36 2024-09-17 12:29:46 2024-09-17 13:09:19 0:39:33 0:27:25 0:12:08 smithi main ubuntu 22.04 orch/cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_ctdb_res_dom} 4
fail 7907862 2024-09-16 20:13:37 2024-09-17 12:31:36 2024-09-17 13:22:19 0:50:43 0:38:33 0:12:10 smithi main centos 9.stream orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 7907863 2024-09-16 20:13:38 2024-09-17 12:35:17 2024-09-17 12:58:21 0:23:04 0:13:24 0:09:40 smithi main centos 9.stream orch/cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} 1
pass 7907864 2024-09-16 20:13:39 2024-09-17 12:35:17 2024-09-17 12:59:06 0:23:49 0:12:12 0:11:37 smithi main centos 9.stream orch/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7907865 2024-09-16 20:13:40 2024-09-17 12:36:08 2024-09-17 13:04:44 0:28:36 0:14:57 0:13:39 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
pass 7907866 2024-09-16 20:13:42 2024-09-17 12:36:18 2024-09-17 13:34:03 0:57:45 0:44:57 0:12:48 smithi main ubuntu 22.04 orch/cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
pass 7907867 2024-09-16 20:13:43 2024-09-17 12:38:29 2024-09-17 13:04:44 0:26:15 0:14:39 0:11:36 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_host_drain} 3
pass 7907868 2024-09-16 20:13:44 2024-09-17 12:39:30 2024-09-17 13:18:22 0:38:52 0:28:26 0:10:26 smithi main centos 9.stream orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
dead 7907869 2024-09-16 20:13:46 2024-09-17 12:39:30 2024-09-17 12:49:35 0:10:05 smithi main centos 9.stream orch/cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_ctdb_res_ips} 4
Failure Reason:

SSH connection to smithi179 was lost: 'sudo yum install -y kernel'

pass 7907870 2024-09-16 20:13:47 2024-09-17 12:42:01 2024-09-17 13:15:59 0:33:58 0:24:01 0:09:57 smithi main ubuntu 22.04 orch/cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7907871 2024-09-16 20:13:48 2024-09-17 12:42:01 2024-09-17 13:31:33 0:49:32 0:40:16 0:09:16 smithi main ubuntu 22.04 orch/cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rotate-keys} 2
pass 7907872 2024-09-16 20:13:49 2024-09-17 12:42:02 2024-09-17 13:05:12 0:23:10 0:13:19 0:09:51 smithi main centos 9.stream orch/cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
pass 7907873 2024-09-16 20:13:51 2024-09-17 12:42:12 2024-09-17 13:03:47 0:21:35 0:10:43 0:10:52 smithi main centos 9.stream orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
fail 7907874 2024-09-16 20:13:52 2024-09-17 12:42:43 2024-09-17 13:00:16 0:17:33 0:07:51 0:09:42 smithi main ubuntu 22.04 orch/rook/smoke/{0-distro/ubuntu_22.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench cluster/1-node k8s/1.21 net/calico rook/master} 1
Failure Reason:

Command failed on smithi086 with status 100: "sudo apt update && sudo apt install -y apt-transport-https ca-certificates curl && sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && echo 'deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list && sudo apt update && sudo apt install -y kubelet kubeadm kubectl bridge-utils"

pass 7907875 2024-09-16 20:13:53 2024-09-17 12:42:43 2024-09-17 13:03:07 0:20:24 0:12:44 0:07:40 smithi main centos 9.stream orch/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1