Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7645883 2024-04-08 19:16:03 2024-04-08 19:19:09 2024-04-08 19:46:24 0:27:15 0:13:38 0:13:37 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-04-08T19:42:23.881306+0000 mon.smithi063 (mon.0) 782 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7645884 2024-04-08 19:16:05 2024-04-08 19:19:30 2024-04-08 20:07:40 0:48:10 0:37:28 0:10:42 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7645885 2024-04-08 19:16:06 2024-04-08 19:19:50 2024-04-08 20:27:34 1:07:44 0:57:25 0:10:19 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-04-08T19:38:37.950462+0000 mon.a (mon.0) 172 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

dead 7645886 2024-04-08 19:16:08 2024-04-08 19:19:50 2024-04-09 07:28:08 12:08:18 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
Failure Reason:

hit max job timeout

fail 7645887 2024-04-08 19:16:09 2024-04-08 19:19:51 2024-04-08 19:54:15 0:34:24 0:21:56 0:12:28 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-04-08T19:45:05.626428+0000 mon.smithi103 (mon.0) 686 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 7645888 2024-04-08 19:16:11 2024-04-08 19:22:01 2024-04-08 20:10:07 0:48:06 0:34:39 0:13:27 smithi main centos 9.stream orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
Failure Reason:

"2024-04-08T19:43:47.357875+0000 mon.smithi113 (mon.0) 494 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7645889 2024-04-08 19:16:12 2024-04-08 19:24:32 2024-04-08 20:20:41 0:56:09 0:46:38 0:09:31 smithi main ubuntu 22.04 orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{ubuntu_latest} tasks/nfs} 1
Failure Reason:

"2024-04-08T19:50:02.911275+0000 mon.a (mon.0) 321 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)" in cluster log

fail 7645890 2024-04-08 19:16:14 2024-04-08 19:25:02 2024-04-08 19:49:30 0:24:28 0:14:14 0:10:14 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-04-08T19:45:58.885355+0000 mon.a (mon.0) 463 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 7645891 2024-04-08 19:16:15 2024-04-08 19:25:03 2024-04-08 19:47:55 0:22:52 0:12:26 0:10:26 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/off orchestrator_cli} 2
fail 7645892 2024-04-08 19:16:17 2024-04-08 19:25:03 2024-04-08 20:05:44 0:40:41 0:27:56 0:12:45 smithi main centos 9.stream orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-04-08T19:45:02.875015+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7645893 2024-04-08 19:16:18 2024-04-08 19:28:34 2024-04-08 19:49:51 0:21:17 0:11:42 0:09:35 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_domain} 2
Failure Reason:

"2024-04-08T19:46:13.047870+0000 mon.a (mon.0) 251 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

dead 7645894 2024-04-08 19:16:20 2024-04-08 19:28:34 2024-04-09 07:38:12 12:09:38 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

hit max job timeout

pass 7645895 2024-04-08 19:16:21 2024-04-08 19:29:35 2024-04-08 19:58:47 0:29:12 0:18:43 0:10:29 smithi main ubuntu 22.04 orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} 1
fail 7645896 2024-04-08 19:16:23 2024-04-08 19:30:25 2024-04-08 19:56:02 0:25:37 0:11:59 0:13:38 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-04-08T19:50:20.943896+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 7645897 2024-04-08 19:16:25 2024-04-08 19:32:56 2024-04-08 19:49:55 0:16:59 0:06:39 0:10:20 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} 1
fail 7645898 2024-04-08 19:16:26 2024-04-08 19:32:56 2024-04-08 20:02:02 0:29:06 0:16:43 0:12:23 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-04-08T19:55:54.106915+0000 mon.smithi148 (mon.0) 777 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645899 2024-04-08 19:16:28 2024-04-08 19:34:47 2024-04-08 20:07:38 0:32:51 0:18:48 0:14:03 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-04-08T20:03:17.116477+0000 mon.smithi133 (mon.0) 764 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7645900 2024-04-08 19:16:29 2024-04-08 19:38:08 2024-04-08 20:13:40 0:35:32 0:22:36 0:12:56 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-04-08T20:07:34.828646+0000 mon.smithi049 (mon.0) 770 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645901 2024-04-08 19:16:31 2024-04-08 19:38:08 2024-04-08 23:32:43 3:54:35 3:40:02 0:14:33 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi087 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fb6f9800392d73c10dc835c8e8e86c5884686270 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

fail 7645902 2024-04-08 19:16:32 2024-04-08 19:39:20 2024-04-08 20:20:58 0:41:38 0:31:28 0:10:10 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-04-08T20:04:33.431512+0000 mon.a (mon.0) 851 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log

fail 7645903 2024-04-08 19:16:34 2024-04-08 19:40:11 2024-04-08 20:27:15 0:47:04 0:37:10 0:09:54 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2
Failure Reason:

"2024-04-08T20:07:37.534095+0000 mon.a (mon.0) 1169 : cluster [WRN] Health check failed: 2 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

dead 7645904 2024-04-08 19:16:35 2024-04-08 19:40:11 2024-04-08 19:41:25 0:01:14 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi023

dead 7645905 2024-04-08 19:16:37 2024-04-08 19:40:22 2024-04-08 19:41:26 0:01:04 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi154

fail 7645906 2024-04-08 19:16:38 2024-04-08 19:40:22 2024-04-08 20:22:36 0:42:14 0:27:48 0:14:26 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} 5
Failure Reason:

"2024-04-08T20:08:29.633886+0000 mon.a (mon.0) 271 : cluster [WRN] Health detail: HEALTH_WARN 1/4 mons down, quorum a,e,c" in cluster log

fail 7645907 2024-04-08 19:16:40 2024-04-08 19:41:33 2024-04-08 20:27:58 0:46:25 0:35:39 0:10:46 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_res_dom} 2
Failure Reason:

reached maximum tries (301) after waiting for 300 seconds

pass 7645908 2024-04-08 19:16:41 2024-04-08 19:42:13 2024-04-08 20:07:44 0:25:31 0:14:25 0:11:06 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
fail 7645909 2024-04-08 19:16:43 2024-04-08 19:44:44 2024-04-08 20:20:11 0:35:27 0:26:25 0:09:02 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
Failure Reason:

"2024-04-08T20:06:12.353635+0000 mon.a (mon.0) 395 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7645910 2024-04-08 19:16:45 2024-04-08 19:44:44 2024-04-08 20:09:25 0:24:41 0:14:59 0:09:42 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-04-08T20:05:17.823802+0000 mon.smithi055 (mon.0) 782 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7645911 2024-04-08 19:16:46 2024-04-08 19:44:45 2024-04-08 20:10:55 0:26:10 0:16:14 0:09:56 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} 3
Failure Reason:

"2024-04-08T20:07:08.493633+0000 mon.a (mon.0) 615 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7645912 2024-04-08 19:16:48 2024-04-08 19:45:45 2024-04-08 20:19:38 0:33:53 0:20:46 0:13:07 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-04-08T20:13:57.834835+0000 mon.smithi105 (mon.0) 760 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645913 2024-04-08 19:16:49 2024-04-08 19:47:56 2024-04-08 20:30:32 0:42:36 0:33:01 0:09:35 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1712607336.505119 mon.smithi026 (mon.0) 954 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

pass 7645914 2024-04-08 19:16:51 2024-04-08 19:48:06 2024-04-08 20:17:11 0:29:05 0:17:49 0:11:16 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
fail 7645915 2024-04-08 19:16:53 2024-04-08 19:49:57 2024-04-08 20:15:41 0:25:44 0:12:16 0:13:28 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
Failure Reason:

"2024-04-08T20:11:33.141452+0000 mon.a (mon.0) 386 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7645916 2024-04-08 19:16:54 2024-04-08 19:54:08 2024-04-08 20:51:44 0:57:36 0:47:54 0:09:42 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} 2
Failure Reason:

"2024-04-08T20:09:56.424693+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7645917 2024-04-08 19:16:56 2024-04-08 19:54:08 2024-04-08 20:20:55 0:26:47 0:15:50 0:10:57 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
Failure Reason:

"2024-04-08T20:16:16.295858+0000 mon.smithi050 (mon.0) 795 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645918 2024-04-08 19:16:58 2024-04-08 19:55:59 2024-04-08 20:40:46 0:44:47 0:35:37 0:09:10 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-04-08T20:18:20.873374+0000 mon.a (mon.0) 745 : cluster [WRN] Health check failed: Degraded data redundancy: 3/594 objects degraded (0.505%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 7645919 2024-04-08 19:16:59 2024-04-08 19:55:59 2024-04-08 20:29:00 0:33:01 0:23:17 0:09:44 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} 2
fail 7645920 2024-04-08 19:17:01 2024-04-08 19:55:59 2024-04-08 20:24:25 0:28:26 0:14:35 0:13:51 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
Failure Reason:

"2024-04-08T20:20:32.437970+0000 mon.a (mon.0) 299 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

pass 7645921 2024-04-08 19:17:02 2024-04-08 19:58:50 2024-04-08 20:24:59 0:26:09 0:12:27 0:13:42 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} 1
fail 7645922 2024-04-08 19:17:04 2024-04-08 20:02:51 2024-04-08 20:30:45 0:27:54 0:12:56 0:14:58 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

"2024-04-08T20:27:43.212309+0000 mon.smithi031 (mon.0) 795 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7645923 2024-04-08 19:17:05 2024-04-08 20:07:52 2024-04-08 20:36:57 0:29:05 0:18:51 0:10:14 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_basic} 2
Failure Reason:

"2024-04-08T20:33:42.509787+0000 mon.a (mon.0) 246 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645924 2024-04-08 19:17:07 2024-04-08 20:09:12 2024-04-08 20:38:34 0:29:22 0:18:55 0:10:27 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
Failure Reason:

"2024-04-08T20:31:27.585717+0000 mon.smithi028 (mon.0) 674 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 7645925 2024-04-08 19:17:08 2024-04-08 20:09:13 2024-04-08 20:34:50 0:25:37 0:13:41 0:11:56 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
fail 7645926 2024-04-08 19:17:10 2024-04-08 20:11:23 2024-04-08 20:47:32 0:36:09 0:25:18 0:10:51 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
Failure Reason:

"2024-04-08T20:35:25.765937+0000 mon.a (mon.0) 432 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7645927 2024-04-08 19:17:12 2024-04-08 20:11:24 2024-04-08 21:28:15 1:16:51 1:07:46 0:09:05 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7645928 2024-04-08 19:17:13 2024-04-08 20:11:24 2024-04-08 20:34:29 0:23:05 0:14:12 0:08:53 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
Failure Reason:

"2024-04-08T20:29:25.593601+0000 mon.smithi092 (mon.0) 598 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645929 2024-04-08 19:17:15 2024-04-08 20:11:24 2024-04-08 20:36:21 0:24:57 0:15:25 0:09:32 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
Failure Reason:

"2024-04-08T20:27:52.185874+0000 mon.a (mon.0) 420 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7645930 2024-04-08 19:17:17 2024-04-08 20:11:25 2024-04-08 20:43:04 0:31:39 0:21:14 0:10:25 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
Failure Reason:

"2024-04-08T20:33:50.744507+0000 mon.smithi055 (mon.0) 591 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645931 2024-04-08 19:17:18 2024-04-08 20:11:25 2024-04-08 20:40:47 0:29:22 0:19:40 0:09:42 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
Failure Reason:

"2024-04-08T20:37:06.674123+0000 mon.smithi017 (mon.0) 764 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7645932 2024-04-08 19:17:20 2024-04-08 20:11:25 2024-04-08 21:09:13 0:57:48 0:46:59 0:10:49 smithi main ubuntu 22.04 orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
Failure Reason:

"2024-04-08T20:33:41.493113+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 7645933 2024-04-08 19:17:21 2024-04-08 20:11:26 2024-04-08 21:03:17 0:51:51 0:39:06 0:12:45 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
Failure Reason:

"2024-04-08T20:34:52.995119+0000 mon.a (mon.0) 200 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 7645934 2024-04-08 19:17:23 2024-04-08 20:13:26 2024-04-08 20:42:24 0:28:58 0:14:23 0:14:35 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} 3
Failure Reason:

"2024-04-08T20:36:51.971072+0000 mon.a (mon.0) 209 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c" in cluster log

fail 7645935 2024-04-08 19:17:24 2024-04-08 20:17:17 2024-04-08 20:39:51 0:22:34 0:14:32 0:08:02 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} 1
Failure Reason:

"2024-04-08T20:38:47.087630+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log

pass 7645936 2024-04-08 19:17:26 2024-04-08 20:17:17 2024-04-08 20:39:35 0:22:18 0:12:38 0:09:40 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} 2
fail 7645937 2024-04-08 19:17:27 2024-04-08 20:18:38 2024-04-08 20:40:48 0:22:10 0:11:53 0:10:17 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_domain} 2
Failure Reason:

"2024-04-08T20:37:38.621032+0000 mon.a (mon.0) 242 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

pass 7645938 2024-04-08 19:17:29 2024-04-08 20:19:59 2024-04-08 20:48:33 0:28:34 0:14:29 0:14:05 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7645939 2024-04-08 19:17:31 2024-04-08 20:23:10 2024-04-08 20:44:15 0:21:05 0:12:23 0:08:42 smithi main centos 9.stream orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final} 1
fail 7645940 2024-04-08 19:17:32 2024-04-08 20:23:10 2024-04-08 20:47:44 0:24:34 0:12:19 0:12:15 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
Failure Reason:

"2024-04-08T20:41:34.393305+0000 mon.a (mon.0) 265 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7645941 2024-04-08 19:17:34 2024-04-08 20:24:40 2024-04-08 20:49:58 0:25:18 0:15:37 0:09:41 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
Failure Reason:

"2024-04-08T20:45:27.994680+0000 mon.smithi106 (mon.0) 810 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645942 2024-04-08 19:17:36 2024-04-08 20:24:41 2024-04-08 21:10:23 0:45:42 0:31:59 0:13:43 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1712609741.41841 mon.smithi043 (mon.0) 948 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7645943 2024-04-08 19:17:37 2024-04-08 20:26:51 2024-04-08 21:17:27 0:50:36 0:39:19 0:11:17 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
Failure Reason:

"2024-04-08T20:46:27.056768+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7645944 2024-04-08 19:17:40 2024-04-08 20:26:52 2024-04-08 20:55:06 0:28:14 0:16:17 0:11:57 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
Failure Reason:

"2024-04-08T20:48:47.511901+0000 mon.a (mon.0) 424 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7645945 2024-04-08 19:17:41 2024-04-08 20:26:52 2024-04-08 20:59:32 0:32:40 0:21:58 0:10:42 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
Failure Reason:

"2024-04-08T20:54:32.833595+0000 mon.smithi064 (mon.0) 864 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645946 2024-04-08 19:17:43 2024-04-08 20:26:53 2024-04-08 20:49:36 0:22:43 0:13:48 0:08:55 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

"2024-04-08T20:47:22.733116+0000 mon.smithi114 (mon.0) 792 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7645947 2024-04-08 19:17:44 2024-04-08 20:26:53 2024-04-08 20:54:27 0:27:34 0:15:37 0:11:57 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
Failure Reason:

"2024-04-08T20:48:37.161350+0000 mon.smithi006 (mon.0) 816 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645948 2024-04-08 19:17:46 2024-04-08 20:26:53 2024-04-08 20:51:31 0:24:38 0:14:21 0:10:17 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} 1
Failure Reason:

"2024-04-08T20:48:18.252795+0000 mon.a (mon.0) 463 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7645949 2024-04-08 19:17:47 2024-04-08 20:26:54 2024-04-08 20:47:53 0:20:59 0:12:00 0:08:59 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_domain} 2
Failure Reason:

"2024-04-08T20:44:52.260216+0000 mon.a (mon.0) 244 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645950 2024-04-08 19:17:49 2024-04-08 20:26:54 2024-04-08 20:51:51 0:24:57 0:15:13 0:09:44 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
Failure Reason:

"2024-04-08T20:47:10.216633+0000 mon.a (mon.0) 671 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645951 2024-04-08 19:17:51 2024-04-08 20:26:54 2024-04-08 20:53:58 0:27:04 0:17:19 0:09:45 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

"2024-04-08T20:47:26.578308+0000 mon.smithi152 (mon.0) 868 : cluster [WRN] Health check failed: Failed to place 2 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

fail 7645952 2024-04-08 19:17:52 2024-04-08 20:26:55 2024-04-08 21:40:09 1:13:14 1:04:14 0:09:00 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
Failure Reason:

"2024-04-08T20:50:00.000188+0000 mon.a (mon.0) 704 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log

fail 7645953 2024-04-08 19:17:54 2024-04-08 20:26:55 2024-04-08 21:24:08 0:57:13 0:44:44 0:12:29 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} 2
Failure Reason:

"2024-04-08T21:10:00.000166+0000 mon.a (mon.0) 2593 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log

pass 7645954 2024-04-08 19:17:55 2024-04-08 20:26:55 2024-04-08 20:50:46 0:23:51 0:12:46 0:11:05 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} 2
fail 7645955 2024-04-08 19:17:57 2024-04-08 20:27:16 2024-04-08 21:01:37 0:34:21 0:23:57 0:10:24 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
Failure Reason:

"2024-04-08T20:54:07.425829+0000 mon.smithi044 (mon.0) 750 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645956 2024-04-08 19:17:58 2024-04-08 20:28:06 2024-04-08 21:45:24 1:17:18 1:07:25 0:09:53 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7645957 2024-04-08 19:18:00 2024-04-08 20:28:07 2024-04-08 20:52:22 0:24:15 0:13:43 0:10:32 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

"2024-04-08T20:49:44.595751+0000 mon.smithi186 (mon.0) 783 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7645958 2024-04-08 19:18:01 2024-04-08 20:29:07 2024-04-08 20:55:48 0:26:41 0:15:39 0:11:02 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
Failure Reason:

"2024-04-08T20:52:00.580645+0000 mon.smithi007 (mon.0) 838 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

dead 7645959 2024-04-08 19:18:03 2024-04-08 20:30:58 2024-04-09 08:43:52 12:12:54 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
Failure Reason:

hit max job timeout

fail 7645960 2024-04-08 19:18:05 2024-04-08 20:34:59 2024-04-08 21:03:07 0:28:08 0:17:23 0:10:45 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} 1
Failure Reason:

Command failed (workunit test cephadm/test_cephadm.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fb6f9800392d73c10dc835c8e8e86c5884686270 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'

fail 7645961 2024-04-08 19:18:06 2024-04-08 20:35:39 2024-04-08 21:04:51 0:29:12 0:14:17 0:14:55 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
Failure Reason:

"2024-04-08T21:00:32.955167+0000 mon.smithi150 (mon.0) 785 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log

fail 7645962 2024-04-08 19:18:08 2024-04-08 20:39:40 2024-04-08 21:30:09 0:50:29 0:36:44 0:13:45 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} 5
Failure Reason:

"2024-04-08T21:06:40.341079+0000 mon.a (mon.0) 277 : cluster [WRN] Health detail: HEALTH_WARN 1/4 mons down, quorum a,e,c" in cluster log

fail 7645963 2024-04-08 19:18:09 2024-04-08 20:42:21 2024-04-08 21:57:56 1:15:35 1:04:52 0:10:43 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_dom} 2
Failure Reason:

reached maximum tries (301) after waiting for 300 seconds

fail 7645964 2024-04-08 19:18:11 2024-04-08 20:42:21 2024-04-08 21:12:20 0:29:59 0:20:41 0:09:18 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
Failure Reason:

"2024-04-08T21:05:09.464957+0000 mon.smithi026 (mon.0) 576 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645965 2024-04-08 19:18:12 2024-04-08 20:42:22 2024-04-08 21:23:11 0:40:49 0:31:44 0:09:05 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-04-08T21:07:08.682004+0000 mon.a (mon.0) 867 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS)" in cluster log

fail 7645966 2024-04-08 19:18:14 2024-04-08 20:42:22 2024-04-08 21:15:21 0:32:59 0:23:29 0:09:30 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} 2
Failure Reason:

"2024-04-08T21:10:00.000185+0000 mon.a (mon.0) 1101 : cluster [WRN] Health detail: HEALTH_WARN 2 pool(s) do not have an application enabled" in cluster log

pass 7645967 2024-04-08 19:18:16 2024-04-08 20:42:22 2024-04-08 20:57:14 0:14:52 0:06:31 0:08:21 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
fail 7645968 2024-04-08 19:18:17 2024-04-08 20:42:23 2024-04-08 21:14:01 0:31:38 0:21:39 0:09:59 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
Failure Reason:

"2024-04-08T21:08:43.705806+0000 mon.smithi038 (mon.0) 770 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7645969 2024-04-08 19:18:19 2024-04-08 20:42:23 2024-04-08 21:24:26 0:42:03 0:31:57 0:10:06 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"1712610663.5275354 mon.smithi092 (mon.0) 919 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log

fail 7645970 2024-04-08 19:18:20 2024-04-08 20:42:23 2024-04-08 21:06:12 0:23:49 0:14:27 0:09:22 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
Failure Reason:

"2024-04-08T21:03:14.196465+0000 mon.smithi032 (mon.0) 769 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7645971 2024-04-08 19:18:22 2024-04-08 20:42:24 2024-04-08 21:35:08 0:52:44 0:43:51 0:08:53 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi012 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4432bbac-f5ea-11ee-b649-cb9ed24678a4 -e sha1=fb6f9800392d73c10dc835c8e8e86c5884686270 -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''