Pulpito

adking-2024-08-15_21:01:13-orch:cephadm-wip-adk-testing-2024-08-15-1432-distro-default-smithi

User Scheduled Started Updated Runtime Suite Branch Machine Type Revision Pass Fail Dead
adking 2024-08-15 21:01:13 2024-08-15 21:05:46 2024-08-16 05:36:33 8:30:47 orch:cephadm wip-adk-testing-2024-08-15-1432 smithi 2623527 87 11 1

Average wait time: 0:10:53

See other runs of suite 'orch:cephadm' on branch 'wip-adk-testing-2024-08-15-1432'?

See other runs on branch 'wip-adk-testing-2024-08-15-1432'?

See other runs scheduled on 2024-08-15?

See detail view?

Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7857140 2024-08-15 21:01:17 2024-08-15 21:05:46 2024-08-15 21:56:51 0:51:05 0:40:02 0:11:03 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 7857141 2024-08-15 21:01:18 2024-08-15 21:07:57 2024-08-15 22:04:00 0:56:03 0:45:18 0:10:45 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
pass 7857142 2024-08-15 21:01:20 2024-08-15 21:08:17 2024-08-15 21:31:50 0:23:33 0:13:55 0:09:38 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7857143 2024-08-15 21:01:21 2024-08-15 21:08:28 2024-08-15 21:41:28 0:33:00 0:21:20 0:11:40 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
pass 7857144 2024-08-15 21:01:22 2024-08-15 21:09:48 2024-08-15 22:03:20 0:53:32 0:43:46 0:09:46 smithi main centos 9.stream orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} 2
fail 7857145 2024-08-15 21:01:24 2024-08-15 21:10:29 2024-08-15 21:45:20 0:34:51 0:25:10 0:09:41 smithi main centos 9.stream orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignore_mgr_down ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} 1
Failure Reason:

Test failure: test_create_export_via_apply (tasks.cephfs.test_nfs.TestNFS)

pass 7857146 2024-08-15 21:01:25 2024-08-15 21:10:29 2024-08-15 21:35:40 0:25:11 0:15:49 0:09:22 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} 1
pass 7857147 2024-08-15 21:01:27 2024-08-15 21:10:39 2024-08-15 21:34:30 0:23:51 0:14:51 0:09:00 smithi main ubuntu 22.04 orch:cephadm/orchestrator_cli/{0-random-distro$/{ubuntu_22.04} 2-node-mgr agent/off orchestrator_cli} 2
pass 7857148 2024-08-15 21:01:28 2024-08-15 21:10:50 2024-08-15 21:42:44 0:31:54 0:19:30 0:12:24 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7857149 2024-08-15 21:01:29 2024-08-15 21:13:10 2024-08-15 21:50:21 0:37:11 0:25:13 0:11:58 smithi main centos 9.stream orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
pass 7857150 2024-08-15 21:01:31 2024-08-15 21:15:01 2024-08-15 21:42:58 0:27:57 0:16:06 0:11:51 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_ctdb_res_dom} 4
fail 7857151 2024-08-15 21:01:32 2024-08-15 21:15:21 2024-08-15 21:35:56 0:20:35 0:09:01 0:11:34 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
Failure Reason:

Command failed on smithi026 with status 5: 'sudo systemctl stop ceph-d78c9864-5b4d-11ef-bccf-c7b262605968@mon.smithi026'

pass 7857152 2024-08-15 21:01:33 2024-08-15 21:15:22 2024-08-15 21:42:57 0:27:35 0:18:21 0:09:14 smithi main ubuntu 22.04 orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} 1
pass 7857153 2024-08-15 21:01:35 2024-08-15 21:15:22 2024-08-15 21:37:23 0:22:01 0:11:24 0:10:37 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
pass 7857154 2024-08-15 21:01:36 2024-08-15 21:16:13 2024-08-15 21:33:28 0:17:15 0:06:41 0:10:34 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} 1
pass 7857155 2024-08-15 21:01:37 2024-08-15 21:16:53 2024-08-15 21:46:27 0:29:34 0:15:54 0:13:40 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
pass 7857156 2024-08-15 21:01:39 2024-08-15 21:17:03 2024-08-15 21:44:33 0:27:30 0:13:39 0:13:51 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_domain} 2
pass 7857157 2024-08-15 21:01:40 2024-08-15 21:17:04 2024-08-15 21:49:19 0:32:15 0:21:02 0:11:13 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
fail 7857158 2024-08-15 21:01:42 2024-08-15 21:17:34 2024-08-15 21:47:28 0:29:54 0:20:02 0:09:52 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi076 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 113bf0be-5b4e-11ef-bccf-c7b262605968 -e sha1=2623527e21ba4ca1f3bfb13431997fb684c05bae -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

fail 7857159 2024-08-15 21:01:43 2024-08-15 21:18:05 2024-08-15 22:27:52 1:09:47 0:58:35 0:11:12 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-squid 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi087 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 376cf544-5b4e-11ef-bccf-c7b262605968 -e sha1=2623527e21ba4ca1f3bfb13431997fb684c05bae -- bash -c \'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e \'"\'"\'.up_to_date | length == 7\'"\'"\'\''

pass 7857160 2024-08-15 21:01:44 2024-08-15 21:18:05 2024-08-15 21:41:51 0:23:46 0:13:19 0:10:27 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7857161 2024-08-15 21:01:46 2024-08-15 21:19:05 2024-08-15 22:06:43 0:47:38 0:30:38 0:17:00 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} 2
pass 7857162 2024-08-15 21:01:47 2024-08-15 21:19:06 2024-08-15 22:12:39 0:53:33 0:43:00 0:10:33 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} 2
pass 7857163 2024-08-15 21:01:48 2024-08-15 21:19:16 2024-08-15 21:48:52 0:29:36 0:19:37 0:09:59 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} 2
pass 7857164 2024-08-15 21:01:50 2024-08-15 21:19:17 2024-08-15 21:42:58 0:23:41 0:14:11 0:09:30 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7857165 2024-08-15 21:01:51 2024-08-15 21:19:57 2024-08-15 22:00:04 0:40:07 0:29:48 0:10:19 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} 5
pass 7857166 2024-08-15 21:01:52 2024-08-15 21:20:58 2024-08-15 21:48:43 0:27:45 0:17:52 0:09:53 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_basic} 2
pass 7857167 2024-08-15 21:01:54 2024-08-15 21:20:58 2024-08-15 21:48:22 0:27:24 0:13:10 0:14:14 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7857168 2024-08-15 21:01:55 2024-08-15 21:21:09 2024-08-15 21:56:36 0:35:27 0:26:14 0:09:13 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7857169 2024-08-15 21:01:56 2024-08-15 21:21:09 2024-08-15 21:53:30 0:32:21 0:15:42 0:16:39 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} 3
pass 7857170 2024-08-15 21:01:58 2024-08-15 21:24:20 2024-08-15 21:54:28 0:30:08 0:19:36 0:10:32 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
pass 7857171 2024-08-15 21:01:59 2024-08-15 21:24:20 2024-08-15 21:48:11 0:23:51 0:13:39 0:10:12 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7857172 2024-08-15 21:02:01 2024-08-15 21:25:21 2024-08-15 22:15:01 0:49:40 0:35:33 0:14:07 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
pass 7857173 2024-08-15 21:02:02 2024-08-15 21:25:41 2024-08-15 21:49:46 0:24:05 0:11:18 0:12:47 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_res_dom} 2
dead 7857174 2024-08-15 21:02:03 2024-08-15 21:27:12 2024-08-16 05:36:33 8:09:21 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} 2
Failure Reason:

hit max job timeout

pass 7857175 2024-08-15 21:02:05 2024-08-15 21:27:42 2024-08-15 21:50:42 0:23:00 0:11:06 0:11:54 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 3
pass 7857176 2024-08-15 21:02:06 2024-08-15 21:29:43 2024-08-15 22:18:04 0:48:21 0:33:01 0:15:20 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
pass 7857177 2024-08-15 21:02:07 2024-08-15 21:31:13 2024-08-15 21:59:28 0:28:15 0:16:45 0:11:30 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} 2
pass 7857178 2024-08-15 21:02:08 2024-08-15 21:32:04 2024-08-15 22:20:45 0:48:41 0:36:18 0:12:23 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
pass 7857179 2024-08-15 21:02:10 2024-08-15 21:33:44 2024-08-15 22:04:35 0:30:51 0:21:42 0:09:09 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} 2
pass 7857180 2024-08-15 21:02:11 2024-08-15 21:33:45 2024-08-15 22:01:34 0:27:49 0:18:07 0:09:42 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7857181 2024-08-15 21:02:13 2024-08-15 21:34:45 2024-08-15 21:56:24 0:21:39 0:12:31 0:09:08 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} 1
fail 7857182 2024-08-15 21:02:14 2024-08-15 21:34:46 2024-08-15 22:01:11 0:26:25 0:14:19 0:12:06 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

reached maximum tries (120) after waiting for 120 seconds

pass 7857183 2024-08-15 21:02:15 2024-08-15 21:36:06 2024-08-15 21:59:45 0:23:39 0:14:30 0:09:09 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} 2
pass 7857184 2024-08-15 21:02:17 2024-08-15 21:36:47 2024-08-15 22:07:32 0:30:45 0:19:39 0:11:06 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} 2
pass 7857185 2024-08-15 21:02:18 2024-08-15 21:37:37 2024-08-15 22:04:53 0:27:16 0:13:33 0:13:43 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} 2
fail 7857186 2024-08-15 21:02:19 2024-08-15 21:37:37 2024-08-15 22:15:19 0:37:42 0:25:31 0:12:11 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
Failure Reason:

Command failed on smithi081 with status 5: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2623527e21ba4ca1f3bfb13431997fb684c05bae shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7a35123c-5b51-11ef-bccf-c7b262605968 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v1/status\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts | jq -e \'"\'"\'.data | .[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n\''

fail 7857187 2024-08-15 21:02:21 2024-08-15 21:39:38 2024-08-15 22:10:50 0:31:12 0:19:24 0:11:48 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi164 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4220b90a-5b51-11ef-bccf-c7b262605968 -e sha1=2623527e21ba4ca1f3bfb13431997fb684c05bae -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

fail 7857188 2024-08-15 21:02:22 2024-08-15 21:41:49 2024-08-15 22:34:52 0:53:03 0:42:32 0:10:31 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-squid 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

Command failed on smithi042 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b18eaae-5b51-11ef-bccf-c7b262605968 -e sha1=2623527e21ba4ca1f3bfb13431997fb684c05bae -- bash -c \'ceph versions | jq -e \'"\'"\'.rgw | length == 1\'"\'"\'\''

pass 7857189 2024-08-15 21:02:23 2024-08-15 21:42:09 2024-08-15 22:10:01 0:27:52 0:18:47 0:09:05 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} 2
pass 7857190 2024-08-15 21:02:25 2024-08-15 21:42:30 2024-08-15 22:05:38 0:23:08 0:13:27 0:09:41 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} 2
pass 7857191 2024-08-15 21:02:26 2024-08-15 21:42:40 2024-08-15 22:05:36 0:22:56 0:14:08 0:08:48 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
pass 7857192 2024-08-15 21:02:27 2024-08-15 21:42:50 2024-08-15 22:12:17 0:29:27 0:19:03 0:10:24 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7857193 2024-08-15 21:02:29 2024-08-15 21:42:51 2024-08-15 22:14:45 0:31:54 0:22:02 0:09:52 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} 2
pass 7857194 2024-08-15 21:02:30 2024-08-15 21:42:51 2024-08-15 22:32:33 0:49:42 0:39:56 0:09:46 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} 2
pass 7857195 2024-08-15 21:02:31 2024-08-15 21:42:52 2024-08-15 22:09:12 0:26:20 0:13:08 0:13:12 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} 3
pass 7857196 2024-08-15 21:02:33 2024-08-15 21:44:02 2024-08-15 22:06:38 0:22:36 0:13:40 0:08:56 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} 1
pass 7857197 2024-08-15 21:02:34 2024-08-15 21:44:02 2024-08-15 22:08:25 0:24:23 0:12:22 0:12:01 smithi main centos 9.stream orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream} 2-node-mgr agent/on orchestrator_cli} 2
pass 7857198 2024-08-15 21:02:35 2024-08-15 21:44:53 2024-08-15 22:08:06 0:23:13 0:10:44 0:12:29 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_basic} 2
pass 7857199 2024-08-15 21:02:37 2024-08-15 21:46:44 2024-08-15 22:10:54 0:24:10 0:13:45 0:10:25 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} 2
pass 7857200 2024-08-15 21:02:38 2024-08-15 21:47:44 2024-08-15 22:08:18 0:20:34 0:12:57 0:07:37 smithi main centos 9.stream orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} 1
pass 7857201 2024-08-15 21:02:39 2024-08-15 21:47:44 2024-08-15 22:10:58 0:23:14 0:12:18 0:10:56 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} 3
pass 7857202 2024-08-15 21:02:41 2024-08-15 21:47:45 2024-08-15 22:13:24 0:25:39 0:14:42 0:10:57 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} 2
pass 7857203 2024-08-15 21:02:42 2024-08-15 21:48:25 2024-08-15 22:11:40 0:23:15 0:13:21 0:09:54 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
pass 7857204 2024-08-15 21:02:43 2024-08-15 21:48:36 2024-08-15 22:32:02 0:43:26 0:33:28 0:09:58 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7857205 2024-08-15 21:02:45 2024-08-15 21:48:56 2024-08-15 22:32:43 0:43:47 0:34:39 0:09:08 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} 2
Failure Reason:

Command failed on smithi070 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7754a7fc-5b52-11ef-bccf-c7b262605968 -e sha1=2623527e21ba4ca1f3bfb13431997fb684c05bae -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\''

pass 7857206 2024-08-15 21:02:46 2024-08-15 21:49:06 2024-08-15 22:18:29 0:29:23 0:15:51 0:13:32 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
pass 7857207 2024-08-15 21:02:48 2024-08-15 21:49:57 2024-08-15 22:15:51 0:25:54 0:14:54 0:11:00 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_ctdb_res_basic} 4
pass 7857208 2024-08-15 21:02:49 2024-08-15 21:50:38 2024-08-15 22:22:33 0:31:55 0:21:34 0:10:21 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} 2
pass 7857209 2024-08-15 21:02:50 2024-08-15 21:50:58 2024-08-15 22:14:49 0:23:51 0:13:54 0:09:57 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} 2
pass 7857210 2024-08-15 21:02:52 2024-08-15 21:51:49 2024-08-15 22:14:47 0:22:58 0:13:41 0:09:17 smithi main centos 9.stream orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} 1
pass 7857211 2024-08-15 21:02:53 2024-08-15 21:51:49 2024-08-15 22:15:49 0:24:00 0:13:07 0:10:53 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7857212 2024-08-15 21:02:54 2024-08-15 21:53:00 2024-08-15 22:30:09 0:37:09 0:26:21 0:10:48 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_ctdb_res_dom} 4
pass 7857213 2024-08-15 21:02:56 2024-08-15 21:53:50 2024-08-15 22:16:57 0:23:07 0:13:58 0:09:09 smithi main centos 9.stream orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
pass 7857214 2024-08-15 21:02:57 2024-08-15 21:53:51 2024-08-15 22:20:10 0:26:19 0:15:29 0:10:50 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} 2
pass 7857215 2024-08-15 21:02:58 2024-08-15 21:54:41 2024-08-15 23:04:08 1:09:27 0:58:16 0:11:11 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} 2
pass 7857216 2024-08-15 21:03:00 2024-08-15 21:55:12 2024-08-15 22:48:47 0:53:35 0:44:22 0:09:13 smithi main ubuntu 22.04 orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} 2
pass 7857217 2024-08-15 21:03:01 2024-08-15 21:55:12 2024-08-15 22:17:56 0:22:44 0:11:45 0:10:59 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} 2
pass 7857218 2024-08-15 21:03:02 2024-08-15 21:56:53 2024-08-15 22:30:29 0:33:36 0:24:17 0:09:19 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} 2
fail 7857219 2024-08-15 21:03:04 2024-08-15 21:57:03 2024-08-15 22:29:02 0:31:59 0:19:23 0:12:36 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi037 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:squid shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid da354024-5b53-11ef-bccf-c7b262605968 -e sha1=2623527e21ba4ca1f3bfb13431997fb684c05bae -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

pass 7857220 2024-08-15 21:03:05 2024-08-15 21:59:44 2024-08-15 23:15:43 1:15:59 1:06:24 0:09:35 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04-squid 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
pass 7857221 2024-08-15 21:03:06 2024-08-15 22:00:04 2024-08-15 22:21:24 0:21:20 0:11:06 0:10:14 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_domain} 2
pass 7857222 2024-08-15 21:03:08 2024-08-15 22:00:14 2024-08-15 22:23:17 0:23:03 0:14:04 0:08:59 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} 2
pass 7857223 2024-08-15 21:03:09 2024-08-15 22:00:25 2024-08-15 22:24:35 0:24:10 0:12:55 0:11:15 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 3
pass 7857224 2024-08-15 21:03:10 2024-08-15 22:01:25 2024-08-15 22:30:33 0:29:08 0:19:58 0:09:10 smithi main ubuntu 22.04 orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} 1
pass 7857225 2024-08-15 21:03:12 2024-08-15 22:01:26 2024-08-15 22:31:02 0:29:36 0:19:33 0:10:03 smithi main ubuntu 22.04 orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7857226 2024-08-15 21:03:13 2024-08-15 22:01:26 2024-08-15 22:26:48 0:25:22 0:14:37 0:10:45 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} 2
pass 7857227 2024-08-15 21:03:14 2024-08-15 22:01:57 2024-08-15 22:53:32 0:51:35 0:39:13 0:12:22 smithi main ubuntu 22.04 orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} 5
pass 7857228 2024-08-15 21:03:16 2024-08-15 22:03:37 2024-08-15 22:23:46 0:20:09 0:10:35 0:09:34 smithi main centos 9.stream orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_res_basic} 2
pass 7857229 2024-08-15 21:03:17 2024-08-15 22:04:48 2024-08-15 22:37:53 0:33:05 0:22:20 0:10:45 smithi main ubuntu 22.04 orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} 2
pass 7857230 2024-08-15 21:03:18 2024-08-15 22:05:08 2024-08-15 22:46:48 0:41:40 0:30:16 0:11:24 smithi main centos 9.stream orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} 2
pass 7857231 2024-08-15 21:03:20 2024-08-15 22:05:49 2024-08-15 22:36:39 0:30:50 0:21:28 0:09:22 smithi main centos 9.stream orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} 2
pass 7857232 2024-08-15 21:03:21 2024-08-15 22:05:49 2024-08-15 22:20:48 0:14:59 0:06:18 0:08:41 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
pass 7857233 2024-08-15 21:03:23 2024-08-15 22:05:49 2024-08-15 22:54:40 0:48:51 0:33:48 0:15:03 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
pass 7857234 2024-08-15 21:03:24 2024-08-15 22:07:00 2024-08-15 22:31:13 0:24:13 0:12:47 0:11:26 smithi main centos 9.stream orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} 2
pass 7857235 2024-08-15 21:03:25 2024-08-15 22:07:50 2024-08-15 22:51:41 0:43:51 0:31:37 0:12:14 smithi main centos 9.stream orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} 2
pass 7857236 2024-08-15 21:03:27 2024-08-15 22:08:21 2024-08-15 22:31:21 0:23:00 0:13:28 0:09:32 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7857237 2024-08-15 21:03:28 2024-08-15 22:08:31 2024-08-15 22:44:57 0:36:26 0:25:55 0:10:31 smithi main ubuntu 22.04 orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
pass 7857238 2024-08-15 21:03:29 2024-08-15 22:08:42 2024-08-15 22:36:59 0:28:17 0:18:43 0:09:34 smithi main ubuntu 22.04 orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_dom} 2