| User | Scheduled | Started | Updated | Runtime | Suite | Branch | Machine Type | Revision | Pass | Fail | Dead |
|---|---|---|---|---|---|---|---|---|---|---|---|
| adking | 2024-05-06 20:43:05 | 2024-05-06 22:04:34 | 2024-05-07 14:14:33 | 16:09:59 | orch:cephadm | wip-adk-testing-2024-05-06-0906 | smithi | 9a2db5c | 27 | 65 | 2 |
| Status | Job ID | Links | Posted | Started | Updated | Runtime |
Duration |
In Waiting |
Machine | Teuthology Branch | OS Type | OS Version | Description | Nodes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| fail | 7694450 |
|
2024-05-06 20:43:10 | 2024-05-06 20:50:55 | 2024-05-06 21:40:25 | 0:49:30 | 0:38:58 | 0:10:32 | smithi | main | centos | 9.stream | orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
reached maximum tries (51) after waiting for 300 seconds |
||||||||||||||
| fail | 7694451 |
|
2024-05-06 20:43:11 | 2024-05-06 22:04:32 | 2024-05-06 23:12:58 | 1:08:26 | 0:57:14 | 0:11:12 | smithi | main | ubuntu | 22.04 | orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/connectivity} | 2 |
Failure Reason:
"2024-05-06T22:40:13.358085+0000 mon.a (mon.0) 982 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi152 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694452 |
|
2024-05-06 20:43:12 | 2024-05-06 22:04:33 | 2024-05-07 02:15:24 | 4:10:51 | 0:18:13 | 3:52:38 | smithi | main | centos | 9.stream | orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} | 2 |
| fail | 7694453 |
|
2024-05-06 20:43:13 | 2024-05-06 22:04:33 | 2024-05-06 22:36:45 | 0:32:12 | 0:22:17 | 0:09:55 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} | 2 |
Failure Reason:
"2024-05-06T22:25:28.897359+0000 mon.smithi099 (mon.0) 230 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi099 on smithi099 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694454 |
|
2024-05-06 20:43:14 | 2024-05-06 22:04:34 | 2024-05-06 22:50:24 | 0:45:50 | 0:34:10 | 0:11:40 | smithi | main | centos | 9.stream | orch:cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} | 2 |
Failure Reason:
"2024-05-06T22:31:42.187170+0000 mon.smithi084 (mon.0) 871 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi084 on smithi084 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694455 |
|
2024-05-06 20:43:15 | 2024-05-06 22:04:34 | 2024-05-06 22:55:16 | 0:50:42 | 0:37:34 | 0:13:08 | smithi | main | centos | 9.stream | orch:cephadm/nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} | 1 |
Failure Reason:
"2024-05-06T22:38:29.681934+0000 mds.nfs-cephfs.smithi049.tutlsp (mds.0) 1 : cluster [WRN] client.15223 isn't responding to mclientcaps(revoke), ino 0x1 pending pAsLsXs issued pAsLsXsFs, sent 60.203407 seconds ago" in cluster log |
||||||||||||||
| pass | 7694456 |
|
2024-05-06 20:43:16 | 2024-05-06 22:04:34 | 2024-05-06 22:27:19 | 0:22:45 | 0:13:41 | 0:09:04 | smithi | main | centos | 9.stream | orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/classic task/test_orch_cli} | 1 |
| pass | 7694457 |
|
2024-05-06 20:43:17 | 2024-05-06 22:04:35 | 2024-05-06 22:31:34 | 0:26:59 | 0:11:45 | 0:15:14 | smithi | main | centos | 9.stream | orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/off orchestrator_cli} | 2 |
| fail | 7694458 |
|
2024-05-06 20:43:18 | 2024-05-06 22:08:25 | 2024-05-06 22:40:30 | 0:32:05 | 0:22:10 | 0:09:55 | smithi | main | ubuntu | 22.04 | orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} | 2 |
Failure Reason:
"2024-05-06T22:29:27.356002+0000 mon.smithi012 (mon.0) 235 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi012 on smithi012 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694459 |
|
2024-05-06 20:43:19 | 2024-05-06 22:09:06 | 2024-05-06 22:57:12 | 0:48:06 | 0:26:14 | 0:21:52 | smithi | main | centos | 9.stream | orch:cephadm/rbd_iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} | 3 |
Failure Reason:
"2024-05-06T22:37:05.888223+0000 mon.a (mon.0) 203 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log |
||||||||||||||
| pass | 7694460 |
|
2024-05-06 20:43:20 | 2024-05-06 22:19:48 | 2024-05-06 22:47:16 | 0:27:28 | 0:18:15 | 0:09:13 | smithi | main | ubuntu | 22.04 | orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_domain} | 2 |
| fail | 7694461 |
|
2024-05-06 20:43:21 | 2024-05-06 22:19:58 | 2024-05-06 22:47:08 | 0:27:10 | 0:16:44 | 0:10:26 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} | 2 |
Failure Reason:
"2024-05-06T22:37:03.898891+0000 mon.smithi119 (mon.0) 251 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi119 on smithi119 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694462 |
|
2024-05-06 20:43:22 | 2024-05-06 22:19:58 | 2024-05-06 22:45:08 | 0:25:10 | 0:15:04 | 0:10:06 | smithi | main | centos | 9.stream | orch:cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/basic 3-final} | 1 |
Failure Reason:
"2024-05-06T22:36:05.492063+0000 mon.smithi160 (mon.0) 245 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi160 on smithi160 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694463 |
|
2024-05-06 20:43:23 | 2024-05-06 22:19:59 | 2024-05-06 22:41:09 | 0:21:10 | 0:11:29 | 0:09:41 | smithi | main | centos | 9.stream | orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} | 3 |
Failure Reason:
"2024-05-06T22:39:24.004364+0000 mon.a (mon.0) 568 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi164 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694464 |
|
2024-05-06 20:43:24 | 2024-05-06 22:19:59 | 2024-05-06 22:34:50 | 0:14:51 | 0:06:19 | 0:08:32 | smithi | main | centos | 9.stream | orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_cephadm_repos} | 1 |
| fail | 7694465 |
|
2024-05-06 20:43:25 | 2024-05-06 22:20:00 | 2024-05-06 22:47:00 | 0:27:00 | 0:17:15 | 0:09:45 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} | 2 |
Failure Reason:
"2024-05-06T22:37:05.524588+0000 mon.smithi002 (mon.0) 243 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi002 on smithi002 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694466 |
|
2024-05-06 20:43:26 | 2024-05-06 22:20:00 | 2024-05-06 22:51:17 | 0:31:17 | 0:22:07 | 0:09:10 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} | 2 |
Failure Reason:
"2024-05-06T22:39:48.901602+0000 mon.smithi086 (mon.0) 231 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi086 on smithi086 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694467 |
|
2024-05-06 20:43:27 | 2024-05-06 22:20:00 | 2024-05-06 23:05:32 | 0:45:32 | 0:34:29 | 0:11:03 | smithi | main | centos | 9.stream | orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
"2024-05-06T22:50:00.000166+0000 mon.smithi088 (mon.0) 983 : cluster 3 [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) ['daemon prometheus.smithi088 on smithi088 is in unknown state']" in cluster log |
||||||||||||||
| fail | 7694468 |
|
2024-05-06 20:43:28 | 2024-05-06 22:20:01 | 2024-05-06 22:47:01 | 0:27:00 | 0:17:07 | 0:09:53 | smithi | main | centos | 9.stream | orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} | 2 |
Failure Reason:
"2024-05-06T22:37:33.620436+0000 mon.smithi005 (mon.0) 259 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi005 on smithi005 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694469 |
|
2024-05-06 20:43:29 | 2024-05-06 22:20:01 | 2024-05-06 22:50:51 | 0:30:50 | 0:09:43 | 0:21:07 | smithi | main | centos | 9.stream | orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_res_basic} | 2 |
| fail | 7694470 |
|
2024-05-06 20:43:31 | 2024-05-06 22:31:43 | 2024-05-06 23:06:12 | 0:34:29 | 0:21:08 | 0:13:21 | smithi | main | centos | 9.stream | orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root} | 2 |
Failure Reason:
"2024-05-06T22:58:45.986593+0000 mon.a (mon.0) 774 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi171 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694471 |
|
2024-05-06 20:43:32 | 2024-05-06 22:34:54 | 2024-05-06 23:06:15 | 0:31:21 | 0:20:24 | 0:10:57 | smithi | main | centos | 9.stream | orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} | 2 |
Failure Reason:
"2024-05-06T22:57:42.052872+0000 mon.a (mon.0) 771 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi142 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694472 |
|
2024-05-06 20:43:33 | 2024-05-06 22:35:14 | 2024-05-06 23:02:55 | 0:27:41 | 0:18:35 | 0:09:06 | smithi | main | ubuntu | 22.04 | orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_extra_daemon_features} | 2 |
| fail | 7694473 |
|
2024-05-06 20:43:34 | 2024-05-06 22:35:25 | 2024-05-06 23:02:10 | 0:26:45 | 0:16:43 | 0:10:02 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs 3-final} | 2 |
Failure Reason:
"2024-05-06T22:51:52.832683+0000 mon.smithi077 (mon.0) 250 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi077 on smithi077 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694474 |
|
2024-05-06 20:43:35 | 2024-05-06 22:35:25 | 2024-05-06 23:21:18 | 0:45:53 | 0:23:58 | 0:21:55 | smithi | main | centos | 9.stream | orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/connectivity task/test_orch_cli_mon} | 5 |
| fail | 7694475 |
|
2024-05-06 20:43:36 | 2024-05-06 22:45:57 | 2024-05-06 23:14:11 | 0:28:14 | 0:16:56 | 0:11:18 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs2 3-final} | 2 |
Failure Reason:
"2024-05-06T23:03:35.650582+0000 mon.smithi055 (mon.0) 246 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi055 on smithi055 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694476 |
|
2024-05-06 20:43:37 | 2024-05-06 22:47:17 | 2024-05-06 23:21:28 | 0:34:11 | 0:24:08 | 0:10:03 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} | 2 |
Failure Reason:
"2024-05-06T23:17:07.711248+0000 mon.a (mon.0) 1094 : cluster [WRN] Health check failed: 1 Cephadm Agent(s) are not reporting. Hosts may be offline (CEPHADM_AGENT_DOWN)" in cluster log |
||||||||||||||
| fail | 7694477 |
|
2024-05-06 20:43:38 | 2024-05-06 22:47:48 | 2024-05-06 23:15:11 | 0:27:23 | 0:15:30 | 0:11:53 | smithi | main | centos | 9.stream | orch:cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_host_drain} | 3 |
Failure Reason:
"2024-05-06T23:11:33.779586+0000 mon.a (mon.0) 640 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon osd.4 on smithi195 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694478 |
|
2024-05-06 20:43:39 | 2024-05-06 22:50:08 | 2024-05-06 23:26:21 | 0:36:13 | 0:21:34 | 0:14:39 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nvmeof 3-final} | 2 |
Failure Reason:
"2024-05-06T23:14:41.628305+0000 mon.smithi119 (mon.0) 229 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi119 on smithi119 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694479 |
|
2024-05-06 20:43:40 | 2024-05-06 22:50:39 | 2024-05-06 23:19:45 | 0:29:06 | 0:17:47 | 0:11:19 | smithi | main | centos | 9.stream | orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} | 2 |
Failure Reason:
"2024-05-06T23:08:29.112097+0000 mon.smithi160 (mon.0) 253 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi160 on smithi160 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694480 |
|
2024-05-06 20:43:41 | 2024-05-06 22:50:39 | 2024-05-06 23:11:40 | 0:21:01 | 0:10:55 | 0:10:06 | smithi | main | centos | 9.stream | orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_res_dom} | 2 |
| fail | 7694481 |
|
2024-05-06 20:43:42 | 2024-05-06 22:50:40 | 2024-05-06 23:35:42 | 0:45:02 | 0:33:28 | 0:11:34 | smithi | main | centos | 9.stream | orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
"1715037533.9471173 mon.smithi002 (mon.0) 905 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi002 on smithi002 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694482 |
|
2024-05-06 20:43:43 | 2024-05-06 22:50:40 | 2024-05-06 23:17:28 | 0:26:48 | 0:17:03 | 0:09:45 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/rgw-ingress 3-final} | 2 |
Failure Reason:
"2024-05-06T23:07:21.564347+0000 mon.smithi086 (mon.0) 246 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi086 on smithi086 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694483 |
|
2024-05-06 20:43:44 | 2024-05-06 22:50:40 | 2024-05-06 23:11:38 | 0:20:58 | 0:11:07 | 0:09:51 | smithi | main | centos | 9.stream | orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} | 3 |
Failure Reason:
"2024-05-06T23:08:44.108741+0000 mon.a (mon.0) 528 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon osd.2 on smithi120 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694484 |
|
2024-05-06 20:43:45 | 2024-05-06 22:50:41 | 2024-05-06 23:47:38 | 0:56:57 | 0:47:41 | 0:09:16 | smithi | main | centos | 9.stream | orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} | 2 |
Failure Reason:
"2024-05-06T23:18:29.342160+0000 mon.a (mon.0) 948 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi099 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694485 |
|
2024-05-06 20:43:46 | 2024-05-06 22:50:41 | 2024-05-06 23:20:15 | 0:29:34 | 0:17:55 | 0:11:39 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/rgw 3-final} | 2 |
Failure Reason:
"2024-05-06T23:08:03.126943+0000 mon.smithi115 (mon.0) 157 : cluster [WRN] Health check failed: Failed to place 1 daemon(s) ['Failed while placing alertmanager.smithi115 on smithi115: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-0b1acef8-0bfd-11ef-bc97-c7b262605968-alertmanager-smithi115\n/usr/bin/podman: stderr Error: no such container ceph-0b1acef8-0bfd-11ef-bc97-c7b262605968-alertmanager-smithi115\nNon-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-0b1acef8-0bfd-11ef-bc97-c7b262605968-alertmanager.smithi115\n/usr/bin/podman: stderr Error: no such container ceph-0b1acef8-0bfd-11ef-bc97-c7b262605968-alertmanager.smithi115\nDeploy daemon alertmanager.smithi115 ...\nNon-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/alertmanager:v0.25.0 -e NODE_NAME=smithi115 quay.io/prometheus/alertmanager:v0.25.0 -c %u %g /etc/alertmanager\nstat: stderr Trying to pull quay.io/prometheus/alertmanager:v0.25.0...\nstat: stderr Error: initializing source docker://quay.io/prometheus/alertmanager:v0.25.0: reading manifest v0.25.0 in quay.io/prometheus/alertmanager: received unexpected HTTP status: 502 Bad Gateway\nNon-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/alertmanager:v0.25.0 -e NODE_NAME=smithi115 quay.io/prometheus/alertmanager:v0.25.0 -c %u %g /etc/prometheus\nstat: stderr Trying to pull quay.io/prometheus/alertmanager:v0.25.0...\nstat: stderr Getting image source signatures\nstat: stderr Copying blob sha256:6c477a8cc220cbe4c1ffdd9cb505ca82284292c367a03335c5bd590ff0c651fc\nstat: stderr Copying blob sha256:d71d159599c38915c22c878fdb13c857684102338f6f33d3f0011e36ad117c04\nstat: stderr Copying blob sha256:b08a0a8262352677ce3e10b697ebda40ffffcfb2cc4dd66a93fc220b940801f5\nstat: stderr Copying blob sha256:05d21abf0535766aaa32ec4541e1213912944d7dea17e40e71a84177f9000b68\nstat: stderr Copying blob sha256:c4dc43cc86853f40d99d6199570e9f823856fa3e7992dcd7ebe94fa4d32a0ae6\nstat: stderr Copying blob sha256:aff850a11e318220e60d82e705674a12e50fb96de4644ab665b2865d7c783796\nstat: stderr Error: copying system image from manifest list: reading blob sha256:b08a0a8262352677ce3e10b697ebda40ffffcfb2cc4dd66a93fc220b940801f5: fetching blob: received unexpected HTTP status: 502 Bad Gateway\nERROR: Failed to extract uid/gid for path /etc/prometheus: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/alertmanager:v0.25.0 -e NODE_NAME=smithi115 quay.io/prometheus/alertmanager:v0.25.0 -c %u %g /etc/prometheus'] (CEPHADM_DAEMON_PLACE_FAIL)" in cluster log |
||||||||||||||
| fail | 7694486 |
|
2024-05-06 20:43:47 | 2024-05-06 22:50:41 | 2024-05-06 23:22:41 | 0:32:00 | 0:20:20 | 0:11:40 | smithi | main | centos | 9.stream | orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} | 2 |
Failure Reason:
"2024-05-06T23:14:22.676861+0000 mon.a (mon.0) 773 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi163 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694487 |
|
2024-05-06 20:43:48 | 2024-05-06 22:50:52 | 2024-05-06 23:22:40 | 0:31:48 | 0:19:53 | 0:11:55 | smithi | main | centos | 9.stream | orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python} | 2 |
Failure Reason:
"2024-05-06T23:14:04.960992+0000 mon.a (mon.0) 775 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi204 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694488 |
|
2024-05-06 20:43:49 | 2024-05-06 22:50:52 | 2024-05-06 23:14:08 | 0:23:16 | 0:13:54 | 0:09:22 | smithi | main | centos | 9.stream | orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} | 1 |
| pass | 7694489 |
|
2024-05-06 20:43:50 | 2024-05-06 22:50:52 | 2024-05-06 23:13:50 | 0:22:58 | 0:12:52 | 0:10:06 | smithi | main | ubuntu | 22.04 | orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/classic task/test_adoption} | 1 |
| fail | 7694490 |
|
2024-05-06 20:43:51 | 2024-05-06 22:50:53 | 2024-05-06 23:18:06 | 0:27:13 | 0:17:54 | 0:09:19 | smithi | main | centos | 9.stream | orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw} | 2 |
Failure Reason:
"2024-05-06T23:07:38.195174+0000 mon.smithi080 (mon.0) 240 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi080 on smithi080 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694491 |
|
2024-05-06 20:43:52 | 2024-05-06 22:50:53 | 2024-05-06 23:16:19 | 0:25:26 | 0:13:00 | 0:12:26 | smithi | main | centos | 9.stream | orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_basic} | 2 |
| fail | 7694492 |
|
2024-05-06 20:43:53 | 2024-05-06 22:53:14 | 2024-05-06 23:27:03 | 0:33:49 | 0:23:24 | 0:10:25 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/basic 3-final} | 2 |
Failure Reason:
"2024-05-06T23:15:10.002808+0000 mon.smithi005 (mon.0) 230 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi005 on smithi005 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694493 |
|
2024-05-06 20:43:55 | 2024-05-06 22:53:14 | 2024-05-06 23:25:29 | 0:32:15 | 0:16:29 | 0:15:46 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/client-keyring 3-final} | 2 |
Failure Reason:
"2024-05-06T23:15:40.076734+0000 mon.smithi027 (mon.0) 244 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi027 on smithi027 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694494 |
|
2024-05-06 20:43:56 | 2024-05-06 22:57:05 | 2024-05-06 23:37:35 | 0:40:30 | 0:25:51 | 0:14:39 | smithi | main | ubuntu | 22.04 | orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_monitoring_stack_basic} | 3 |
Failure Reason:
Command failed on smithi031 with status 7: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a2db5c34e52973e67c30d806fe9e5820c5e10c6 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 64571696-0bff-11ef-bc97-c7b262605968 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v1/status\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts | jq -e \'"\'"\'.data | .[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n\'' |
||||||||||||||
| fail | 7694495 |
|
2024-05-06 20:43:57 | 2024-05-06 23:01:56 | 2024-05-07 00:05:59 | 1:04:03 | 0:53:54 | 0:10:09 | smithi | main | centos | 9.stream | orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
reached maximum tries (51) after waiting for 300 seconds |
||||||||||||||
| fail | 7694496 |
|
2024-05-06 20:43:58 | 2024-05-06 23:02:56 | 2024-05-06 23:33:10 | 0:30:14 | 0:16:38 | 0:13:36 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/iscsi 3-final} | 2 |
Failure Reason:
"2024-05-06T23:22:14.586979+0000 mon.smithi114 (mon.0) 246 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi114 on smithi114 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694497 |
|
2024-05-06 20:43:59 | 2024-05-06 23:06:07 | 2024-05-06 23:29:17 | 0:23:10 | 0:13:23 | 0:09:47 | smithi | main | centos | 9.stream | orch:cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} | 2 |
| fail | 7694498 |
|
2024-05-06 20:44:00 | 2024-05-06 23:06:08 | 2024-05-06 23:40:51 | 0:34:43 | 0:22:59 | 0:11:44 | smithi | main | ubuntu | 22.04 | orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} | 2 |
Failure Reason:
"2024-05-06T23:28:18.619031+0000 mon.smithi072 (mon.0) 232 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi072 on smithi072 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694499 |
|
2024-05-06 20:44:01 | 2024-05-06 23:06:08 | 2024-05-06 23:33:46 | 0:27:38 | 0:17:40 | 0:09:58 | smithi | main | ubuntu | 22.04 | orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_domain} | 2 |
| fail | 7694500 |
|
2024-05-06 20:44:02 | 2024-05-06 23:06:08 | 2024-05-06 23:39:44 | 0:33:36 | 0:23:24 | 0:10:12 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/jaeger 3-final} | 2 |
Failure Reason:
"2024-05-06T23:27:24.563859+0000 mon.smithi136 (mon.0) 227 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi136 on smithi136 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694501 |
|
2024-05-06 20:44:03 | 2024-05-06 23:06:09 | 2024-05-06 23:49:42 | 0:43:33 | 0:30:10 | 0:13:23 | smithi | main | ubuntu | 22.04 | orch:cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} | 2 |
Failure Reason:
"2024-05-06T23:39:30.742769+0000 mon.a (mon.0) 840 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi088 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694502 |
|
2024-05-06 20:44:04 | 2024-05-06 23:06:09 | 2024-05-06 23:45:51 | 0:39:42 | 0:29:17 | 0:10:25 | smithi | main | ubuntu | 22.04 | orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rotate-keys} | 2 |
Failure Reason:
"2024-05-06T23:35:47.310234+0000 mon.a (mon.0) 786 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi196 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694503 |
|
2024-05-06 20:44:05 | 2024-05-06 23:06:19 | 2024-05-06 23:30:15 | 0:23:56 | 0:13:00 | 0:10:56 | smithi | main | centos | 9.stream | orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_rgw_multisite} | 3 |
| pass | 7694504 |
|
2024-05-06 20:44:06 | 2024-05-06 23:06:20 | 2024-05-06 23:29:38 | 0:23:18 | 0:13:36 | 0:09:42 | smithi | main | centos | 9.stream | orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream mon_election/connectivity task/test_cephadm_timeout} | 1 |
| pass | 7694505 |
|
2024-05-06 20:44:07 | 2024-05-06 23:06:20 | 2024-05-06 23:32:58 | 0:26:38 | 0:11:50 | 0:14:48 | smithi | main | centos | 9.stream | orch:cephadm/orchestrator_cli/{0-random-distro$/{centos_9.stream_runc} 2-node-mgr agent/on orchestrator_cli} | 2 |
| fail | 7694506 |
|
2024-05-06 20:44:08 | 2024-05-06 23:11:41 | 2024-05-06 23:41:10 | 0:29:29 | 0:16:54 | 0:12:35 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/mirror 3-final} | 2 |
Failure Reason:
"2024-05-06T23:31:44.164180+0000 mon.smithi053 (mon.0) 252 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi053 on smithi053 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694507 |
|
2024-05-06 20:44:09 | 2024-05-06 23:13:52 | 2024-05-06 23:43:46 | 0:29:54 | 0:19:03 | 0:10:51 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} | 1 |
Failure Reason:
"2024-05-06T23:34:09.406661+0000 mon.smithi176 (mon.0) 244 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi176 on smithi176 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694508 |
|
2024-05-06 20:44:10 | 2024-05-06 23:14:12 | 2024-05-06 23:42:33 | 0:28:21 | 0:10:54 | 0:17:27 | smithi | main | centos | 9.stream | orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} | 3 |
| fail | 7694509 |
|
2024-05-06 20:44:11 | 2024-05-06 23:21:24 | 2024-05-06 23:48:27 | 0:27:03 | 0:16:57 | 0:10:06 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-haproxy-proto 3-final} | 2 |
Failure Reason:
"2024-05-06T23:38:07.709300+0000 mon.smithi003 (mon.0) 250 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi003 on smithi003 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694510 |
|
2024-05-06 20:44:12 | 2024-05-06 23:21:24 | 2024-05-06 23:48:49 | 0:27:25 | 0:17:08 | 0:10:17 | smithi | main | centos | 9.stream | orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} | 2 |
Failure Reason:
"2024-05-06T23:38:53.070486+0000 mon.smithi033 (mon.0) 250 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi033 on smithi033 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694511 |
|
2024-05-06 20:44:13 | 2024-05-06 23:21:24 | 2024-05-06 23:43:42 | 0:22:18 | 0:10:26 | 0:11:52 | smithi | main | centos | 9.stream | orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_basic} | 2 |
| fail | 7694512 |
|
2024-05-06 20:44:14 | 2024-05-06 23:21:35 | 2024-05-07 00:05:41 | 0:44:06 | 0:33:12 | 0:10:54 | smithi | main | centos | 9.stream | orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
"1715039426.6016383 mon.smithi103 (mon.0) 896 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi103 on smithi103 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| dead | 7694513 |
|
2024-05-06 20:44:15 | 2024-05-06 23:21:35 | 2024-05-07 14:14:33 | 14:52:58 | smithi | main | ubuntu | 22.04 | orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/connectivity} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7694514 |
|
2024-05-06 20:44:16 | 2024-05-06 23:21:36 | 2024-05-06 23:46:37 | 0:25:01 | 0:15:56 | 0:09:05 | smithi | main | centos | 9.stream | orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations} | 3 |
| fail | 7694515 |
|
2024-05-06 20:44:17 | 2024-05-06 23:21:36 | 2024-05-06 23:55:01 | 0:33:25 | 0:23:34 | 0:09:51 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final} | 2 |
Failure Reason:
"2024-05-06T23:42:45.940757+0000 mon.smithi012 (mon.0) 227 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi012 on smithi012 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694516 |
|
2024-05-06 20:44:18 | 2024-05-06 23:21:36 | 2024-05-06 23:48:09 | 0:26:33 | 0:16:47 | 0:09:46 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final} | 2 |
Failure Reason:
"2024-05-06T23:37:29.399603+0000 mon.smithi026 (mon.0) 245 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi026 on smithi026 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694517 |
|
2024-05-06 20:44:19 | 2024-05-06 23:21:37 | 2024-05-06 23:44:13 | 0:22:36 | 0:13:51 | 0:08:45 | smithi | main | centos | 9.stream | orch:cephadm/no-agent-workunits/{0-distro/centos_9.stream_runc mon_election/classic task/test_orch_cli} | 1 |
| fail | 7694518 |
|
2024-05-06 20:44:20 | 2024-05-06 23:21:37 | 2024-05-06 23:49:18 | 0:27:41 | 0:17:27 | 0:10:14 | smithi | main | centos | 9.stream | orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} | 2 |
Failure Reason:
"2024-05-06T23:39:25.889325+0000 mon.smithi110 (mon.0) 245 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi110 on smithi110 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694519 |
|
2024-05-06 20:44:21 | 2024-05-06 23:21:37 | 2024-05-06 23:43:28 | 0:21:51 | 0:11:16 | 0:10:35 | smithi | main | centos | 9.stream | orch:cephadm/smb/{0-distro/centos_9.stream_runc tasks/deploy_smb_mgr_domain} | 2 |
| fail | 7694520 |
|
2024-05-06 20:44:23 | 2024-05-06 23:21:38 | 2024-05-06 23:51:21 | 0:29:43 | 0:18:14 | 0:11:29 | smithi | main | centos | 9.stream | orch:cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} | 2 |
Failure Reason:
"2024-05-06T23:42:37.045216+0000 mon.a (mon.0) 772 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi120 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694521 |
|
2024-05-06 20:44:23 | 2024-05-06 23:21:38 | 2024-05-06 23:49:28 | 0:27:50 | 0:15:39 | 0:12:11 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs-ingress 3-final} | 2 |
Failure Reason:
"2024-05-06T23:39:36.809453+0000 mon.smithi157 (mon.0) 256 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi157 on smithi157 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694522 |
|
2024-05-06 20:44:25 | 2024-05-06 23:21:38 | 2024-05-06 23:53:19 | 0:31:41 | 0:20:45 | 0:10:56 | smithi | main | centos | 9.stream | orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} | 2 |
Failure Reason:
"2024-05-06T23:45:07.947715+0000 mon.a (mon.0) 796 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi195 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694523 |
|
2024-05-06 20:44:26 | 2024-05-06 23:21:39 | 2024-05-07 00:01:33 | 0:39:54 | 0:29:29 | 0:10:25 | smithi | main | ubuntu | 22.04 | orch:cephadm/with-work/{0-distro/ubuntu_22.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_api_tests} | 2 |
Failure Reason:
"2024-05-06T23:51:31.928165+0000 mon.a (mon.0) 796 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi106 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694524 |
|
2024-05-06 20:44:27 | 2024-05-06 23:21:40 | 2024-05-06 23:50:44 | 0:29:04 | 0:10:39 | 0:18:25 | smithi | main | centos | 9.stream | orch:cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key} | 2 |
| fail | 7694525 |
|
2024-05-06 20:44:28 | 2024-05-06 23:29:21 | 2024-05-07 00:03:02 | 0:33:41 | 0:21:42 | 0:11:59 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final} | 2 |
Failure Reason:
"2024-05-06T23:52:17.702610+0000 mon.smithi039 (mon.0) 227 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi039 on smithi039 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694526 |
|
2024-05-06 20:44:29 | 2024-05-06 23:30:22 | 2024-05-07 00:35:43 | 1:05:21 | 0:53:16 | 0:12:05 | smithi | main | centos | 9.stream | orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
reached maximum tries (51) after waiting for 300 seconds |
||||||||||||||
| dead | 7694527 |
|
2024-05-06 20:44:30 | 2024-05-06 23:30:22 | 2024-05-07 11:41:28 | 12:11:06 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nfs-keepalive-only 3-final} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7694528 |
|
2024-05-06 20:44:31 | 2024-05-06 23:33:03 | 2024-05-06 23:58:36 | 0:25:33 | 0:10:28 | 0:15:05 | smithi | main | centos | 9.stream | orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} | 3 |
Failure Reason:
"2024-05-06T23:57:09.023723+0000 mon.a (mon.0) 577 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi155 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694529 |
|
2024-05-06 20:44:32 | 2024-05-06 23:36:54 | 2024-05-07 00:06:17 | 0:29:23 | 0:19:26 | 0:09:57 | smithi | main | ubuntu | 22.04 | orch:cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm} | 1 |
| fail | 7694530 |
|
2024-05-06 20:44:33 | 2024-05-06 23:36:54 | 2024-05-07 00:12:16 | 0:35:22 | 0:22:10 | 0:13:12 | smithi | main | ubuntu | 22.04 | orch:cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} | 2 |
Failure Reason:
"2024-05-06T23:59:43.012187+0000 mon.smithi027 (mon.0) 227 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi027 on smithi027 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694531 |
|
2024-05-06 20:44:34 | 2024-05-06 23:36:54 | 2024-05-07 00:02:30 | 0:25:36 | 0:16:59 | 0:08:37 | smithi | main | ubuntu | 22.04 | orch:cephadm/smb/{0-distro/ubuntu_22.04 tasks/deploy_smb_mgr_res_basic} | 2 |
| fail | 7694532 |
|
2024-05-06 20:44:35 | 2024-05-06 23:36:55 | 2024-05-07 00:03:49 | 0:26:54 | 0:16:47 | 0:10:07 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-services/nfs 3-final} | 2 |
Failure Reason:
"2024-05-06T23:53:19.066519+0000 mon.smithi002 (mon.0) 251 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi002 on smithi002 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694533 |
|
2024-05-06 20:44:36 | 2024-05-06 23:36:55 | 2024-05-07 00:25:09 | 0:48:14 | 0:38:16 | 0:09:58 | smithi | main | ubuntu | 22.04 | orch:cephadm/no-agent-workunits/{0-distro/ubuntu_22.04 mon_election/connectivity task/test_orch_cli_mon} | 5 |
| fail | 7694534 |
|
2024-05-06 20:44:37 | 2024-05-06 23:37:05 | 2024-05-07 00:14:14 | 0:37:09 | 0:22:08 | 0:15:01 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke-roleless/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-services/nfs2 3-final} | 2 |
Failure Reason:
"2024-05-07T00:03:11.262786+0000 mon.smithi052 (mon.0) 233 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi052 on smithi052 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694535 |
|
2024-05-06 20:44:38 | 2024-05-06 23:42:37 | 2024-05-07 00:13:46 | 0:31:09 | 0:21:04 | 0:10:05 | smithi | main | centos | 9.stream | orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root} | 2 |
Failure Reason:
"2024-05-07T00:05:57.459919+0000 mon.a (mon.0) 802 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi064 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694536 |
|
2024-05-06 20:44:39 | 2024-05-06 23:42:37 | 2024-05-07 00:12:49 | 0:30:12 | 0:19:31 | 0:10:41 | smithi | main | centos | 9.stream | orch:cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_python} | 2 |
Failure Reason:
"2024-05-07T00:05:32.285358+0000 mon.a (mon.0) 801 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi152 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694537 |
|
2024-05-06 20:44:40 | 2024-05-06 23:43:38 | 2024-05-07 00:00:12 | 0:16:34 | 0:06:24 | 0:10:10 | smithi | main | centos | 9.stream | orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} | 1 |
| fail | 7694538 |
|
2024-05-06 20:44:41 | 2024-05-06 23:43:48 | 2024-05-07 00:27:44 | 0:43:56 | 0:32:40 | 0:11:16 | smithi | main | centos | 9.stream | orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
"1715040738.4323533 mon.smithi045 (mon.0) 920 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi045 on smithi045 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694539 |
|
2024-05-06 20:44:42 | 2024-05-06 23:44:19 | 2024-05-07 00:13:48 | 0:29:29 | 0:16:25 | 0:13:04 | smithi | main | centos | 9.stream | orch:cephadm/smoke-roleless/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-services/nvmeof 3-final} | 2 |
Failure Reason:
"2024-05-07T00:03:10.773076+0000 mon.smithi022 (mon.0) 255 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi022 on smithi022 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| fail | 7694540 |
|
2024-05-06 20:44:43 | 2024-05-06 23:46:39 | 2024-05-07 05:24:40 | 5:38:01 | 5:23:31 | 0:14:30 | smithi | main | centos | 9.stream | orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/on mon_election/classic} | 2 |
Failure Reason:
Command failed on smithi086 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 64cc546e-0c05-11ef-bc97-c7b262605968 -e sha1=9a2db5c34e52973e67c30d806fe9e5820c5e10c6 -- bash -c \'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk \'"\'"\'{print $2}\'"\'"\')\'' |
||||||||||||||
| fail | 7694541 |
|
2024-05-06 20:44:44 | 2024-05-06 23:50:50 | 2024-05-07 00:19:09 | 0:28:19 | 0:16:48 | 0:11:31 | smithi | main | centos | 9.stream | orch:cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} | 2 |
Failure Reason:
"2024-05-07T00:09:34.071585+0000 mon.smithi079 (mon.0) 389 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.smithi079 on smithi079 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7694542 |
|
2024-05-06 20:44:45 | 2024-05-06 23:52:21 | 2024-05-07 00:13:08 | 0:20:47 | 0:10:58 | 0:09:49 | smithi | main | centos | 9.stream | orch:cephadm/smb/{0-distro/centos_9.stream tasks/deploy_smb_mgr_res_dom} | 2 |
| fail | 7694543 |
|
2024-05-06 20:44:46 | 2024-05-06 23:52:21 | 2024-05-07 00:27:54 | 0:35:33 | 0:25:07 | 0:10:26 | smithi | main | ubuntu | 22.04 | orch:cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/classic start} | 2 |
Failure Reason:
"2024-05-07T00:22:16.016222+0000 mon.a (mon.0) 1128 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) ['daemon prometheus.a on smithi117 is in unknown state'] (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
Pulpito