Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7735348 2024-05-31 02:08:55 2024-06-01 00:22:49 2024-06-01 00:57:51 0:35:02 0:25:28 0:09:34 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7735349 2024-05-31 02:08:56 2024-06-01 00:22:49 2024-06-01 00:42:15 0:19:26 0:10:04 0:09:22 smithi main ubuntu 22.04 upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} 1
Failure Reason:

"1717202413.443547 mon.a (mon.0) 62 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7735350 2024-05-31 02:08:57 2024-06-01 00:22:49 2024-06-01 02:40:32 2:17:43 2:09:05 0:08:38 smithi main ubuntu 22.04 upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-06-01T00:44:57.495616+0000 mgr.x (mgr.24109) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

dead 7735351 2024-05-31 02:08:58 2024-06-01 00:22:50 2024-06-01 12:32:32 12:09:42 smithi main centos 8.stream upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks} 2
Failure Reason:

hit max job timeout

fail 7735352 2024-05-31 02:08:59 2024-06-01 00:22:50 2024-06-01 01:12:27 0:49:37 0:38:45 0:10:52 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7735353 2024-05-31 02:09:00 2024-06-01 00:23:01 2024-06-01 02:20:38 1:57:37 1:48:45 0:08:52 smithi main ubuntu 22.04 upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-06-01T01:04:46.838542+0000 mon.a (mon.0) 33 : cluster [ERR] Health check failed: 8 osds(s) are not reachable (OSD_UNREACHABLE)" in cluster log

fail 7735354 2024-05-31 02:09:01 2024-06-01 00:23:01 2024-06-01 03:59:17 3:36:16 3:24:08 0:12:08 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-06-01T00:44:01.362113+0000 mgr.x (mgr.24112) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

pass 7735355 2024-05-31 02:09:02 2024-06-01 00:23:21 2024-06-01 00:45:34 0:22:13 0:11:50 0:10:23 smithi main centos 8.stream upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} 1
pass 7735356 2024-05-31 02:09:03 2024-06-01 00:24:32 2024-06-01 00:45:27 0:20:55 0:11:39 0:09:16 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/new_ops/{newops}}} 2
fail 7735357 2024-05-31 02:09:04 2024-06-01 00:24:32 2024-06-01 03:29:31 3:04:59 2:55:35 0:09:24 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"1717202846.6831312 mon.a (mon.0) 698 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

dead 7735358 2024-05-31 02:09:05 2024-06-01 00:24:33 2024-06-01 00:25:37 0:01:04 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi152

fail 7735359 2024-05-31 02:09:06 2024-06-01 00:24:33 2024-06-01 02:56:08 2:31:35 2:22:09 0:09:26 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1717202733.1144595 mon.a (mon.0) 505 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7735360 2024-05-31 02:09:07 2024-06-01 00:25:04 2024-06-01 00:59:32 0:34:28 0:20:28 0:14:00 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-06-01T00:48:03.118593+0000 mgr.smithi160.dxcuke (mgr.14238) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7735361 2024-05-31 02:09:08 2024-06-01 00:28:14 2024-06-01 04:05:47 3:37:33 3:27:45 0:09:48 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-06-01T00:49:51.825409+0000 mgr.x (mgr.24109) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

dead 7735362 2024-05-31 02:09:09 2024-06-01 00:28:15 2024-06-01 00:29:19 0:01:04 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi193

fail 7735363 2024-05-31 02:09:10 2024-06-01 00:28:15 2024-06-01 03:48:13 3:19:58 3:09:21 0:10:37 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-06-01T00:47:38.274539+0000 mgr.x (mgr.24115) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

dead 7735364 2024-05-31 02:09:11 2024-06-01 00:28:56 2024-06-01 00:30:00 0:01:04 smithi main centos 8.stream upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi005

fail 7735365 2024-05-31 02:09:13 2024-06-01 00:28:56 2024-06-01 03:19:39 2:50:43 2:37:40 0:13:03 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1717203246.3970022 mon.a (mon.0) 509 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7735366 2024-05-31 02:09:14 2024-06-01 00:32:17 2024-06-01 01:23:26 0:51:09 0:40:33 0:10:36 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7735367 2024-05-31 02:09:15 2024-06-01 00:32:17 2024-06-01 03:42:31 3:10:14 2:59:41 0:10:33 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-06-01T00:51:34.291051+0000 mgr.x (mgr.24115) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7735368 2024-05-31 02:09:16 2024-06-01 00:33:18 2024-06-01 03:16:29 2:43:11 2:32:23 0:10:48 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"1717203307.7819722 mon.a (mon.0) 520 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7735369 2024-05-31 02:09:17 2024-06-01 00:33:18 2024-06-01 01:16:25 0:43:07 0:32:32 0:10:35 smithi main centos 9.stream upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks} 2
Failure Reason:

"1717203312.1130178 mon.a (mon.0) 661 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

dead 7735370 2024-05-31 02:09:18 2024-06-01 00:33:19 2024-06-01 00:34:23 0:01:04 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi155

pass 7735371 2024-05-31 02:09:19 2024-06-01 00:33:19 2024-06-01 01:16:12 0:42:53 0:33:25 0:09:28 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7735372 2024-05-31 02:09:20 2024-06-01 00:33:20 2024-06-01 02:13:13 1:39:53 1:30:10 0:09:43 smithi main centos 9.stream upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1717203251.8821816 mon.a (mon.0) 503 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7735373 2024-05-31 02:09:21 2024-06-01 03:37:56 10402 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-06-01T00:56:17.329212+0000 mgr.x (mgr.24109) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7735374 2024-05-31 02:09:22 2024-06-01 00:34:10 2024-06-01 03:32:52 2:58:42 2:47:46 0:10:56 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1717203797.159698 mon.a (mon.0) 560 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 7735375 2024-05-31 02:09:23 2024-06-01 00:35:11 2024-06-01 01:06:20 0:31:09 0:21:35 0:09:34 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
fail 7735376 2024-05-31 02:09:24 2024-06-01 00:35:11 2024-06-01 01:42:55 1:07:44 0:56:31 0:11:13 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7735377 2024-05-31 02:09:25 2024-06-01 00:35:12 2024-06-01 01:24:28 0:49:16 0:39:00 0:10:16 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7735378 2024-05-31 02:09:26 2024-06-01 00:35:12 2024-06-01 04:13:36 3:38:24 3:27:15 0:11:09 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"1717203440.1838558 mon.a (mon.0) 183 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 7735379 2024-05-31 02:09:27 2024-06-01 00:35:12 2024-06-01 03:22:25 2:47:13 2:38:46 0:08:27 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-06-01T00:53:30.701177+0000 mgr.x (mgr.24106) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7735380 2024-05-31 02:09:28 2024-06-01 00:35:13 2024-06-01 03:06:25 2:31:12 2:20:32 0:10:40 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1717203377.3405316 mon.a (mon.0) 520 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

pass 7735381 2024-05-31 02:09:29 2024-06-01 00:35:13 2024-06-01 01:20:22 0:45:09 0:34:27 0:10:42 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7735382 2024-05-31 02:09:30 2024-06-01 00:35:24 2024-06-01 03:51:47 3:16:23 3:02:49 0:13:34 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-06-01T00:57:23.446554+0000 mgr.x (mgr.24112) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log

fail 7735383 2024-05-31 02:09:31 2024-06-01 00:38:34 2024-06-01 03:18:27 2:39:53 2:28:41 0:11:12 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"1717203463.3351257 mon.a (mon.0) 174 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log