Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7694546 2024-05-06 20:50:15 2024-05-06 20:51:57 2024-05-06 23:35:03 2:43:06 2:32:36 0:10:30 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"1715029956.1924665 mon.a (mon.0) 514 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 7694547 2024-05-06 20:50:17 2024-05-06 20:51:57 2024-05-06 21:47:14 0:55:17 0:41:00 0:14:17 smithi main centos 8.stream upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
fail 7694548 2024-05-06 20:50:18 2024-05-06 20:51:58 2024-05-07 00:17:00 3:25:02 3:14:04 0:10:58 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-05-06T21:15:21.029643+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7694549 2024-05-06 20:50:19 2024-05-06 20:51:58 2024-05-06 22:09:20 1:17:22 1:06:45 0:10:37 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7694550 2024-05-06 20:50:21 2024-05-06 20:51:59 2024-05-06 21:54:05 1:02:06 0:51:48 0:10:18 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7694551 2024-05-06 20:50:22 2024-05-06 20:51:59 2024-05-07 00:38:04 3:46:05 3:35:50 0:10:15 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-05-06T21:13:55.033740+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

dead 7694552 2024-05-06 20:50:24 2024-05-06 20:51:59 2024-05-06 20:57:38 0:05:39 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

Error reimaging machines: Expected smithi161's OS to be centos 8 but found ubuntu 22.04

dead 7694553 2024-05-06 20:50:25 2024-05-06 20:52:00 2024-05-06 20:58:00 0:06:00 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

Error reimaging machines: list index out of range

pass 7694554 2024-05-06 20:50:26 2024-05-06 20:52:00 2024-05-06 21:36:04 0:44:04 0:33:09 0:10:55 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7694555 2024-05-06 20:50:28 2024-05-06 20:52:21 2024-05-06 23:40:34 2:48:13 2:35:57 0:12:16 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-05-06T21:50:00.000154+0000 mon.a (mon.0) 1687 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

fail 7694556 2024-05-06 20:50:29 2024-05-06 20:55:32 2024-05-06 23:21:31 2:25:59 2:14:42 0:11:17 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1715030117.8913076 mon.a (mon.0) 372 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7694557 2024-05-06 20:50:30 2024-05-06 20:56:12 2024-05-06 21:37:16 0:41:04 0:31:36 0:09:28 smithi main centos 9.stream upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks} 2
Failure Reason:

"1715030043.7998435 mon.a (mon.0) 172 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

pass 7694558 2024-05-06 20:50:32 2024-05-06 20:56:12 2024-05-06 21:54:48 0:58:36 0:42:39 0:15:57 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7694559 2024-05-06 20:50:33 2024-05-06 20:57:23 2024-05-06 21:36:47 0:39:24 0:25:35 0:13:49 smithi main centos 8.stream upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} 1
Failure Reason:

ceph version 18.2.2-1306.gbf64272e was not installed, found 16.2.15-0.el8.

pass 7694560 2024-05-06 20:50:34 2024-05-06 20:57:43 2024-05-06 21:17:22 0:19:39 0:09:52 0:09:47 smithi main ubuntu 22.04 upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} 1
dead 7694561 2024-05-06 20:50:36 2024-05-06 20:57:44 2024-05-06 21:03:13 0:05:29 smithi main ubuntu 22.04 upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

Error reimaging machines: Expected smithi161's OS to be ubuntu 22.04 but found centos 9

fail 7694562 2024-05-06 20:50:37 2024-05-06 20:57:54 2024-05-06 22:17:34 1:19:40 1:09:03 0:10:37 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7694563 2024-05-06 20:50:38 2024-05-06 20:57:55 2024-05-06 22:37:06 1:39:11 1:28:28 0:10:43 smithi main centos 9.stream upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1715030249.5482547 mon.a (mon.0) 502 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7694564 2024-05-06 20:50:40 2024-05-06 20:57:55 2024-05-07 00:09:26 3:11:31 3:00:40 0:10:51 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-05-06T21:40:00.002669+0000 mon.a (mon.0) 953 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

pass 7694565 2024-05-06 20:50:41 2024-05-06 20:58:06 2024-05-06 21:53:12 0:55:06 0:41:35 0:13:31 smithi main centos 8.stream upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
dead 7694566 2024-05-06 20:50:42 2024-05-06 20:58:06 2024-05-06 21:03:39 0:05:33 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

Error reimaging machines: list index out of range

fail 7694567 2024-05-06 20:50:44 2024-05-06 20:58:26 2024-05-06 22:23:45 1:25:19 1:14:33 0:10:46 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7694568 2024-05-06 20:50:45 2024-05-06 20:59:57 2024-05-06 21:43:17 0:43:20 0:32:10 0:11:10 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-05-06T21:30:00.000215+0000 mon.smithi032 (mon.0) 295 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 13/366 objects degraded (3.552%), 3 pgs degraded" in cluster log

fail 7694569 2024-05-06 20:50:46 2024-05-06 20:59:57 2024-05-06 22:01:51 1:01:54 0:51:34 0:10:20 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7694570 2024-05-06 20:50:48 2024-05-06 21:00:08 2024-05-07 00:18:10 3:18:02 3:05:51 0:12:11 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-05-06T21:40:00.000148+0000 mon.a (mon.0) 729 : cluster 3 [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set" in cluster log

pass 7694571 2024-05-06 20:50:49 2024-05-06 21:00:28 2024-05-06 21:55:50 0:55:22 0:39:15 0:16:07 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
fail 7694572 2024-05-06 20:50:51 2024-05-06 21:00:39 2024-05-07 00:04:50 3:04:11 2:53:33 0:10:38 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"1715030525.784227 mon.a (mon.0) 516 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7694573 2024-05-06 20:50:52 2024-05-06 21:01:19 2024-05-06 21:51:18 0:49:59 0:38:20 0:11:39 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7694574 2024-05-06 20:50:53 2024-05-06 21:01:20 2024-05-06 23:16:57 2:15:37 2:05:08 0:10:29 smithi main ubuntu 22.04 upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-05-06T21:23:13.804456+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7694575 2024-05-06 20:50:55 2024-05-06 21:01:20 2024-05-06 21:59:24 0:58:04 0:43:51 0:14:13 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7694576 2024-05-06 20:50:56 2024-05-06 21:02:20 2024-05-06 22:40:10 1:37:50 1:28:09 0:09:41 smithi main centos 9.stream upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1715030497.8190818 mon.a (mon.0) 372 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

pass 7694577 2024-05-06 20:50:57 2024-05-06 21:02:21 2024-05-06 21:54:09 0:51:48 0:34:24 0:17:24 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/fsstress}} 2
fail 7694578 2024-05-06 20:50:59 2024-05-06 21:02:21 2024-05-06 23:52:47 2:50:26 2:38:40 0:11:46 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-05-06T21:50:00.000139+0000 mon.a (mon.0) 1650 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log

pass 7694579 2024-05-06 20:51:00 2024-05-06 21:03:22 2024-05-06 21:44:28 0:41:06 0:32:14 0:08:52 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7694580 2024-05-06 20:51:01 2024-05-06 21:03:22 2024-05-06 23:44:42 2:41:20 2:32:06 0:09:14 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1715030724.247799 mon.a (mon.0) 664 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7694581 2024-05-06 20:51:03 2024-05-06 21:03:43 2024-05-06 21:46:46 0:43:03 0:29:02 0:14:01 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}} 2
Failure Reason:

Command failed on smithi161 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd'

fail 7694582 2024-05-06 20:51:04 2024-05-06 21:03:43 2024-05-07 00:40:43 3:37:00 3:24:47 0:12:13 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-05-06T21:37:25.934491+0000 mon.a (mon.0) 222 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7694583 2024-05-06 20:51:05 2024-05-06 21:04:54 2024-05-06 22:00:47 0:55:53 0:41:18 0:14:35 smithi main centos 8.stream upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
fail 7694584 2024-05-06 20:51:06 2024-05-06 21:05:24 2024-05-07 00:28:48 3:23:24 3:14:05 0:09:19 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"1715030799.1273637 mon.a (mon.0) 506 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7694585 2024-05-06 20:51:08 2024-05-06 21:05:25 2024-05-07 00:19:30 3:14:05 3:00:37 0:13:28 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-05-06T21:30:15.599171+0000 mon.a (mon.0) 184 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7694586 2024-05-06 20:51:09 2024-05-06 21:07:45 2024-05-06 21:59:32 0:51:47 0:38:45 0:13:02 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7694587 2024-05-06 20:51:11 2024-05-06 21:09:06 2024-05-07 00:09:07 3:00:01 2:49:25 0:10:36 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1715031593.319906 mon.a (mon.0) 720 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

pass 7694588 2024-05-06 20:51:12 2024-05-06 21:09:06 2024-05-06 22:03:41 0:54:35 0:36:31 0:18:04 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
fail 7694589 2024-05-06 20:51:13 2024-05-06 21:48:49 2024-05-07 01:00:00 3:11:11 3:00:49 0:10:22 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-05-06T22:30:00.000117+0000 mon.a (mon.0) 901 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 5969/209115 objects degraded (2.854%), 2 pgs degraded" in cluster log

fail 7694590 2024-05-06 20:51:15 2024-05-06 21:48:49 2024-05-07 00:29:46 2:40:57 2:31:33 0:09:24 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"1715033490.305902 mon.a (mon.0) 682 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

dead 7694591 2024-05-06 20:51:16 2024-05-06 21:48:50 2024-05-07 09:58:19 12:09:29 smithi main centos 8.stream upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_8.stream_container_tools} 0-start 1-tasks} 2
Failure Reason:

hit max job timeout

pass 7694592 2024-05-06 20:51:18 2024-05-06 21:49:00 2024-05-06 22:45:56 0:56:56 0:42:43 0:14:13 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7694593 2024-05-06 20:51:19 2024-05-06 21:49:00 2024-05-06 22:37:59 0:48:59 0:38:44 0:10:15 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7694594 2024-05-06 20:51:20 2024-05-06 21:49:01 2024-05-06 22:08:23 0:19:22 0:10:03 0:09:19 smithi main ubuntu 22.04 upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} 1
fail 7694595 2024-05-06 20:51:22 2024-05-06 21:49:01 2024-05-06 23:47:51 1:58:50 1:48:15 0:10:35 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-05-06T23:00:00.000232+0000 mon.a (mon.0) 2709 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log

fail 7694596 2024-05-06 20:51:23 2024-05-06 21:49:01 2024-05-06 22:28:03 0:39:02 0:25:48 0:13:14 smithi main centos 8.stream upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} 1
Failure Reason:

ceph version 18.2.2-1306.gbf64272e was not installed, found 16.2.15-0.el8.

pass 7694597 2024-05-06 20:51:24 2024-05-06 21:49:02 2024-05-06 22:50:01 1:00:59 0:44:49 0:16:10 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}} 2
fail 7694598 2024-05-06 20:51:26 2024-05-06 21:49:02 2024-05-06 23:49:00 1:59:58 1:44:30 0:15:28 smithi main ubuntu 22.04 upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1715034108.6037297 mon.a (mon.0) 515 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7694599 2024-05-06 20:51:27 2024-05-06 21:53:13 2024-05-07 01:22:33 3:29:20 3:15:49 0:13:31 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-05-06T22:25:03.263696+0000 mon.a (mon.0) 16 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log

fail 7694600 2024-05-06 20:51:29 2024-05-06 21:54:14 2024-05-06 22:38:12 0:43:58 0:29:11 0:14:47 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} 2
Failure Reason:

Command failed on smithi204 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd'

pass 7694601 2024-05-06 20:51:30 2024-05-06 21:54:54 2024-05-06 22:53:10 0:58:16 0:40:39 0:17:37 smithi main centos 8.stream upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
fail 7694602 2024-05-06 20:51:31 2024-05-06 21:55:55 2024-05-07 00:41:06 2:45:11 2:35:54 0:09:17 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1715033804.812859 mon.a (mon.0) 518 : cluster [WRN] Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 7694603 2024-05-06 20:51:33 2024-05-06 21:55:55 2024-05-06 22:55:11 0:59:16 0:46:23 0:12:53 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

pass 7694604 2024-05-06 20:51:34 2024-05-06 21:59:26 2024-05-06 22:43:40 0:44:14 0:32:03 0:12:11 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
fail 7694605 2024-05-06 20:51:35 2024-05-06 22:00:57 2024-05-06 22:57:00 0:56:03 0:45:23 0:10:40 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7694606 2024-05-06 20:51:37 2024-05-06 22:00:57 2024-05-07 01:22:56 3:21:59 3:09:27 0:12:32 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-05-06T22:50:00.000148+0000 mon.a (mon.0) 1560 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub flag(s) set" in cluster log

pass 7694607 2024-05-06 20:51:38 2024-05-06 22:03:48 2024-05-06 22:56:58 0:53:10 0:37:13 0:15:57 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
fail 7694608 2024-05-06 20:51:39 2024-05-06 22:04:18 2024-05-06 22:56:36 0:52:18 0:39:22 0:12:56 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7694609 2024-05-06 20:51:41 2024-05-06 22:04:19 2024-05-07 00:37:27 2:33:08 2:22:17 0:10:51 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1715034428.659614 mon.a (mon.0) 685 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7694610 2024-05-06 20:51:42 2024-05-06 22:04:19 2024-05-07 00:03:46 1:59:27 1:48:33 0:10:54 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-05-06T22:40:00.000199+0000 mon.a (mon.0) 773 : cluster 3 [WRN] PG_DEGRADED: Degraded data redundancy: 76/1356 objects degraded (5.605%), 4 pgs degraded" in cluster log

pass 7694611 2024-05-06 20:51:43 2024-05-06 22:04:19 2024-05-06 23:01:49 0:57:30 0:43:29 0:14:01 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7694612 2024-05-06 20:51:45 2024-05-06 22:04:30 2024-05-06 23:44:21 1:39:51 1:29:20 0:10:31 smithi main centos 9.stream upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1715034390.0848992 mon.a (mon.0) 504 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7694613 2024-05-06 20:51:46 2024-05-06 22:04:30 2024-05-07 01:32:17 3:27:47 3:15:47 0:12:00 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} 2
Failure Reason:

"2024-05-06T22:24:07.567535+0000 mon.a (mon.0) 174 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7694614 2024-05-06 20:51:47 2024-05-06 22:04:31 2024-05-06 22:47:38 0:43:07 0:33:32 0:09:35 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2