Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7642330 2024-04-06 03:09:28 2024-04-06 04:15:00 2024-04-06 04:51:42 0:36:42 0:25:21 0:11:21 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7642331 2024-04-06 03:09:29 2024-04-06 04:16:30 2024-04-06 04:36:18 0:19:48 0:10:09 0:09:39 smithi main ubuntu 22.04 upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} 1
Failure Reason:

"2024-04-06T04:35:25.664759+0000 mon.a (mon.0) 38 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7642332 2024-04-06 03:09:30 2024-04-06 04:16:31 2024-04-06 06:16:55 2:00:24 1:47:30 0:12:54 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-04-06T04:48:10.018513+0000 mon.a (mon.0) 352 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

dead 7642333 2024-04-06 03:09:30 2024-04-06 04:18:51 2024-04-06 16:28:49 12:09:58 smithi main centos 8.stream upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_8.stream_container_tools_crun} 0-start 1-tasks} 2
Failure Reason:

hit max job timeout

fail 7642334 2024-04-06 03:09:31 2024-04-06 04:19:42 2024-04-06 05:07:38 0:47:56 0:37:27 0:10:29 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7642335 2024-04-06 03:09:32 2024-04-06 04:20:22 2024-04-06 06:00:31 1:40:09 1:29:41 0:10:28 smithi main centos 9.stream upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1712378601.130643 mon.a (mon.0) 666 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7642336 2024-04-06 03:09:33 2024-04-06 04:21:03 2024-04-06 05:44:36 1:23:33 1:12:33 0:11:00 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

pass 7642337 2024-04-06 03:09:34 2024-04-06 04:22:53 2024-04-06 04:46:47 0:23:54 0:12:04 0:11:50 smithi main centos 8.stream upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} 1
fail 7642338 2024-04-06 03:09:35 2024-04-06 04:25:54 2024-04-06 05:31:03 1:05:09 0:52:13 0:12:56 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7642339 2024-04-06 03:09:37 2024-04-06 04:26:15 2024-04-06 07:38:12 3:11:57 3:01:07 0:10:50 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} 2
Failure Reason:

"2024-04-06T06:23:32.370161+0000 osd.2 (osd.2) 10 : cluster [ERR] 59.1a soid 59:5be339ec:::smithi123745507-33:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log

fail 7642340 2024-04-06 03:09:38 2024-04-06 04:26:35 2024-04-06 05:10:36 0:44:01 0:32:33 0:11:28 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} 2
Failure Reason:

timeout expired in wait_until_healthy

dead 7642341 2024-04-06 03:09:39 2024-04-06 04:27:26 2024-04-06 16:58:21 12:30:55 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

hit max job timeout

fail 7642342 2024-04-06 03:09:40 2024-04-06 04:30:26 2024-04-06 07:30:05 2:59:39 2:47:48 0:11:51 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

"2024-04-06T06:38:29.581324+0000 osd.5 (osd.5) 6 : cluster [ERR] 59.16 soid 59:68fe90f4:::smithi066785146-41:162 : object info inconsistent , attr value mismatch '__header'" in cluster log

fail 7642343 2024-04-06 03:09:41 2024-04-06 04:30:27 2024-04-06 05:21:27 0:51:00 0:40:47 0:10:13 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7642344 2024-04-06 03:09:42 2024-04-06 04:30:27 2024-04-06 08:07:40 3:37:13 3:28:01 0:09:12 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} 2
Failure Reason:

"2024-04-06T06:37:02.220450+0000 osd.3 (osd.3) 40 : cluster [ERR] 62.b soid 62:d2e0579c:::smithi082775949-37:13c : object info inconsistent , attr value mismatch '__header'" in cluster log

pass 7642345 2024-04-06 03:09:43 2024-04-06 04:30:28 2024-04-06 04:59:24 0:28:56 0:19:43 0:09:13 smithi main centos 8.stream upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
fail 7642346 2024-04-06 03:09:44 2024-04-06 04:30:28 2024-04-06 05:27:32 0:57:04 0:47:03 0:10:01 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7642347 2024-04-06 03:09:45 2024-04-06 04:30:28 2024-04-06 05:17:53 0:47:25 0:38:10 0:09:15 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7642348 2024-04-06 03:09:46 2024-04-06 04:30:49 2024-04-06 05:29:15 0:58:26 0:45:23 0:13:03 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7642349 2024-04-06 03:09:47 2024-04-06 04:33:10 2024-04-06 05:26:55 0:53:45 0:44:58 0:08:47 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7642350 2024-04-06 03:09:48 2024-04-06 04:33:10 2024-04-06 05:24:58 0:51:48 0:41:45 0:10:03 smithi main ubuntu 22.04 upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks} 2
Failure Reason:

"1712379602.8240392 mon.a (mon.0) 513 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7642351 2024-04-06 03:09:49 2024-04-06 04:33:20 2024-04-06 06:47:03 2:13:43 2:03:57 0:09:46 smithi main ubuntu 22.04 upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-04-06T04:54:54.398932+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7642352 2024-04-06 03:09:50 2024-04-06 04:33:21 2024-04-06 05:48:51 1:15:30 0:59:56 0:15:34 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi105 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a3a1393a-f3d1-11ee-b649-cb9ed24678a4 -e sha1=378890f9ebb03f3af32caf3912974fcbab0e2abf -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\''

fail 7642353 2024-04-06 03:09:51 2024-04-06 04:33:41 2024-04-06 06:27:48 1:54:07 1:43:09 0:10:58 smithi main ubuntu 22.04 upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1712379627.1631262 mon.a (mon.0) 511 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7642354 2024-04-06 03:09:52 2024-04-06 04:34:02 2024-04-06 07:40:05 3:06:03 2:53:43 0:12:20 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} 2
Failure Reason:

"2024-04-06T06:22:14.247321+0000 osd.6 (osd.6) 33 : cluster [ERR] 57.5 soid 57:a0249c33:::smithi186678291-199:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log

fail 7642355 2024-04-06 03:09:53 2024-04-06 04:34:42 2024-04-06 05:20:54 0:46:12 0:34:02 0:12:10 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7642356 2024-04-06 03:09:54 2024-04-06 04:35:13 2024-04-06 05:10:07 0:34:54 0:21:25 0:13:29 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

Command failed on smithi038 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.30 deep-scrub'

fail 7642357 2024-04-06 03:09:55 2024-04-06 04:38:53 2024-04-06 08:07:22 3:28:29 3:15:36 0:12:53 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

"2024-04-06T05:10:00.000161+0000 mon.a (mon.0) 1177 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log

fail 7642358 2024-04-06 03:09:56 2024-04-06 04:39:34 2024-04-06 05:27:14 0:47:40 0:37:17 0:10:23 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7642359 2024-04-06 03:09:57 2024-04-06 04:40:04 2024-04-06 05:34:01 0:53:57 0:41:16 0:12:41 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

dead 7642360 2024-04-06 03:09:58 2024-04-06 04:41:55 2024-04-06 04:47:00 0:05:05 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} 2
Failure Reason:

Error reimaging machines: Failed to power on smithi031

fail 7642361 2024-04-06 03:09:59 2024-04-06 04:45:56 2024-04-06 05:41:39 0:55:43 0:44:50 0:10:53 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/connectivity} 2
Failure Reason:

timeout expired in wait_until_healthy

dead 7642362 2024-04-06 03:10:00 2024-04-06 04:45:56 2024-04-06 16:54:26 12:08:30 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

hit max job timeout

fail 7642363 2024-04-06 03:10:01 2024-04-06 04:45:57 2024-04-06 07:41:10 2:55:13 2:44:33 0:10:40 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

"2024-04-06T05:16:56.545866+0000 mon.a (mon.0) 487 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7642364 2024-04-06 03:10:02 2024-04-06 04:45:57 2024-04-06 05:38:58 0:53:01 0:43:35 0:09:26 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

pass 7642365 2024-04-06 03:10:03 2024-04-06 04:45:58 2024-04-06 06:28:41 1:42:43 1:33:36 0:09:07 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/kernel_untar_build}} 2