Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7675765 2024-04-27 03:08:42 2024-04-27 03:24:25 2024-04-27 04:01:39 0:37:14 0:24:47 0:12:27 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7675766 2024-04-27 03:08:43 2024-04-27 03:26:06 2024-04-27 03:45:45 0:19:39 0:09:57 0:09:42 smithi main ubuntu 22.04 upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} 1
Failure Reason:

"2024-04-27T03:44:44.323875+0000 mon.a (mon.0) 38 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7675767 2024-04-27 03:08:44 2024-04-27 03:26:06 2024-04-27 05:24:21 1:58:15 1:47:45 0:10:30 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-04-27T03:50:00.000129+0000 mon.a (mon.0) 1058 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log

fail 7675768 2024-04-27 03:08:45 2024-04-27 03:28:47 2024-04-27 03:46:45 0:17:58 0:07:08 0:10:50 smithi main ubuntu 22.04 upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=pacific

fail 7675769 2024-04-27 03:08:46 2024-04-27 03:29:07 2024-04-27 04:13:12 0:44:05 0:37:07 0:06:58 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7675770 2024-04-27 03:08:47 2024-04-27 03:29:58 2024-04-27 05:23:55 1:53:57 1:43:36 0:10:21 smithi main ubuntu 22.04 upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-04-27T04:08:14.008998+0000 mon.a (mon.0) 18 : cluster [ERR] Health check failed: 8 osds(s) are not reachable (OSD_UNREACHABLE)" in cluster log

fail 7675771 2024-04-27 03:08:48 2024-04-27 03:30:18 2024-04-27 06:57:01 3:26:43 3:18:21 0:08:22 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-04-27T05:29:51.031134+0000 osd.0 (osd.0) 30 : cluster [ERR] 62.e soid 62:716f7562:::smithi148706666-6:f7 : object info inconsistent , attr value mismatch '__header'" in cluster log

pass 7675772 2024-04-27 03:08:49 2024-04-27 03:31:29 2024-04-27 03:52:42 0:21:13 0:11:36 0:09:37 smithi main centos 8.stream upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} 1
pass 7675773 2024-04-27 03:08:50 2024-04-27 03:31:59 2024-04-27 03:53:38 0:21:39 0:12:08 0:09:31 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/new_ops/{newops}}} 2
fail 7675774 2024-04-27 03:08:51 2024-04-27 03:32:50 2024-04-27 06:44:47 3:11:57 3:01:20 0:10:37 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-04-27T05:09:50.437845+0000 osd.3 (osd.3) 105 : cluster [ERR] 54.1b soid 54:d89b1da2:::smithi089498371-46:9c : object info inconsistent , attr value mismatch '__header'" in cluster log

fail 7675775 2024-04-27 03:08:52 2024-04-27 03:35:30 2024-04-27 06:18:09 2:42:39 2:33:31 0:09:08 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-04-27T04:03:42.834624+0000 mon.a (mon.0) 260 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7675776 2024-04-27 03:08:53 2024-04-27 03:36:31 2024-04-27 05:57:25 2:20:54 2:11:28 0:09:26 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1714189954.1054058 mon.a (mon.0) 172 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 7675777 2024-04-27 03:08:54 2024-04-27 03:38:32 2024-04-27 04:52:14 1:13:42 1:01:12 0:12:30 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi046 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b1d3a2d0-0449-11ef-bc93-c7b262605968 -e sha1=4e213a58e787437ff508e6c890ccd5fd32dc96b5 -- bash -c 'ceph orch ps'"

fail 7675778 2024-04-27 03:08:55 2024-04-27 03:39:02 2024-04-27 04:35:39 0:56:37 0:45:50 0:10:47 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 7675779 2024-04-27 03:08:56 2024-04-27 03:39:23 2024-04-27 06:52:23 3:13:00 3:02:44 0:10:16 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-04-27T05:30:00.930030+0000 osd.1 (osd.1) 67 : cluster [ERR] 49.17 soid 49:eb6b7ef9:::smithi016368073-41:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log

fail 7675780 2024-04-27 03:08:57 2024-04-27 03:40:33 2024-04-27 07:04:42 3:24:09 3:15:08 0:09:01 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-04-27T04:06:30.510592+0000 mon.a (mon.0) 15 : cluster [WRN] Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled" in cluster log

pass 7675781 2024-04-27 03:08:59 2024-04-27 03:43:34 2024-04-27 04:13:43 0:30:09 0:20:43 0:09:26 smithi main centos 8.stream upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
fail 7675782 2024-04-27 03:09:00 2024-04-27 03:44:24 2024-04-27 06:28:26 2:44:02 2:35:18 0:08:44 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-04-27T05:25:29.316125+0000 osd.2 (osd.2) 41 : cluster [ERR] 87.0 soid 87:0264cea0:::smithi080536477-33:8c : object info inconsistent , attr value mismatch '__header'" in cluster log

fail 7675783 2024-04-27 03:09:01 2024-04-27 03:45:05 2024-04-27 04:31:09 0:46:04 0:38:45 0:07:19 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7675784 2024-04-27 03:09:02 2024-04-27 03:45:45 2024-04-27 06:45:44 2:59:59 2:53:06 0:06:53 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-04-27T05:30:02.243530+0000 osd.4 (osd.4) 78 : cluster [ERR] 59.13 soid 59:cf4f3704:::smithi113790443-39:77 : object info inconsistent , attr value mismatch '__header'" in cluster log

fail 7675785 2024-04-27 03:09:03 2024-04-27 03:45:56 2024-04-27 06:35:45 2:49:49 2:43:19 0:06:30 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-04-27T05:22:10.284187+0000 osd.2 (osd.2) 85 : cluster [ERR] 51.32 soid 51:4c82980a:::smithi045599431-38:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log

fail 7675786 2024-04-27 03:09:04 2024-04-27 03:46:06 2024-04-27 04:23:30 0:37:24 0:29:39 0:07:45 smithi main centos 9.stream upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream} 0-start 1-tasks} 2
Failure Reason:

"1714190579.6498067 mon.a (mon.0) 516 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7675787 2024-04-27 03:09:05 2024-04-27 03:46:27 2024-04-27 05:28:46 1:42:19 1:34:59 0:07:20 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

Command failed (workunit test rbd/test_librbd.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh'

dead 7675788 2024-04-27 03:09:06 2024-04-27 03:47:37 2024-04-27 16:16:18 12:28:41 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

hit max job timeout

fail 7675789 2024-04-27 03:09:07 2024-04-27 05:23:10 5194 smithi main centos 9.stream upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1714190811.844381 mon.a (mon.0) 664 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

pass 7675790 2024-04-27 03:09:08 2024-04-27 03:49:28 2024-04-27 04:39:43 0:50:15 0:36:37 0:13:38 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/kernel_untar_build}} 2
fail 7675791 2024-04-27 03:09:09 2024-04-27 03:52:29 2024-04-27 07:03:30 3:11:01 2:59:44 0:11:17 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-04-27T06:02:02.370938+0000 osd.1 (osd.1) 34 : cluster [ERR] 58.1f soid 58:f923ddaf:::smithi038474460-14:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log

fail 7675792 2024-04-27 03:09:10 2024-04-27 03:53:40 2024-04-27 06:43:53 2:50:13 2:40:16 0:09:57 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"1714191627.1958978 mon.a (mon.0) 518 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

pass 7675793 2024-04-27 03:09:11 2024-04-27 03:54:10 2024-04-27 04:26:55 0:32:45 0:22:52 0:09:53 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
fail 7675794 2024-04-27 03:09:12 2024-04-27 03:54:11 2024-04-27 07:43:54 3:49:43 3:36:36 0:13:07 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-04-27T04:18:39.532041+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

fail 7675795 2024-04-27 03:09:13 2024-04-27 03:54:11 2024-04-27 05:08:30 1:14:19 1:06:33 0:07:46 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7675796 2024-04-27 03:09:14 2024-04-27 03:55:02 2024-04-27 07:29:07 3:34:05 3:19:21 0:14:44 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-04-27T06:10:29.720796+0000 osd.4 (osd.4) 53 : cluster [ERR] 53.10 soid 53:0ad8f6ff:::smithi002372312-41:1d2 : object info inconsistent , attr value mismatch '__header'" in cluster log

fail 7675797 2024-04-27 03:09:15 2024-04-27 03:59:13 2024-04-27 06:34:44 2:35:31 2:27:07 0:08:24 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-04-27T05:35:53.826341+0000 osd.3 (osd.3) 16 : cluster [ERR] 58.6 soid 58:60c4f45a:::smithi064683022-406:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log

fail 7675798 2024-04-27 03:09:16 2024-04-27 03:59:23 2024-04-27 06:19:02 2:19:39 2:10:04 0:09:35 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

"2024-04-27T05:28:48.672634+0000 osd.5 (osd.5) 4 : cluster [ERR] 51.0 soid 51:0377c049:::smithi121508203-26:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log

fail 7675799 2024-04-27 03:09:17 2024-04-27 04:01:44 2024-04-27 07:47:18 3:45:34 3:35:04 0:10:30 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi150 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3067e9f133f954b642fdf21114a79edb58590948 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

fail 7675800 2024-04-27 03:09:18 2024-04-27 04:01:44 2024-04-27 07:04:29 3:02:45 2:52:34 0:10:11 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-04-27T05:44:18.811479+0000 osd.0 (osd.0) 16 : cluster [ERR] 57.14 soid 57:28658920:::smithi129778568-93:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log

fail 7675801 2024-04-27 03:09:19 2024-04-27 04:04:25 2024-04-27 07:04:38 3:00:13 2:53:07 0:07:06 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

"2024-04-27T05:38:20.894888+0000 osd.1 (osd.1) 14 : cluster [ERR] 49.16 soid 49:68184bc8:::smithi153640144-422:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log