| User | Scheduled | Started | Updated | Runtime | Suite | Branch | Machine Type | Revision | Pass | Fail | Dead |
|---|---|---|---|---|---|---|---|---|---|---|---|
| yuriw | 2024-04-11 17:03:56 | 2024-04-11 23:32:20 | 2024-04-12 12:05:52 | 12:33:32 | upgrade | wip-yuri6-testing-2024-04-02-1310 | smithi | a5074d4 | 5 | 44 | 1 |
| Status | Job ID | Links | Posted | Started | Updated | Runtime |
Duration |
In Waiting |
Machine | Teuthology Branch | OS Type | OS Version | Description | Nodes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| fail | 7652413 |
|
2024-04-11 17:04:27 | 2024-04-11 23:32:20 | 2024-04-12 01:24:15 | 1:51:55 | 1:41:36 | 0:10:19 | smithi | main | ubuntu | 22.04 | upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
"1712880044.3607485 mon.a (mon.0) 654 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log |
||||||||||||||
| fail | 7652414 |
|
2024-04-11 17:04:28 | 2024-04-11 23:32:20 | 2024-04-12 02:33:13 | 3:00:53 | 2:49:14 | 0:11:39 | smithi | main | centos | 9.stream | upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-11T23:53:30.726263+0000 mon.a (mon.0) 173 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log |
||||||||||||||
| fail | 7652415 |
|
2024-04-11 17:04:29 | 2024-04-11 23:32:20 | 2024-04-12 02:34:38 | 3:02:18 | 2:47:57 | 0:14:21 | smithi | main | centos | 9.stream | upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"1712880000.0001683 mon.a (mon.0) 781 : cluster [ERR] Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds" in cluster log |
||||||||||||||
| fail | 7652416 |
|
2024-04-11 17:04:30 | 2024-04-11 23:35:21 | 2024-04-12 00:06:35 | 0:31:14 | 0:21:20 | 0:09:54 | smithi | main | centos | 8.stream | upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} | 3 |
Failure Reason:
Command failed on smithi008 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.d deep-scrub' |
||||||||||||||
| fail | 7652417 |
|
2024-04-11 17:04:31 | 2024-04-11 23:35:22 | 2024-04-12 02:47:41 | 3:12:19 | 3:02:18 | 0:10:01 | smithi | main | ubuntu | 22.04 | upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"2024-04-11T23:57:44.927912+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log |
||||||||||||||
| fail | 7652418 |
|
2024-04-11 17:04:33 | 2024-04-11 23:35:42 | 2024-04-12 02:14:09 | 2:38:27 | 2:27:23 | 0:11:04 | smithi | main | ubuntu | 22.04 | upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"1712880130.2303743 mon.a (mon.0) 518 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log |
||||||||||||||
| fail | 7652419 |
|
2024-04-11 17:04:34 | 2024-04-11 23:36:13 | 2024-04-12 03:29:00 | 3:52:47 | 3:42:21 | 0:10:26 | smithi | main | ubuntu | 22.04 | upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-12T01:51:29.348933+0000 osd.0 (osd.0) 20 : cluster [ERR] 61.8 soid 61:10408764:::smithi060489920-187:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| fail | 7652420 |
|
2024-04-11 17:04:35 | 2024-04-11 23:37:03 | 2024-04-12 03:12:49 | 3:35:46 | 3:23:20 | 0:12:26 | smithi | main | ubuntu | 22.04 | upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"1712880514.3627746 mon.a (mon.0) 716 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log |
||||||||||||||
| fail | 7652421 |
|
2024-04-11 17:04:36 | 2024-04-11 23:38:04 | 2024-04-12 02:27:05 | 2:49:01 | 2:37:15 | 0:11:46 | smithi | main | centos | 9.stream | upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"2024-04-12T01:22:41.864949+0000 osd.1 (osd.1) 43 : cluster [ERR] 58.1 soid 58:8080e171:::smithi099676674-390:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| fail | 7652422 |
|
2024-04-11 17:04:37 | 2024-04-11 23:39:44 | 2024-04-12 02:03:35 | 2:23:51 | 2:12:24 | 0:11:27 | smithi | main | centos | 9.stream | upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"2024-04-12T01:17:41.288380+0000 osd.3 (osd.3) 55 : cluster [ERR] 51.1d soid 51:b8be0aa0:::smithi040512622-23:136 : object info inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| fail | 7652423 |
|
2024-04-11 17:04:38 | 2024-04-11 23:40:15 | 2024-04-12 00:19:34 | 0:39:19 | 0:30:02 | 0:09:17 | smithi | main | centos | 9.stream | upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream} 0-start 1-tasks} | 2 |
Failure Reason:
"1712880145.3318021 mon.a (mon.0) 656 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log |
||||||||||||||
| pass | 7652424 |
|
2024-04-11 17:04:39 | 2024-04-11 23:40:15 | 2024-04-12 00:17:21 | 0:37:06 | 0:25:15 | 0:11:51 | smithi | main | centos | 8.stream | upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} | 3 |
| fail | 7652425 |
|
2024-04-11 17:04:40 | 2024-04-11 23:42:06 | 2024-04-12 00:01:27 | 0:19:21 | 0:09:48 | 0:09:33 | smithi | main | ubuntu | 22.04 | upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} | 1 |
Failure Reason:
"2024-04-12T00:00:02.648792+0000 mon.a (mon.0) 38 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log |
||||||||||||||
| fail | 7652426 |
|
2024-04-11 17:04:41 | 2024-04-11 23:42:06 | 2024-04-12 01:56:29 | 2:14:23 | 2:04:08 | 0:10:15 | smithi | main | ubuntu | 22.04 | upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
"2024-04-12T00:23:15.023295+0000 mon.a (mon.0) 421 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log |
||||||||||||||
| fail | 7652427 |
|
2024-04-11 17:04:42 | 2024-04-11 23:42:17 | 2024-04-12 00:04:37 | 0:22:20 | 0:10:54 | 0:11:26 | smithi | main | centos | 8.stream | upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} | 1 |
Failure Reason:
ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8. |
||||||||||||||
| fail | 7652428 |
|
2024-04-11 17:04:44 | 2024-04-11 23:42:17 | 2024-04-12 01:20:36 | 1:38:19 | 1:27:55 | 0:10:24 | smithi | main | centos | 9.stream | upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
"1712880241.3168797 mon.a (mon.0) 504 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log |
||||||||||||||
| fail | 7652429 |
|
2024-04-11 17:04:45 | 2024-04-11 23:43:08 | 2024-04-12 01:05:02 | 1:21:54 | 1:09:59 | 0:11:55 | smithi | main | centos | 9.stream | upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
reached maximum tries (51) after waiting for 300 seconds |
||||||||||||||
| fail | 7652430 |
|
2024-04-11 17:04:46 | 2024-04-11 23:45:49 | 2024-04-12 03:14:18 | 3:28:29 | 3:18:19 | 0:10:10 | smithi | main | centos | 9.stream | upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-12T00:10:00.000129+0000 mon.a (mon.0) 692 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled" in cluster log |
||||||||||||||
| fail | 7652431 |
|
2024-04-11 17:04:47 | 2024-04-11 23:46:59 | 2024-04-12 02:36:10 | 2:49:11 | 2:39:16 | 0:09:55 | smithi | main | centos | 9.stream | upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"1712880510.6210227 mon.a (mon.0) 503 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log |
||||||||||||||
| fail | 7652432 |
|
2024-04-11 17:04:48 | 2024-04-11 23:47:00 | 2024-04-12 03:13:14 | 3:26:14 | 3:15:17 | 0:10:57 | smithi | main | ubuntu | 22.04 | upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"2024-04-12T00:08:21.904596+0000 mon.a (mon.0) 180 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log |
||||||||||||||
| pass | 7652433 |
|
2024-04-11 17:04:49 | 2024-04-11 23:47:10 | 2024-04-12 00:33:29 | 0:46:19 | 0:33:36 | 0:12:43 | smithi | main | centos | 9.stream | upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
| fail | 7652434 |
|
2024-04-11 17:04:50 | 2024-04-11 23:50:01 | 2024-04-12 02:36:30 | 2:46:29 | 2:35:22 | 0:11:07 | smithi | main | ubuntu | 22.04 | upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"1712881295.1855903 mon.a (mon.0) 691 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log |
||||||||||||||
| fail | 7652435 |
|
2024-04-11 17:04:51 | 2024-04-11 23:50:01 | 2024-04-12 03:05:26 | 3:15:25 | 3:02:20 | 0:13:05 | smithi | main | centos | 9.stream | upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-12T01:41:53.189925+0000 osd.4 (osd.4) 19 : cluster [ERR] 58.15 soid 58:a8583926:::smithi027748387-448:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| fail | 7652436 |
|
2024-04-11 17:04:52 | 2024-04-11 23:50:52 | 2024-04-12 02:38:14 | 2:47:22 | 2:37:23 | 0:09:59 | smithi | main | centos | 9.stream | upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-12T01:31:36.612722+0000 osd.5 (osd.5) 50 : cluster [ERR] 51.6 soid 51:6330c76a:::smithi078514566-33:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| fail | 7652437 |
|
2024-04-11 17:04:53 | 2024-04-11 23:50:52 | 2024-04-12 02:06:14 | 2:15:22 | 2:03:21 | 0:12:01 | smithi | main | ubuntu | 22.04 | upgrade/reef-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
"2024-04-12T00:32:52.017889+0000 mon.a (mon.0) 315 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log |
||||||||||||||
| fail | 7652438 |
|
2024-04-11 17:04:54 | 2024-04-11 23:51:43 | 2024-04-12 00:40:42 | 0:48:59 | 0:39:33 | 0:09:26 | smithi | main | centos | 9.stream | upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
reached maximum tries (51) after waiting for 300 seconds |
||||||||||||||
| fail | 7652439 |
|
2024-04-11 17:04:55 | 2024-04-11 23:51:43 | 2024-04-12 01:28:52 | 1:37:09 | 1:27:48 | 0:09:21 | smithi | main | centos | 9.stream | upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
"1712880804.3013058 mon.a (mon.0) 662 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log |
||||||||||||||
| fail | 7652440 |
|
2024-04-11 17:04:56 | 2024-04-11 23:51:43 | 2024-04-12 02:46:52 | 2:55:09 | 2:45:35 | 0:09:34 | smithi | main | centos | 9.stream | upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"2024-04-12T01:58:18.353703+0000 osd.7 (osd.7) 62 : cluster [ERR] 59.16 soid 59:68108c2a:::smithi047789157-29:1a5 : object info inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| fail | 7652441 |
|
2024-04-11 17:04:57 | 2024-04-11 23:51:44 | 2024-04-12 02:39:56 | 2:48:12 | 2:38:22 | 0:09:50 | smithi | main | centos | 9.stream | upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"1712880864.8266687 mon.a (mon.0) 503 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log |
||||||||||||||
| pass | 7652442 |
|
2024-04-11 17:04:58 | 2024-04-11 23:52:34 | 2024-04-12 00:36:09 | 0:43:35 | 0:33:27 | 0:10:08 | smithi | main | centos | 9.stream | upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
| fail | 7652443 |
|
2024-04-11 17:04:59 | 2024-04-11 23:53:05 | 2024-04-12 03:31:15 | 3:38:10 | 3:25:49 | 0:12:21 | smithi | main | centos | 9.stream | upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-12T00:20:00.000234+0000 mon.a (mon.0) 747 : cluster 4 [ERR] MDS_ALL_DOWN: 1 filesystem is offline" in cluster log |
||||||||||||||
| fail | 7652444 |
|
2024-04-11 17:05:01 | 2024-04-11 23:55:46 | 2024-04-12 03:07:43 | 3:11:57 | 2:59:41 | 0:12:16 | smithi | main | centos | 9.stream | upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-12T01:32:23.788571+0000 osd.3 (osd.3) 30 : cluster [ERR] 53.6 soid 53:601bbee3:::smithi007551084-280:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| fail | 7652445 |
|
2024-04-11 17:05:02 | 2024-04-11 23:55:46 | 2024-04-12 00:18:59 | 0:23:13 | 0:12:42 | 0:10:31 | smithi | main | centos | 8.stream | upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}} | 2 |
Failure Reason:
Command failed on smithi142 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd' |
||||||||||||||
| fail | 7652446 |
|
2024-04-11 17:05:03 | 2024-04-11 23:55:46 | 2024-04-12 03:05:45 | 3:09:59 | 2:59:39 | 0:10:20 | smithi | main | ubuntu | 22.04 | upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"2024-04-12T00:36:17.935694+0000 mon.a (mon.0) 75 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log |
||||||||||||||
| fail | 7652447 |
|
2024-04-11 17:05:04 | 2024-04-11 23:55:57 | 2024-04-12 02:35:59 | 2:40:02 | 2:30:18 | 0:09:44 | smithi | main | ubuntu | 22.04 | upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"1712881059.0069582 mon.a (mon.0) 181 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log |
||||||||||||||
| fail | 7652448 |
|
2024-04-11 17:05:05 | 2024-04-11 23:56:07 | 2024-04-12 00:45:27 | 0:49:20 | 0:38:13 | 0:11:07 | smithi | main | centos | 9.stream | upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
reached maximum tries (51) after waiting for 300 seconds |
||||||||||||||
| fail | 7652449 |
|
2024-04-11 17:05:07 | 2024-04-11 23:56:08 | 2024-04-12 03:22:29 | 3:26:21 | 3:14:14 | 0:12:07 | smithi | main | centos | 9.stream | upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-12T01:54:56.952239+0000 osd.5 (osd.5) 97 : cluster [ERR] 58.21 soid 58:84d0aa43:::smithi104744111-36:8c : object info inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| fail | 7652450 |
|
2024-04-11 17:05:08 | 2024-04-11 23:57:18 | 2024-04-12 02:48:17 | 2:50:59 | 2:40:00 | 0:10:59 | smithi | main | centos | 9.stream | upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-12T01:40:02.899091+0000 osd.7 (osd.7) 34 : cluster [ERR] 50.25 soid 50:a4cc24c2:::smithi117515493-23:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| dead | 7652452 |
|
2024-04-11 17:05:09 | 2024-04-11 23:57:20 | 2024-04-12 12:05:52 | 12:08:32 | smithi | main | centos | 8.stream | upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_8.stream_container_tools} 0-start 1-tasks} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7652454 |
|
2024-04-11 17:05:10 | 2024-04-11 23:57:41 | 2024-04-12 00:32:46 | 0:35:05 | 0:25:31 | 0:09:34 | smithi | main | centos | 8.stream | upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} | 3 |
| fail | 7652456 |
|
2024-04-11 17:05:11 | 2024-04-11 23:57:42 | 2024-04-12 00:20:20 | 0:22:38 | 0:09:47 | 0:12:51 | smithi | main | ubuntu | 22.04 | upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} | 1 |
Failure Reason:
"2024-04-12T00:18:50.851767+0000 mon.a (mon.0) 38 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log |
||||||||||||||
| fail | 7652458 |
|
2024-04-11 17:05:12 | 2024-04-12 00:00:43 | 2024-04-12 02:00:19 | 1:59:36 | 1:46:00 | 0:13:36 | smithi | main | centos | 9.stream | upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
"2024-04-12T00:31:36.441011+0000 mon.a (mon.0) 318 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log |
||||||||||||||
| fail | 7652460 |
|
2024-04-11 17:05:13 | 2024-04-12 00:09:14 | 2024-04-12 01:03:30 | 0:54:16 | 0:38:37 | 0:15:39 | smithi | main | centos | 9.stream | upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} | 2 |
Failure Reason:
reached maximum tries (51) after waiting for 300 seconds |
||||||||||||||
| fail | 7652462 |
|
2024-04-11 17:05:14 | 2024-04-12 00:13:26 | 2024-04-12 02:11:26 | 1:58:00 | 1:42:57 | 0:15:03 | smithi | main | ubuntu | 22.04 | upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
"1712882691.7846346 mon.a (mon.0) 508 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log |
||||||||||||||
| fail | 7652464 |
|
2024-04-11 17:05:15 | 2024-04-12 00:17:27 | 2024-04-12 00:38:36 | 0:21:09 | 0:11:15 | 0:09:54 | smithi | main | centos | 8.stream | upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} | 1 |
Failure Reason:
ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8. |
||||||||||||||
| fail | 7652466 |
|
2024-04-11 17:05:16 | 2024-04-12 00:18:28 | 2024-04-12 03:39:46 | 3:21:18 | 3:11:18 | 0:10:00 | smithi | main | centos | 9.stream | upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"2024-04-12T02:36:50.841062+0000 osd.2 (osd.2) 36 : cluster [ERR] 103.15 soid 103:aa7f7378:::smithi082927016-5:head : object info inconsistent , snapset inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| pass | 7652468 |
|
2024-04-11 17:05:17 | 2024-04-12 00:26:30 | 2024-04-12 01:14:30 | 0:48:00 | 0:27:04 | 0:20:56 | smithi | main | centos | 8.stream | upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/nautilus centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/blogbench}} | 2 |
| fail | 7652470 |
|
2024-04-11 17:05:18 | 2024-04-12 00:28:51 | 2024-04-12 03:03:57 | 2:35:06 | 2:25:55 | 0:09:11 | smithi | main | centos | 9.stream | upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} | 2 |
Failure Reason:
"2024-04-12T02:12:59.519545+0000 osd.7 (osd.7) 36 : cluster [ERR] 87.1a soid 87:5c075576:::smithi077573484-41:f7 : object info inconsistent , attr value mismatch '__header'" in cluster log |
||||||||||||||
| fail | 7652471 |
|
2024-04-11 17:05:19 | 2024-04-12 00:28:51 | 2024-04-12 03:58:36 | 3:29:45 | 3:20:37 | 0:09:08 | smithi | main | ubuntu | 22.04 | upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} | 2 |
Failure Reason:
"2024-04-12T00:49:37.745559+0000 mon.a (mon.0) 182 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log |
||||||||||||||
| fail | 7652473 |
|
2024-04-11 17:05:20 | 2024-04-12 00:28:52 | 2024-04-12 02:20:21 | 1:51:29 | 1:42:10 | 0:09:19 | smithi | main | ubuntu | 22.04 | upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
"1712882973.5672646 mon.a (mon.0) 182 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log |
||||||||||||||
Pulpito