Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7650630 2024-04-10 14:18:56 2024-04-10 16:12:57 6157 smithi main ubuntu 22.04 upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1712760630.6249855 mon.a (mon.0) 658 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

pass 7650631 2024-04-10 14:18:57 2024-04-10 14:20:50 2024-04-10 15:06:07 0:45:17 0:32:59 0:12:18 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
dead 7650632 2024-04-10 14:18:58 2024-04-10 14:23:00 2024-04-11 03:32:12 13:09:12 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

dead 7650633 2024-04-10 14:18:59 2024-04-10 14:24:21 2024-04-11 03:23:31 12:59:10 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

fail 7650634 2024-04-10 14:19:00 2024-04-10 14:25:41 2024-04-10 14:59:23 0:33:42 0:21:21 0:12:21 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

Command failed on smithi057 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.1d deep-scrub'

dead 7650635 2024-04-10 14:19:01 2024-04-10 14:25:42 2024-04-11 03:07:53 12:42:11 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

hit max job timeout

fail 7650636 2024-04-10 14:19:02 2024-04-10 14:27:23 2024-04-10 15:44:15 1:16:52 1:06:57 0:09:55 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

dead 7650637 2024-04-10 14:19:03 2024-04-10 14:27:23 2024-04-11 03:05:19 12:37:56 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

hit max job timeout

dead 7650638 2024-04-10 14:19:04 2024-04-10 14:27:23 2024-04-11 03:53:04 13:25:41 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

dead 7650639 2024-04-10 14:19:05 2024-04-10 14:27:24 2024-04-11 03:31:21 13:03:57 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

pass 7650641 2024-04-10 14:19:06 2024-04-10 14:27:26 2024-04-10 15:10:21 0:42:55 0:33:10 0:09:45 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
dead 7650643 2024-04-10 14:19:07 2024-04-10 14:27:37 2024-04-11 03:16:44 12:49:07 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

hit max job timeout

dead 7650645 2024-04-10 14:19:08 2024-04-10 14:29:18 2024-04-11 03:10:05 12:40:47 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

hit max job timeout

fail 7650647 2024-04-10 14:19:09 2024-04-10 14:30:29 2024-04-10 15:10:15 0:39:46 0:30:07 0:09:39 smithi main centos 9.stream upgrade/telemetry-upgrade/quincy-x/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks} 2
Failure Reason:

"1712760694.3031788 mon.a (mon.0) 522 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

pass 7650649 2024-04-10 14:19:10 2024-04-10 14:31:20 2024-04-10 15:11:04 0:39:44 0:24:51 0:14:53 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7650651 2024-04-10 14:19:11 2024-04-10 14:36:51 2024-04-10 14:56:16 0:19:25 0:10:38 0:08:47 smithi main ubuntu 22.04 upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} 1
Failure Reason:

"2024-04-10T14:55:38.264327+0000 mon.a (mon.0) 34 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7650653 2024-04-10 14:19:12 2024-04-10 14:40:03 2024-04-10 16:38:30 1:58:27 1:46:44 0:11:43 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-04-10T15:10:09.172504+0000 mon.a (mon.0) 359 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7650655 2024-04-10 14:19:13 2024-04-10 14:41:34 2024-04-10 15:00:37 0:19:03 0:09:51 0:09:12 smithi main centos 8.stream upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} 1
Failure Reason:

ceph version 18.2.2-1189.gc96c8d47 was not installed, found 16.2.15-0.el8.

fail 7650657 2024-04-10 14:19:14 2024-04-10 14:41:34 2024-04-10 16:20:03 1:38:29 1:27:30 0:10:59 smithi main centos 9.stream upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1712761371.3916407 mon.a (mon.0) 505 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7650659 2024-04-10 14:19:15 2024-04-10 14:43:05 2024-04-10 16:00:06 1:17:01 1:07:06 0:09:55 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

dead 7650661 2024-04-10 14:19:16 2024-04-10 14:43:06 2024-04-11 03:43:30 13:00:24 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

dead 7650663 2024-04-10 14:19:17 2024-04-10 14:43:17 2024-04-11 03:54:25 13:11:08 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

dead 7650665 2024-04-10 14:19:18 2024-04-10 14:49:18 2024-04-11 03:35:37 12:46:19 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

hit max job timeout

pass 7650667 2024-04-10 14:19:19 2024-04-10 14:52:50 2024-04-10 15:36:00 0:43:10 0:32:44 0:10:26 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
dead 7650671 2024-04-10 14:19:21 2024-04-10 14:56:52 2024-04-11 03:56:48 12:59:56 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

pass 7650673 2024-04-10 14:19:22 2024-04-10 14:58:33 2024-04-10 15:29:50 0:31:17 0:19:54 0:11:23 smithi main centos 8.stream upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
dead 7650675 2024-04-10 14:19:23 2024-04-10 15:00:44 2024-04-11 04:06:22 13:05:38 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

fail 7650677 2024-04-10 14:19:24 2024-04-10 15:05:16 2024-04-10 17:03:26 1:58:10 1:47:24 0:10:46 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"2024-04-10T15:35:25.153712+0000 mon.a (mon.0) 345 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 7650679 2024-04-10 14:19:25 2024-04-10 15:09:17 2024-04-10 15:56:39 0:47:22 0:37:55 0:09:27 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

fail 7650681 2024-04-10 14:19:26 2024-04-10 15:09:38 2024-04-10 16:49:23 1:39:45 1:28:31 0:11:14 smithi main centos 9.stream upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

"1712762960.0813818 mon.a (mon.0) 370 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

dead 7650683 2024-04-10 14:19:28 2024-04-10 15:10:29 2024-04-11 03:55:10 12:44:41 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

hit max job timeout

dead 7650685 2024-04-10 14:19:29 2024-04-10 15:10:29 2024-04-11 03:54:51 12:44:22 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

hit max job timeout

pass 7650686 2024-04-10 14:19:30 2024-04-10 15:11:10 2024-04-10 15:54:45 0:43:35 0:32:38 0:10:57 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
dead 7650688 2024-04-10 14:19:31 2024-04-10 15:11:11 2024-04-11 04:22:16 13:11:05 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

fail 7650690 2024-04-10 14:19:32 2024-04-10 15:13:12 2024-04-10 15:45:11 0:31:59 0:21:36 0:10:23 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

Command failed on smithi055 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 4.38 deep-scrub'

dead 7650692 2024-04-10 14:19:33 2024-04-10 15:14:13 2024-04-11 04:31:28 13:17:15 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

fail 7650694 2024-04-10 14:19:34 2024-04-10 15:22:05 2024-04-10 15:45:01 0:22:56 0:13:38 0:09:18 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/iozone}} 2
Failure Reason:

Command failed on smithi087 with status 1: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd'

dead 7650696 2024-04-10 14:19:35 2024-04-10 15:22:45 2024-04-11 04:04:49 12:42:04 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

hit max job timeout

fail 7650699 2024-04-10 14:19:37 2024-04-10 15:29:38 2024-04-10 16:16:40 0:47:02 0:38:38 0:08:24 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

dead 7650701 2024-04-10 14:19:38 2024-04-10 15:29:38 2024-04-11 04:26:44 12:57:06 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

dead 7650703 2024-04-10 14:19:39 2024-04-10 15:31:49 2024-04-11 04:24:26 12:52:37 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

hit max job timeout

fail 7650704 2024-04-10 14:19:40 2024-04-10 15:32:30 2024-04-10 15:47:52 0:15:22 0:04:22 0:11:00 smithi main centos 9.stream upgrade/telemetry-upgrade/pacific-x/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=pacific

pass 7650707 2024-04-10 14:19:41 2024-04-10 15:32:51 2024-04-10 16:08:09 0:35:18 0:23:52 0:11:26 smithi main centos 8.stream upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 7650709 2024-04-10 14:19:42 2024-04-10 15:37:02 2024-04-10 15:56:54 0:19:52 0:10:24 0:09:28 smithi main ubuntu 22.04 upgrade/quincy-x/filestore-remove-check/{0-cluster/{openstack start} 1-ceph-install/quincy 2 - upgrade objectstore/filestore-xfs ubuntu_latest} 1
Failure Reason:

"2024-04-10T15:56:31.284434+0000 mon.a (mon.0) 38 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

dead 7650711 2024-04-10 14:19:43 2024-04-10 15:42:14 2024-04-11 03:55:43 12:13:29 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

hit max job timeout

dead 7650714 2024-04-10 14:19:45 2024-04-10 16:09:32 676 smithi main ubuntu 22.04 upgrade/quincy-x/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

['CPAN command failed: /usr/bin/cpanm Amazon::S3']

dead 7650716 2024-04-10 14:19:46 2024-04-10 15:50:07 2024-04-10 16:11:39 0:21:32 0:11:30 0:10:02 smithi main centos 8.stream upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} 1
Failure Reason:

['<https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>,', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at']

dead 7650718 2024-04-10 14:19:47 2024-04-10 16:20:15 872 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

smithi017.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - dnf-utils - sysstat - libedit - boost-thread - xfsprogs - gdisk - parted - libgcrypt - fuse-libs - openssl - libuuid - podman - attr - lsof - gettext - bc - xfsdump - blktrace - usbredir - valgrind - nfs-utils - ncurses-devel - gcc - git - genisoimage - qemu-img - qemu-kvm-core - qemu-kvm-block-rbd - libacl-devel - autoconf - gdb - iozone nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download packages: numactl-libs-2.0.16-4.el8.x86_64: Cannot download, all mirrors were already tried without success' results: []smithi076.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - dnf-utils - sysstat - libedit - boost-thread - xfsprogs - gdisk - parted - libgcrypt - fuse-libs - openssl - libuuid - podman - attr - lsof - gettext - bc - xfsdump - blktrace - usbredir - valgrind - nfs-utils - ncurses-devel - gcc - git - genisoimage - qemu-img - qemu-kvm-core - qemu-kvm-block-rbd - libacl-devel - autoconf - gdb - iozone nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download packages: numactl-libs-2.0.16-4.el8.x86_64: Cannot download, all mirrors were already tried without success' results: []

fail 7650720 2024-04-10 14:19:48 2024-04-10 16:00:40 2024-04-10 16:09:42 0:09:02 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream_runc 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7ecb9d8ac0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

fail 7650722 2024-04-10 14:19:49 2024-04-10 16:00:41 2024-04-10 16:10:34 0:09:53 smithi main ubuntu 22.04 upgrade/reef-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f3494181a60>: Failed to establish a new connection: [Errno 110] Connection timed out'))

fail 7650724 2024-04-10 14:19:50 2024-04-10 16:04:52 2024-04-10 16:13:37 0:08:45 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5ecc51fb20>: Failed to establish a new connection: [Errno 110] Connection timed out'))

fail 7650726 2024-04-10 14:19:51 2024-04-10 16:04:53 2024-04-10 16:18:07 0:13:14 smithi main centos 8.stream upgrade/cephfs/upgraded_client/{bluestore-bitmap branch/pacific centos_8.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/dbench}} 2
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f3ada762970>: Failed to establish a new connection: [Errno 110] Connection timed out'))

fail 7650727 2024-04-10 14:19:52 2024-04-10 16:08:14 2024-04-10 16:18:01 0:09:47 smithi main ubuntu 22.04 upgrade/quincy-x/stress-split/{0-distro/ubuntu_22.04 0-roles 1-start 2-first-half-tasks/readwrite 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f310d08aa90>: Failed to establish a new connection: [Errno 110] Connection timed out'))

fail 7650729 2024-04-10 14:19:53 2024-04-10 16:16:11 2024-04-10 16:26:00 0:09:49 smithi main centos 9.stream upgrade/reef-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f12112ffa90>: Failed to establish a new connection: [Errno 110] Connection timed out'))

fail 7650731 2024-04-10 14:19:54 2024-04-10 16:16:22 2024-04-10 16:26:12 0:09:50 smithi main centos 8.stream upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7d8c9e49d0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

fail 7650733 2024-04-10 14:19:55 2024-04-10 16:16:23 2024-04-10 16:25:18 0:08:55 smithi main centos 9.stream upgrade/quincy-x/stress-split/{0-distro/centos_9.stream 0-roles 1-start 2-first-half-tasks/snaps-few-objects 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fe1d4d1ea90>: Failed to establish a new connection: [Errno 110] Connection timed out'))

fail 7650735 2024-04-10 14:19:56 2024-04-10 16:16:23 2024-04-10 16:26:23 0:10:00 smithi main centos 9.stream upgrade/reef-x/parallel/{0-random-distro$/{centos_9.stream} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f93bf1e49d0>: Failed to establish a new connection: [Errno 110] Connection timed out'))

dead 7650737 2024-04-10 14:19:57 2024-04-10 16:16:24 2024-04-10 16:23:10 0:06:46 smithi main centos 9.stream upgrade/cephfs/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Error reimaging machines: Expected smithi152's OS to be centos 9 but found ubuntu 22.04

fail 7650739 2024-04-10 14:19:58 2024-04-10 16:16:25 2024-04-10 16:26:30 0:10:05 smithi main centos 9.stream upgrade/quincy-x/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd39cdc1a00>: Failed to establish a new connection: [Errno 110] Connection timed out'))

queued 2024-04-10 14:18:56 2024-04-10 14:18:56