Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7437361 2023-10-25 15:27:07 2023-10-25 15:28:07 2023-10-25 18:54:57 3:26:50 3:15:34 0:11:16 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-many-objects} 2
pass 7437362 2023-10-25 15:27:07 2023-10-25 15:29:27 2023-10-25 21:18:09 5:48:42 5:37:38 0:11:04 smithi main ubuntu 22.04 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-many-objects} 2
pass 7437363 2023-10-25 15:27:08 2023-10-25 15:29:58 2023-10-25 20:34:25 5:04:27 4:53:24 0:11:03 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-many-objects} 2
pass 7437364 2023-10-25 15:27:09 2023-10-25 15:30:38 2023-10-25 19:10:41 3:40:03 3:29:34 0:10:29 smithi main ubuntu 22.04 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-many-objects} 2
pass 7437365 2023-10-25 15:27:10 2023-10-25 15:30:48 2023-10-25 21:03:48 5:33:00 5:15:47 0:17:13 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-many-objects} 2
pass 7437366 2023-10-25 15:27:11 2023-10-25 15:38:00 2023-10-25 20:21:00 4:43:00 4:31:25 0:11:35 smithi main ubuntu 22.04 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-many-objects} 2
pass 7437367 2023-10-25 15:27:12 2023-10-25 15:38:40 2023-10-25 19:13:31 3:34:51 3:24:56 0:09:55 smithi main ubuntu 22.04 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-many-objects} 2
pass 7437368 2023-10-25 15:27:12 2023-10-25 15:39:21 2023-10-25 20:29:13 4:49:52 4:39:19 0:10:33 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-many-objects} 2
pass 7437369 2023-10-25 15:27:13 2023-10-25 15:39:51 2023-10-25 20:12:39 4:32:48 4:21:31 0:11:17 smithi main ubuntu 22.04 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-many-objects} 2
pass 7437370 2023-10-25 15:27:14 2023-10-25 15:40:01 2023-10-25 19:06:26 3:26:25 3:16:00 0:10:25 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/snaps-many-objects} 2
pass 7437371 2023-10-25 15:27:15 2023-10-25 15:41:12 2023-10-25 23:38:56 7:57:44 7:46:12 0:11:32 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/snaps-many-objects} 2
pass 7437372 2023-10-25 15:27:16 2023-10-25 15:43:13 2023-10-25 21:03:59 5:20:46 5:09:07 0:11:39 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/snaps-many-objects} 2
fail 7437373 2023-10-25 15:27:16 2023-10-25 15:44:43 2023-10-26 00:31:57 8:47:14 8:34:58 0:12:16 smithi main ubuntu 22.04 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-many-objects} 2
Failure Reason:

"2023-10-25T16:46:25.543350+0000 osd.2 (osd.2) 365 : cluster [WRN] osd.2 PgScrubber: 3.14c timeout on reserving replicas (since 2023-10-25T16:46:20:537975+0000)" in cluster log

pass 7437374 2023-10-25 15:27:17 2023-10-25 15:45:34 2023-10-25 23:09:15 7:23:41 7:14:37 0:09:04 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/snaps-many-objects} 2
dead 7437375 2023-10-25 15:27:18 2023-10-25 15:45:34 2023-10-26 04:50:20 13:04:46 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/snaps-many-objects} 2
Failure Reason:

hit max job timeout

pass 7437376 2023-10-25 15:27:19 2023-10-25 15:45:54 2023-10-26 01:27:10 9:41:16 9:29:46 0:11:30 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/snaps-many-objects} 2
pass 7437377 2023-10-25 15:27:20 2023-10-25 15:46:35 2023-10-25 23:29:21 7:42:46 7:32:00 0:10:46 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/snaps-many-objects} 2
fail 7437378 2023-10-25 15:27:20 2023-10-25 15:47:05 2023-10-25 21:59:26 6:12:21 6:00:04 0:12:17 smithi main ubuntu 22.04 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-many-objects} 2
Failure Reason:

failed to complete snap trimming before timeout

pass 7437379 2023-10-25 15:27:21 2023-10-25 15:49:26 2023-10-26 01:09:13 9:19:47 9:08:52 0:10:55 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/snaps-many-objects} 2
fail 7437380 2023-10-25 15:27:22 2023-10-25 15:49:26 2023-10-26 00:19:39 8:30:13 8:19:40 0:10:33 smithi main centos 9.stream rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/snaps-many-objects} 2
Failure Reason:

"2023-10-25T19:17:45.141137+0000 osd.6 (osd.6) 183 : cluster [WRN] osd.6 PgScrubber: 3.92a timeout on reserving replicas (since 2023-10-25T19:17:40:140844+0000)" in cluster log