| User | Scheduled | Started | Updated | Runtime | Suite | Branch | Machine Type | Revision | Pass | Fail | Dead |
|---|---|---|---|---|---|---|---|---|---|---|---|
| nmordech | 2024-09-01 08:15:15 | 2024-09-01 08:16:58 | 2024-09-01 23:51:33 | 15:34:35 | rados:thrash | main | smithi | 2c345e1 | 20 | 46 | 27 |
| Status | Job ID | Links | Posted | Started | Updated | Runtime |
Duration |
In Waiting |
Machine | Teuthology Branch | OS Type | OS Version | Description | Nodes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| fail | 7884509 |
|
2024-09-01 08:15:33 | 2024-09-01 08:16:58 | 2024-09-01 10:26:35 | 2:09:37 | 1:57:30 | 0:12:07 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T09:24:18.656970+0000 mon.a (mon.0) 10483 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 69 sec, osd.4 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884510 |
|
2024-09-01 08:15:34 | 2024-09-01 08:18:38 | 2024-09-01 16:31:51 | 8:13:13 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7884511 |
|
2024-09-01 08:15:36 | 2024-09-01 08:19:59 | 2024-09-01 16:33:33 | 8:13:34 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7884512 |
|
2024-09-01 08:15:37 | 2024-09-01 08:22:10 | 2024-09-01 16:33:41 | 8:11:31 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7884513 |
|
2024-09-01 08:15:39 | 2024-09-01 08:23:51 | 2024-09-01 16:40:48 | 8:16:57 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7884514 |
|
2024-09-01 08:15:40 | 2024-09-01 08:24:21 | 2024-09-01 10:31:53 | 2:07:32 | 2:01:03 | 0:06:29 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T10:19:13.943885+0000 osd.1 (osd.1) 141 : cluster [ERR] osd.1 found snap mapper error on pg 3.3 oid 3:c2580596:::smithi18727165-33:8c2 snaps in mapper: {}, oi: {89a} ...repaired" in cluster log |
||||||||||||||
| pass | 7884515 |
|
2024-09-01 08:15:41 | 2024-09-01 08:24:32 | 2024-09-01 10:31:32 | 2:07:00 | 1:58:34 | 0:08:26 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| dead | 7884516 |
|
2024-09-01 08:15:43 | 2024-09-01 08:26:23 | 2024-09-01 16:56:25 | 8:30:02 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7884517 |
|
2024-09-01 08:15:44 | 2024-09-01 08:29:04 | 2024-09-01 10:31:26 | 2:02:22 | 1:54:54 | 0:07:28 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| pass | 7884518 |
|
2024-09-01 08:15:45 | 2024-09-01 08:30:15 | 2024-09-01 10:38:31 | 2:08:16 | 1:58:14 | 0:10:02 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884519 |
|
2024-09-01 08:15:47 | 2024-09-01 08:30:55 | 2024-09-01 10:37:36 | 2:06:41 | 1:58:38 | 0:08:03 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T09:56:37.952310+0000 mon.a (mon.0) 18533 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 32 sec, osd.6 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884520 |
|
2024-09-01 08:15:48 | 2024-09-01 08:31:26 | 2024-09-01 10:42:45 | 2:11:19 | 1:59:52 | 0:11:27 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T09:01:08.462599+0000 mon.a (mon.0) 2170 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 34 sec, osd.2 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884521 |
|
2024-09-01 08:15:50 | 2024-09-01 08:32:57 | 2024-09-01 16:43:09 | 8:10:12 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7884522 |
|
2024-09-01 08:15:51 | 2024-09-01 08:33:27 | 2024-09-01 10:34:04 | 2:00:37 | 1:52:57 | 0:07:40 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884523 |
|
2024-09-01 08:15:52 | 2024-09-01 08:34:18 | 2024-09-01 10:40:44 | 2:06:26 | 1:59:23 | 0:07:03 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T09:55:23.640127+0000 mon.a (mon.0) 16704 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec, osd.14 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884524 |
|
2024-09-01 08:15:54 | 2024-09-01 08:35:29 | 2024-09-01 16:47:31 | 8:12:02 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7884525 |
|
2024-09-01 08:15:55 | 2024-09-01 08:37:30 | 2024-09-01 10:39:14 | 2:01:44 | 1:51:09 | 0:10:35 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| pass | 7884526 |
|
2024-09-01 08:15:56 | 2024-09-01 08:41:51 | 2024-09-01 11:01:45 | 2:19:54 | 2:05:26 | 0:14:28 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884527 |
|
2024-09-01 08:15:58 | 2024-09-01 08:45:52 | 2024-09-01 10:53:18 | 2:07:26 | 1:52:33 | 0:14:53 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T10:21:11.187028+0000 mon.a (mon.0) 19546 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec, osd.15 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884528 |
|
2024-09-01 08:15:59 | 2024-09-01 08:53:54 | 2024-09-01 17:07:14 | 8:13:20 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7884529 |
|
2024-09-01 08:16:00 | 2024-09-01 08:56:25 | 2024-09-01 17:06:13 | 8:09:48 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7884530 |
|
2024-09-01 08:16:02 | 2024-09-01 08:57:26 | 2024-09-01 11:12:15 | 2:14:49 | 2:02:52 | 0:11:57 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T09:24:55.494769+0000 mon.a (mon.0) 1185 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 31 sec, osd.4 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884531 |
|
2024-09-01 08:16:03 | 2024-09-01 08:59:57 | 2024-09-01 11:00:12 | 2:00:15 | 1:51:36 | 0:08:39 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T09:36:04.418260+0000 mon.a (mon.0) 5747 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 30 sec, osd.14 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| pass | 7884532 |
|
2024-09-01 08:16:04 | 2024-09-01 09:00:57 | 2024-09-01 11:14:30 | 2:13:33 | 1:59:11 | 0:14:22 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| pass | 7884533 |
|
2024-09-01 08:16:06 | 2024-09-01 09:04:39 | 2024-09-01 11:23:41 | 2:19:02 | 1:59:10 | 0:19:52 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| pass | 7884534 |
|
2024-09-01 08:16:07 | 2024-09-01 09:13:31 | 2024-09-01 11:22:50 | 2:09:19 | 1:57:08 | 0:12:11 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884535 |
|
2024-09-01 08:16:08 | 2024-09-01 09:14:31 | 2024-09-01 11:21:13 | 2:06:42 | 1:56:02 | 0:10:40 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T10:55:54.989390+0000 mon.a (mon.0) 20666 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 92 sec, osd.6 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884536 |
|
2024-09-01 08:16:10 | 2024-09-01 09:18:12 | 2024-09-01 11:39:50 | 2:21:38 | 2:00:26 | 0:21:12 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T10:11:56.343813+0000 mon.a (mon.0) 6485 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 34 sec, osd.4 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884537 |
|
2024-09-01 08:16:11 | 2024-09-01 09:32:46 | 2024-09-01 18:05:39 | 8:32:53 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7884538 |
|
2024-09-01 08:16:13 | 2024-09-01 09:56:50 | 2024-09-01 12:35:56 | 2:39:06 | 2:02:37 | 0:36:29 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T11:08:07.204493+0000 mon.a (mon.0) 7253 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 34 sec, osd.0 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884539 |
|
2024-09-01 08:16:14 | 2024-09-01 10:26:46 | 2024-09-01 12:51:30 | 2:24:44 | 2:09:41 | 0:15:03 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T11:08:03.171713+0000 mon.a (mon.0) 3865 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 55 sec, osd.8 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| pass | 7884540 |
|
2024-09-01 08:16:15 | 2024-09-01 10:31:38 | 2024-09-01 12:39:04 | 2:07:26 | 2:01:26 | 0:06:00 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| pass | 7884541 |
|
2024-09-01 08:16:17 | 2024-09-01 10:31:48 | 2024-09-01 12:52:24 | 2:20:36 | 2:10:31 | 0:10:05 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| pass | 7884542 |
|
2024-09-01 08:16:18 | 2024-09-01 10:32:09 | 2024-09-01 12:56:42 | 2:24:33 | 2:12:33 | 0:12:00 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| dead | 7884543 |
|
2024-09-01 08:16:19 | 2024-09-01 10:34:20 | 2024-09-01 10:55:56 | 0:21:36 | 0:07:30 | 0:14:06 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
['ubuntu@smithi179.front.sepia.ceph.com: Permission denied (publickey).'] |
||||||||||||||
| fail | 7884544 |
|
2024-09-01 08:16:21 | 2024-09-01 10:37:51 | 2024-09-01 12:46:06 | 2:08:15 | 2:00:52 | 0:07:23 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T10:58:52.601022+0000 mon.a (mon.0) 2095 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 31 sec, osd.13 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884545 |
|
2024-09-01 08:16:22 | 2024-09-01 10:38:41 | 2024-09-01 13:01:31 | 2:22:50 | 2:10:55 | 0:11:55 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T11:23:27.126275+0000 mon.a (mon.0) 5372 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 35 sec, osd.5 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884546 |
|
2024-09-01 08:16:23 | 2024-09-01 10:39:32 | 2024-09-01 13:02:23 | 2:22:51 | 2:13:59 | 0:08:52 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T11:47:50.810622+0000 mon.a (mon.0) 11052 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 75 sec, osd.11 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884547 |
|
2024-09-01 08:16:25 | 2024-09-01 10:41:03 | 2024-09-01 12:58:56 | 2:17:53 | 2:05:49 | 0:12:04 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T11:26:34.208883+0000 mon.a (mon.0) 5256 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 30 sec, osd.0 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884548 |
|
2024-09-01 08:16:26 | 2024-09-01 10:42:54 | 2024-09-01 12:58:20 | 2:15:26 | 2:01:55 | 0:13:31 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T11:23:51.459907+0000 mon.a (mon.0) 3996 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 37 sec, osd.6 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| pass | 7884549 |
|
2024-09-01 08:16:28 | 2024-09-01 10:46:26 | 2024-09-01 13:03:25 | 2:16:59 | 2:03:20 | 0:13:39 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884550 |
|
2024-09-01 08:16:29 | 2024-09-01 10:53:27 | 2024-09-01 12:55:17 | 2:01:50 | 1:53:10 | 0:08:40 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T12:01:59.467179+0000 mon.a (mon.0) 13410 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 32 sec, osd.9 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884551 |
|
2024-09-01 08:16:30 | 2024-09-01 10:56:08 | 2024-09-01 11:05:57 | 0:09:49 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
SSH connection to smithi179 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' |
||||||||||||||
| fail | 7884552 |
|
2024-09-01 08:16:32 | 2024-09-01 10:57:49 | 2024-09-01 12:57:44 | 1:59:55 | 1:50:49 | 0:09:06 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T11:33:21.368402+0000 mon.a (mon.0) 4860 : cluster [WRN] Health check failed: 1 slow ops, oldest one blocked for 33 sec, osd.13 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884553 |
|
2024-09-01 08:16:33 | 2024-09-01 11:00:30 | 2024-09-01 13:21:34 | 2:21:04 | 2:09:12 | 0:11:52 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T12:32:45.530773+0000 mon.a (mon.0) 8595 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 34 sec, osd.13 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884554 |
|
2024-09-01 08:16:35 | 2024-09-01 11:02:01 | 2024-09-01 11:24:20 | 0:22:19 | 0:07:29 | 0:14:50 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
['ubuntu@smithi179.front.sepia.ceph.com: Permission denied (publickey).'] |
||||||||||||||
| pass | 7884555 |
|
2024-09-01 08:16:36 | 2024-09-01 11:06:12 | 2024-09-01 13:24:19 | 2:18:07 | 2:04:20 | 0:13:47 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884556 |
|
2024-09-01 08:16:37 | 2024-09-01 11:12:34 | 2024-09-01 13:32:41 | 2:20:07 | 2:08:46 | 0:11:21 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T11:45:05.334628+0000 mon.a (mon.0) 2475 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 46 sec, osd.0 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| pass | 7884557 |
|
2024-09-01 08:16:39 | 2024-09-01 11:14:45 | 2024-09-01 13:31:19 | 2:16:34 | 2:00:09 | 0:16:25 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884558 |
|
2024-09-01 08:16:40 | 2024-09-01 11:21:37 | 2024-09-01 13:31:26 | 2:09:49 | 1:58:27 | 0:11:22 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T13:11:56.061332+0000 mon.a (mon.0) 15084 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 30 sec, osd.13 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884559 |
|
2024-09-01 08:16:41 | 2024-09-01 11:23:07 | 2024-09-01 13:36:05 | 2:12:58 | 2:01:11 | 0:11:47 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T11:46:28.597313+0000 mon.a (mon.0) 596 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 30 sec, osd.14 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884560 |
|
2024-09-01 08:16:43 | 2024-09-01 11:23:58 | 2024-09-01 11:32:16 | 0:08:18 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
SSH connection to smithi179 was lost: 'sudo yum install -y kernel' |
||||||||||||||
| dead | 7884561 |
|
2024-09-01 08:16:44 | 2024-09-01 11:24:39 | 2024-09-01 11:40:34 | 0:15:55 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
SSH connection to smithi179 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' |
||||||||||||||
| fail | 7884562 |
|
2024-09-01 08:16:45 | 2024-09-01 11:32:31 | 2024-09-01 13:39:23 | 2:06:52 | 1:52:17 | 0:14:35 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T13:17:51.953294+0000 mon.c (mon.2) 13316 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 33 sec, osd.2 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884563 |
|
2024-09-01 08:16:47 | 2024-09-01 11:40:12 | 2024-09-01 11:49:38 | 0:09:26 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
SSH connection to smithi179 was lost: 'sudo yum install -y kernel' |
||||||||||||||
| dead | 7884564 |
|
2024-09-01 08:16:48 | 2024-09-01 11:40:53 | 2024-09-01 11:59:45 | 0:18:52 | 0:01:48 | 0:17:04 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
['ubuntu@smithi179.front.sepia.ceph.com: Permission denied (publickey).'] |
||||||||||||||
| dead | 7884565 |
|
2024-09-01 08:16:50 | 2024-09-01 11:49:55 | 2024-09-01 12:18:16 | 0:28:21 | 0:07:26 | 0:20:55 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
['ubuntu@smithi179.front.sepia.ceph.com: Permission denied (publickey).'] |
||||||||||||||
| pass | 7884566 |
|
2024-09-01 08:16:51 | 2024-09-01 12:00:08 | 2024-09-01 14:30:51 | 2:30:43 | 2:00:47 | 0:29:56 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884567 |
|
2024-09-01 08:16:52 | 2024-09-01 12:18:31 | 2024-09-01 14:32:32 | 2:14:01 | 1:53:31 | 0:20:30 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T13:43:11.045434+0000 mon.a (mon.0) 14897 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 42 sec, osd.7 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884568 |
|
2024-09-01 08:16:54 | 2024-09-01 12:36:10 | 2024-09-01 14:56:26 | 2:20:16 | 2:10:24 | 0:09:52 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
Command failed on smithi062 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' |
||||||||||||||
| dead | 7884569 |
|
2024-09-01 08:16:55 | 2024-09-01 12:39:21 | 2024-09-01 20:54:17 | 8:14:56 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7884570 |
|
2024-09-01 08:16:56 | 2024-09-01 12:44:03 | 2024-09-01 20:56:37 | 8:12:34 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7884571 |
|
2024-09-01 08:16:58 | 2024-09-01 12:46:24 | 2024-09-01 21:01:04 | 8:14:40 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7884572 |
|
2024-09-01 08:16:59 | 2024-09-01 12:51:45 | 2024-09-01 21:03:17 | 8:11:32 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7884573 |
|
2024-09-01 08:17:01 | 2024-09-01 12:52:46 | 2024-09-01 21:05:08 | 8:12:22 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7884574 |
|
2024-09-01 08:17:02 | 2024-09-01 12:55:37 | 2024-09-01 15:04:42 | 2:09:05 | 2:01:24 | 0:07:41 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T13:35:25.670365+0000 osd.6 (osd.6) 74 : cluster [ERR] 3.5 scrub : stat mismatch, got 72/71 objects, 18/17 clones, 72/71 dirty, 16/16 omap, 0/0 pinned, 0/0 hit_set_archive, 3/3 whiteouts, 178960322/177074369 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log |
||||||||||||||
| fail | 7884575 |
|
2024-09-01 08:17:03 | 2024-09-01 12:56:58 | 2024-09-01 15:11:40 | 2:14:42 | 2:04:22 | 0:10:20 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T14:25:56.634550+0000 mon.a (mon.0) 15743 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 30 sec, osd.3 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884576 |
|
2024-09-01 08:17:05 | 2024-09-01 12:57:59 | 2024-09-01 15:01:59 | 2:04:00 | 1:55:38 | 0:08:22 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T13:16:31.434392+0000 mon.a (mon.0) 1655 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 30 sec, osd.14 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884577 |
|
2024-09-01 08:17:06 | 2024-09-01 12:58:40 | 2024-09-01 15:07:08 | 2:08:28 | 1:57:47 | 0:10:41 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T13:40:49.500938+0000 mon.a (mon.0) 4768 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 30 sec, osd.5 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884578 |
|
2024-09-01 08:17:07 | 2024-09-01 12:59:10 | 2024-09-01 15:11:54 | 2:12:44 | 2:00:15 | 0:12:29 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T14:32:20.628841+0000 mon.a (mon.0) 16435 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 31 sec, osd.14 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884579 |
|
2024-09-01 08:17:09 | 2024-09-01 13:01:51 | 2024-09-01 21:13:39 | 8:11:48 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7884580 |
|
2024-09-01 08:17:10 | 2024-09-01 13:02:42 | 2024-09-01 15:13:36 | 2:10:54 | 2:02:08 | 0:08:46 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884581 |
|
2024-09-01 08:17:12 | 2024-09-01 13:03:43 | 2024-09-01 15:33:37 | 2:29:54 | 2:02:34 | 0:27:20 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T15:17:27.261169+0000 mon.a (mon.0) 23473 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 30 sec, osd.0 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884582 |
|
2024-09-01 08:17:13 | 2024-09-01 13:21:57 | 2024-09-01 15:33:21 | 2:11:24 | 1:57:12 | 0:14:12 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T14:32:00.980434+0000 mon.a (mon.0) 10892 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 31 sec, osd.5 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| pass | 7884583 |
|
2024-09-01 08:17:14 | 2024-09-01 13:24:37 | 2024-09-01 15:42:43 | 2:18:06 | 2:04:49 | 0:13:17 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884584 |
|
2024-09-01 08:17:16 | 2024-09-01 13:31:39 | 2024-09-01 15:58:11 | 2:26:32 | 2:15:53 | 0:10:39 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T15:01:30.306490+0000 mon.a (mon.0) 16845 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 31 sec, osd.1 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884585 |
|
2024-09-01 08:17:17 | 2024-09-01 13:31:50 | 2024-09-01 15:49:01 | 2:17:11 | 2:05:39 | 0:11:32 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T14:15:44.477887+0000 mon.a (mon.0) 5123 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec, osd.14 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884586 |
|
2024-09-01 08:17:18 | 2024-09-01 13:32:51 | 2024-09-01 15:43:46 | 2:10:55 | 2:00:41 | 0:10:14 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T14:34:15.519649+0000 mon.a (mon.0) 10819 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec, osd.13 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884587 |
|
2024-09-01 08:17:20 | 2024-09-01 13:36:22 | 2024-09-01 15:49:28 | 2:13:06 | 1:59:52 | 0:13:14 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T14:29:32.977966+0000 mon.a (mon.0) 7053 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 38 sec, osd.4 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884588 |
|
2024-09-01 08:17:21 | 2024-09-01 13:39:33 | 2024-09-01 14:55:08 | 1:15:35 | 0:07:43 | 1:07:52 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
['ubuntu@smithi179.front.sepia.ceph.com: Permission denied (publickey).'] |
||||||||||||||
| fail | 7884589 |
|
2024-09-01 08:17:22 | 2024-09-01 14:32:05 | 2024-09-01 16:53:35 | 2:21:30 | 2:09:18 | 0:12:12 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T16:20:11.904073+0000 mon.a (mon.0) 21205 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 31 sec, osd.13 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884590 |
|
2024-09-01 08:17:24 | 2024-09-01 14:32:45 | 2024-09-01 15:10:40 | 0:37:55 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
'connect to smithi179.front.sepia.ceph.com' reached maximum tries (10) after waiting for 145 seconds |
||||||||||||||
| fail | 7884591 |
|
2024-09-01 08:17:25 | 2024-09-01 14:55:30 | 2024-09-01 17:04:33 | 2:09:03 | 1:58:40 | 0:10:23 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T16:13:04.638043+0000 mon.a (mon.0) 13639 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 31 sec, osd.3 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884592 |
|
2024-09-01 08:17:26 | 2024-09-01 14:56:41 | 2024-09-01 23:23:06 | 8:26:25 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7884593 |
|
2024-09-01 08:17:28 | 2024-09-01 15:02:22 | 2024-09-01 17:26:42 | 2:24:20 | 2:14:05 | 0:10:15 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T16:16:13.351926+0000 mon.a (mon.0) 14874 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec, osd.7 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884594 |
|
2024-09-01 08:17:29 | 2024-09-01 15:04:53 | 2024-09-01 17:06:36 | 2:01:43 | 1:52:58 | 0:08:45 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T16:14:05.346552+0000 mon.a (mon.0) 14032 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 33 sec, osd.3 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| pass | 7884595 |
|
2024-09-01 08:17:31 | 2024-09-01 15:07:24 | 2024-09-01 17:15:12 | 2:07:48 | 1:56:08 | 0:11:40 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| pass | 7884596 |
|
2024-09-01 08:17:32 | 2024-09-01 15:10:56 | 2024-09-01 17:11:06 | 2:00:10 | 1:52:47 | 0:07:23 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
| fail | 7884597 |
|
2024-09-01 08:17:33 | 2024-09-01 15:11:56 | 2024-09-01 17:20:18 | 2:08:22 | 1:58:07 | 0:10:15 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T15:38:38.673317+0000 mon.a (mon.0) 1478 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 37 sec, osd.10 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884598 |
|
2024-09-01 08:17:34 | 2024-09-01 15:12:17 | 2024-09-01 17:13:57 | 2:01:40 | 1:53:00 | 0:08:40 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T16:08:02.302549+0000 mon.a (mon.0) 10731 : cluster [WRN] Health check failed: 4 slow ops, oldest one blocked for 34 sec, daemons [osd.13,osd.8] have slow ops. (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884599 |
|
2024-09-01 08:17:36 | 2024-09-01 15:13:58 | 2024-09-01 17:51:58 | 2:38:00 | 2:07:32 | 0:30:28 | smithi | main | ubuntu | 22.04 | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T17:19:50.868367+0000 mon.a (mon.0) 20513 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec, osd.4 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| fail | 7884600 |
|
2024-09-01 08:17:37 | 2024-09-01 15:33:32 | 2024-09-01 17:46:56 | 2:13:24 | 2:07:20 | 0:06:04 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 |
Failure Reason:
"2024-09-01T15:53:50.276376+0000 mon.a (mon.0) 2051 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 33 sec, osd.4 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
| dead | 7884601 |
|
2024-09-01 08:17:38 | 2024-09-01 15:33:52 | 2024-09-01 23:51:33 | 8:17:41 | smithi | main | centos | 9.stream | rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/small-objects-balanced} | 4 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
Pulpito