Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7396018 2023-09-13 09:26:18 2023-09-14 01:51:53 2023-09-14 02:13:55 0:22:02 0:12:50 0:09:12 smithi main ubuntu 20.04 rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} 1
pass 7396019 2023-09-13 09:26:19 2023-09-14 01:52:04 2023-09-14 02:31:10 0:39:06 0:28:38 0:10:28 smithi main ubuntu 22.04 rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/snaps-few-objects} 2
pass 7396020 2023-09-13 09:26:20 2023-09-14 01:52:44 2023-09-14 02:25:41 0:32:57 0:20:57 0:12:00 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} thrashers/pggrow thrashosds-health workloads/small-objects} 2
dead 7396021 2023-09-13 09:26:21 2023-09-14 01:53:05 2023-09-14 14:03:44 12:10:39 smithi main ubuntu 22.04 rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-snaps-few-objects-overwrites} 2
Failure Reason:

hit max job timeout

pass 7396022 2023-09-13 09:26:21 2023-09-14 01:54:15 2023-09-14 02:31:04 0:36:49 0:25:41 0:11:08 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_20.04} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} 2
pass 7396023 2023-09-13 09:26:22 2023-09-14 01:54:45 2023-09-14 02:18:48 0:24:03 0:13:10 0:10:53 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write} 1
dead 7396024 2023-09-13 09:26:23 2023-09-14 01:54:46 2023-09-14 14:05:34 12:10:48 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
Failure Reason:

hit max job timeout

pass 7396025 2023-09-13 09:26:24 2023-09-14 01:54:56 2023-09-14 02:22:41 0:27:45 0:14:03 0:13:42 smithi main ubuntu 20.04 rados/standalone/{supported-random-distro$/{ubuntu_20.04} workloads/c2c} 1
pass 7396026 2023-09-13 09:26:25 2023-09-14 01:57:37 2023-09-14 02:23:43 0:26:06 0:16:02 0:10:04 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed} 2
pass 7396027 2023-09-13 09:26:26 2023-09-14 01:58:07 2023-09-14 02:26:45 0:28:38 0:16:05 0:12:33 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests} 2
pass 7396028 2023-09-13 09:26:26 2023-09-14 02:00:38 2023-09-14 02:21:29 0:20:51 0:09:37 0:11:14 smithi main ubuntu 20.04 rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04}} 1
pass 7396029 2023-09-13 09:26:27 2023-09-14 02:01:49 2023-09-14 03:25:59 1:24:10 1:14:26 0:09:44 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-big} 2
pass 7396030 2023-09-13 09:26:28 2023-09-14 02:02:09 2023-09-14 02:28:32 0:26:23 0:13:19 0:13:04 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04} thrashers/careful thrashosds-health workloads/cache-agent-small} 2
pass 7396031 2023-09-13 09:26:29 2023-09-14 02:04:00 2023-09-14 02:28:07 0:24:07 0:12:52 0:11:15 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} 1
dead 7396032 2023-09-13 09:26:30 2023-09-14 02:04:20 2023-09-14 14:14:37 12:10:17 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} 2
Failure Reason:

hit max job timeout

pass 7396033 2023-09-13 09:26:31 2023-09-14 02:04:51 2023-09-14 02:35:54 0:31:03 0:17:17 0:13:46 smithi main ubuntu 20.04 rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_20.04}} 2
dead 7396034 2023-09-13 09:26:31 2023-09-14 02:07:31 2023-09-14 14:18:02 12:10:31 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} 2
Failure Reason:

hit max job timeout

dead 7396035 2023-09-13 09:26:32 2023-09-14 02:08:12 2023-09-14 14:19:57 12:11:45 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_20.04} thrashers/none thrashosds-health workloads/cache-snaps} 2
Failure Reason:

hit max job timeout

dead 7396036 2023-09-13 09:26:33 2023-09-14 02:09:12 2023-09-14 14:20:13 12:11:01 smithi main ubuntu 22.04 rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} 3
Failure Reason:

hit max job timeout

pass 7396037 2023-09-13 09:26:34 2023-09-14 02:10:23 2023-09-14 02:32:41 0:22:18 0:11:47 0:10:31 smithi main ubuntu 20.04 rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} 1
pass 7396038 2023-09-13 09:26:35 2023-09-14 02:10:33 2023-09-14 02:34:25 0:23:52 0:15:01 0:08:51 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache} 2
pass 7396039 2023-09-13 09:26:35 2023-09-14 02:10:54 2023-09-14 02:44:28 0:33:34 0:23:50 0:09:44 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} 1
pass 7396040 2023-09-13 09:26:36 2023-09-14 02:10:54 2023-09-14 02:36:54 0:26:00 0:16:40 0:09:20 smithi main ubuntu 20.04 rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_20.04}} 1
pass 7396041 2023-09-13 09:26:37 2023-09-14 02:10:54 2023-09-14 02:53:25 0:42:31 0:31:18 0:11:13 smithi main ubuntu 20.04 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} tasks/rados_workunit_loadgen_big} 2
fail 7396042 2023-09-13 09:26:38 2023-09-14 02:11:15 2023-09-14 02:47:37 0:36:22 0:22:29 0:13:53 smithi main ubuntu 20.04 rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
Failure Reason:

Command failed on smithi018 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

pass 7396043 2023-09-13 09:26:39 2023-09-14 02:12:05 2023-09-14 03:06:31 0:54:26 0:44:40 0:09:46 smithi main ubuntu 22.04 rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-radosbench} 2
pass 7396044 2023-09-13 09:26:39 2023-09-14 02:12:46 2023-09-14 02:34:30 0:21:44 0:11:49 0:09:55 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} 1
pass 7396045 2023-09-13 09:26:40 2023-09-14 02:12:56 2023-09-14 02:39:42 0:26:46 0:15:39 0:11:07 smithi main ubuntu 20.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-bitmap} supported-random-distro$/{ubuntu_20.04} tasks/failover} 2
pass 7396046 2023-09-13 09:26:41 2023-09-14 02:12:56 2023-09-14 02:59:37 0:46:41 0:34:48 0:11:53 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_20.04} thrashers/none thrashosds-health workloads/radosbench-high-concurrency} 2
pass 7396047 2023-09-13 09:26:42 2023-09-14 02:13:37 2023-09-14 02:47:25 0:33:48 0:24:14 0:09:34 smithi main ubuntu 22.04 rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush} 1
pass 7396048 2023-09-13 09:26:43 2023-09-14 02:13:37 2023-09-14 02:33:30 0:19:53 0:10:21 0:09:32 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7396049 2023-09-13 09:26:44 2023-09-14 02:13:57 2023-09-14 02:41:38 0:27:41 0:13:12 0:14:29 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/redirect_promote_tests} 2
pass 7396050 2023-09-13 09:26:44 2023-09-14 02:15:58 2023-09-14 02:43:12 0:27:14 0:16:31 0:10:43 smithi main ubuntu 20.04 rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04}} 1
fail 7396051 2023-09-13 09:26:45 2023-09-14 02:17:28 2023-09-14 02:39:07 0:21:39 0:08:24 0:13:15 smithi main ubuntu 20.04 rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/off mon_election/connectivity task/test_nfs} 1
Failure Reason:

Command failed on smithi029 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3a7a56e0a663f340e447b51386e01bec5538febf pull'

pass 7396052 2023-09-13 09:26:46 2023-09-14 02:17:29 2023-09-14 02:43:13 0:25:44 0:13:42 0:12:02 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} thrashers/mapgap thrashosds-health workloads/redirect_set_object} 2
pass 7396053 2023-09-13 09:26:47 2023-09-14 02:18:49 2023-09-14 02:45:09 0:26:20 0:15:12 0:11:08 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_fio} 1
pass 7396054 2023-09-13 09:26:48 2023-09-14 02:19:40 2023-09-14 02:39:49 0:20:09 0:10:09 0:10:00 smithi main ubuntu 20.04 rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} 1
fail 7396055 2023-09-13 09:26:48 2023-09-14 02:19:40 2023-09-14 02:51:39 0:31:59 0:22:21 0:09:38 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized} 2
Failure Reason:

"2023-09-14T02:41:18.991764+0000 osd.3 (osd.3) 36 : cluster [ERR] osd.3 found snap mapper error on pg 3.13 oid 3:cf575b50:::smithi09440388-449:5e snaps in mapper: {4b, 4c, 50, 55, 58, 5a, 5b, 5c, 5e}, oi: {58, 5a, 5b, 5c, 5e} ...repaired" in cluster log

pass 7396056 2023-09-13 09:26:49 2023-09-14 02:19:50 2023-09-14 02:52:40 0:32:50 0:21:31 0:11:19 smithi main ubuntu 20.04 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_20.04} tasks/rados_workunit_loadgen_mix} 2
dead 7396057 2023-09-13 09:26:50 2023-09-14 02:20:41 2023-09-14 14:32:50 12:12:09 smithi main ubuntu 22.04 rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} 3
Failure Reason:

hit max job timeout

pass 7396058 2023-09-13 09:26:51 2023-09-14 02:22:42 2023-09-14 02:41:51 0:19:09 0:08:56 0:10:13 smithi main ubuntu 20.04 rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} 1
pass 7396059 2023-09-13 09:26:52 2023-09-14 02:23:52 2023-09-14 03:02:21 0:38:29 0:26:22 0:12:07 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_20.04} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} 2
fail 7396060 2023-09-13 09:26:53 2023-09-14 02:25:43 2023-09-14 02:55:02 0:29:19 0:18:30 0:10:49 smithi main ubuntu 20.04 rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
Failure Reason:

Command failed on smithi070 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

pass 7396061 2023-09-13 09:26:53 2023-09-14 02:26:53 2023-09-14 02:52:10 0:25:17 0:13:05 0:12:12 smithi main ubuntu 22.04 rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_5925} 2
pass 7396062 2023-09-13 09:26:54 2023-09-14 02:28:14 2023-09-14 02:55:07 0:26:53 0:16:00 0:10:53 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests} 2
pass 7396063 2023-09-13 09:26:55 2023-09-14 02:28:34 2023-09-14 04:06:32 1:37:58 1:27:28 0:10:30 smithi main ubuntu 22.04 rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code} 1
pass 7396064 2023-09-13 09:26:56 2023-09-14 02:28:35 2023-09-14 02:52:52 0:24:17 0:13:09 0:11:08 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/sample_radosbench} 1
pass 7396065 2023-09-13 09:26:57 2023-09-14 02:29:25 2023-09-14 02:53:59 0:24:34 0:13:25 0:11:09 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/cache-agent-small} 2
dead 7396066 2023-09-13 09:26:57 2023-09-14 14:41:26 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_20.04} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} 2
Failure Reason:

hit max job timeout

dead 7396067 2023-09-13 09:26:58 2023-09-14 02:31:06 2023-09-14 14:42:25 12:11:19 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} 2
Failure Reason:

hit max job timeout

pass 7396068 2023-09-13 09:26:59 2023-09-14 02:31:16 2023-09-14 02:51:08 0:19:52 0:10:42 0:09:10 smithi main ubuntu 20.04 rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} 1
pass 7396069 2023-09-13 09:27:00 2023-09-14 02:31:17 2023-09-14 03:17:13 0:45:56 0:33:23 0:12:33 smithi main ubuntu 22.04 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread} 2
pass 7396070 2023-09-13 09:27:01 2023-09-14 02:33:37 2023-09-14 02:54:15 0:20:38 0:09:53 0:10:45 smithi main ubuntu 20.04 rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} 1
pass 7396071 2023-09-13 09:27:02 2023-09-14 02:34:08 2023-09-14 02:54:42 0:20:34 0:08:47 0:11:47 smithi main ubuntu 20.04 rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} tasks/mon_clock_with_skews} 2
dead 7396072 2023-09-13 09:27:02 2023-09-14 02:34:28 2023-09-14 14:43:59 12:09:31 smithi main ubuntu 20.04 rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_20.04} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} 3
Failure Reason:

hit max job timeout

pass 7396073 2023-09-13 09:27:03 2023-09-14 02:34:28 2023-09-14 02:58:17 0:23:49 0:15:09 0:08:40 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} 1
fail 7396074 2023-09-13 09:27:04 2023-09-14 02:34:39 2023-09-14 03:56:19 1:21:40 1:08:22 0:13:18 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/radosbench} 2
Failure Reason:

"2023-09-14T03:10:29.171772+0000 osd.6 (osd.6) 54 : cluster [WRN] osd.6 PgScrubber: 7.7 timeout on reserving replicas (since 2023-09-14T03:10:24:169754+0000)" in cluster log

fail 7396075 2023-09-13 09:27:05 2023-09-14 02:35:59 2023-09-14 06:10:33 3:34:34 3:15:34 0:19:00 smithi main ubuntu 22.04 rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_api_tests} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on smithi140 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3a7a56e0a663f340e447b51386e01bec5538febf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

pass 7396076 2023-09-13 09:27:06 2023-09-14 02:36:50 2023-09-14 03:01:07 0:24:17 0:13:40 0:10:37 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} 2
pass 7396077 2023-09-13 09:27:07 2023-09-14 02:37:00 2023-09-14 04:32:44 1:55:44 1:42:29 0:13:15 smithi main ubuntu 20.04 rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_20.04}} 2
fail 7396078 2023-09-13 09:27:07 2023-09-14 02:39:51 2023-09-14 03:14:32 0:34:41 0:23:06 0:11:35 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-balanced} 2
Failure Reason:

"2023-09-14T03:02:11.234056+0000 osd.0 (osd.0) 13 : cluster [ERR] osd.0 found snap mapper error on pg 3.20 oid 3:06d07653:::smithi02940359-483:27 snaps in mapper: {1c, 1f, 20, 24, 25, 26}, oi: {24, 25, 26} ...repaired" in cluster log

pass 7396079 2023-09-13 09:27:08 2023-09-14 02:39:51 2023-09-14 03:01:35 0:21:44 0:10:22 0:11:22 smithi main ubuntu 20.04 rados/cephadm/workunits/{0-distro/ubuntu_20.04 agent/on mon_election/classic task/test_adoption} 1
pass 7396080 2023-09-13 09:27:09 2023-09-14 02:41:42 2023-09-14 03:36:43 0:55:01 0:44:12 0:10:49 smithi main ubuntu 22.04 rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} 2
fail 7396081 2023-09-13 09:27:10 2023-09-14 02:41:52 2023-09-14 03:02:58 0:21:06 0:06:01 0:15:05 smithi main ubuntu 20.04 rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
Failure Reason:

Command failed on smithi022 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3a7a56e0a663f340e447b51386e01bec5538febf pull'

pass 7396082 2023-09-13 09:27:11 2023-09-14 02:42:43 2023-09-14 03:08:16 0:25:33 0:15:00 0:10:33 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} 1
pass 7396083 2023-09-13 09:27:11 2023-09-14 02:42:43 2023-09-14 03:15:57 0:33:14 0:20:43 0:12:31 smithi main ubuntu 20.04 rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04}} 2
pass 7396084 2023-09-13 09:27:12 2023-09-14 02:43:24 2023-09-14 03:14:27 0:31:03 0:19:29 0:11:34 smithi main ubuntu 20.04 rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_20.04}} 2
dead 7396085 2023-09-13 09:27:13 2023-09-14 02:44:14 2023-09-14 14:53:28 12:09:14 smithi main ubuntu 22.04 rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 2
Failure Reason:

hit max job timeout

pass 7396086 2023-09-13 09:27:14 2023-09-14 02:44:35 2023-09-14 03:17:27 0:32:52 0:18:59 0:13:53 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed} 2
fail 7396087 2023-09-13 09:27:15 2023-09-14 02:47:35 2023-09-14 03:06:01 0:18:26 0:06:05 0:12:21 smithi main ubuntu 20.04 rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
Failure Reason:

Command failed on smithi018 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3a7a56e0a663f340e447b51386e01bec5538febf pull'

pass 7396088 2023-09-13 09:27:16 2023-09-14 02:47:46 2023-09-14 03:19:14 0:31:28 0:19:17 0:12:11 smithi main ubuntu 20.04 rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_20.04}} 1
dead 7396089 2023-09-13 09:27:17 2023-09-14 02:47:46 2023-09-14 14:59:46 12:12:00 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps} 2
Failure Reason:

hit max job timeout

pass 7396090 2023-09-13 09:27:17 2023-09-14 02:50:27 2023-09-14 03:10:00 0:19:33 0:11:00 0:08:33 smithi main ubuntu 22.04 rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}} 1
fail 7396091 2023-09-13 09:27:18 2023-09-14 02:50:27 2023-09-14 03:28:15 0:37:48 0:24:49 0:12:59 smithi main ubuntu 22.04 rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
Failure Reason:

Command failed on smithi067 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

pass 7396092 2023-09-13 09:27:19 2023-09-14 02:53:19 2023-09-14 03:17:03 0:23:44 0:14:04 0:09:40 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read} 1
pass 7396093 2023-09-13 09:27:20 2023-09-14 02:53:19 2023-09-14 03:19:35 0:26:16 0:15:27 0:10:49 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_20.04} thrashers/default thrashosds-health workloads/cache} 2
pass 7396094 2023-09-13 09:27:21 2023-09-14 02:53:29 2023-09-14 03:43:12 0:49:43 0:39:07 0:10:36 smithi main ubuntu 22.04 rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} 1
pass 7396095 2023-09-13 09:27:22 2023-09-14 02:53:30 2023-09-14 03:15:11 0:21:41 0:11:28 0:10:13 smithi main ubuntu 22.04 rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} 1
pass 7396096 2023-09-13 09:27:22 2023-09-14 02:54:00 2023-09-14 03:18:22 0:24:22 0:13:15 0:11:07 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_20.04} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps} 2
fail 7396097 2023-09-13 09:27:23 2023-09-14 02:54:21 2023-09-14 06:23:07 3:28:46 3:16:08 0:12:38 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/rados_api_tests} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on smithi083 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3a7a56e0a663f340e447b51386e01bec5538febf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

pass 7396098 2023-09-13 09:27:24 2023-09-14 02:54:51 2023-09-14 03:21:08 0:26:17 0:14:36 0:11:41 smithi main ubuntu 22.04 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} 2
pass 7396099 2023-09-13 09:27:25 2023-09-14 02:55:11 2023-09-14 04:24:51 1:29:40 1:20:15 0:09:25 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench} 2
dead 7396100 2023-09-13 09:27:26 2023-09-14 02:55:12 2023-09-14 15:04:10 12:08:58 smithi main ubuntu 22.04 rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 2
Failure Reason:

hit max job timeout

pass 7396101 2023-09-13 09:27:26 2023-09-14 02:55:12 2023-09-14 03:22:06 0:26:54 0:13:44 0:13:10 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} 1
pass 7396102 2023-09-13 09:27:27 2023-09-14 02:58:23 2023-09-14 03:21:39 0:23:16 0:12:06 0:11:10 smithi main ubuntu 20.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_20.04} thrashers/none thrashosds-health workloads/redirect_set_object} 2
dead 7396103 2023-09-13 09:27:28 2023-09-14 02:59:44 2023-09-14 15:10:54 12:11:10 smithi main ubuntu 22.04 rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} 3
Failure Reason:

hit max job timeout

dead 7396104 2023-09-13 09:27:29 2023-09-14 03:01:44 2023-09-14 15:13:19 12:11:35 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} 1
Failure Reason:

hit max job timeout