yuriw-2024-06-27_14:07:21-rados-wip-yuri12-testing-2024-06-26-0904-distro-default-smithi

User	Scheduled	Started	Updated	Runtime	Suite	Branch	Machine Type	Revision	Pass	Fail	Dead
yuriw	2024-06-27 14:07:21	2024-06-27 18:02:32	2024-06-28 01:38:11	7:35:39	rados	wip-yuri12-testing-2024-06-26-0904	smithi	f51acc1	274	47	3

Average wait time: 0:10:50

See other runs of suite 'rados' on branch 'wip-yuri12-testing-2024-06-26-0904'?

See other runs on branch 'wip-yuri12-testing-2024-06-26-0904'?

See other runs scheduled on 2024-06-27?

See detail view?

Status	Job ID	Posted	Started	Updated	Runtime	Duration	In Waiting	Machine	Teuthology Branch	OS Type	OS Version	Description	Nodes
pass	7775257	2024-06-27 14:08:51		2024-06-27 18:32:29		614		smithi	main	centos	9.stream	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/scrub_test}	2
pass	7775258	2024-06-27 14:08:52	2024-06-27 18:13:21	2024-06-27 18:40:02	0:26:41	0:15:46	0:10:55	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache}	4
fail	7775259	2024-06-27 14:08:54	2024-06-27 18:13:22	2024-06-27 18:29:33	0:16:11	0:04:30	0:11:41	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add}	2
Failure Reason: Command failed on smithi018 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775260	2024-06-27 14:08:55	2024-06-27 18:14:12	2024-06-27 19:08:21	0:54:09	0:43:34	0:10:35	smithi	main	ubuntu	22.04	rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_mon_osdmap_prune}	2
pass	7775261	2024-06-27 14:08:56	2024-06-27 16:51:36	2024-06-27 17:13:00	0:21:24	0:12:08	0:09:16	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775262	2024-06-27 14:08:57	2024-06-27 18:14:13	2024-06-27 18:38:28	0:24:15	0:13:52	0:10:23	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_read}	1
pass	7775263	2024-06-27 14:08:58	2024-06-27 18:14:13	2024-06-27 18:59:10	0:44:57	0:32:35	0:12:22	smithi	main	centos	9.stream	rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/snaps-few-objects}	3
pass	7775264	2024-06-27 14:08:59	2024-06-27 18:16:04	2024-06-27 18:37:30	0:21:26	0:12:16	0:09:10	smithi	main	ubuntu	22.04	rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775265	2024-06-27 14:09:01	2024-06-27 18:16:04	2024-06-27 18:54:17	0:38:13	0:28:01	0:10:12	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}	3
pass	7775266	2024-06-27 14:09:02	2024-06-27 18:16:04	2024-06-27 18:47:51	0:31:47	0:16:01	0:15:46	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_set_mon_crush_locations}	3
pass	7775267	2024-06-27 14:09:03	2024-06-27 18:16:15	2024-06-27 18:39:40	0:23:25	0:13:09	0:10:16	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed}	4
pass	7775268	2024-06-27 14:09:04	2024-06-27 18:16:35	2024-06-27 18:46:40	0:30:05	0:20:16	0:09:49	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775269	2024-06-27 14:09:05	2024-06-27 18:17:06	2024-06-27 18:38:52	0:21:46	0:13:05	0:08:41	smithi	main	ubuntu	22.04	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/{1-install 2-ceph 3-mgrmodules 4-units/prometheus}}	2
pass	7775270	2024-06-27 14:09:06	2024-06-27 18:17:26	2024-06-27 19:54:54	1:37:28	1:24:43	0:12:45	smithi	main	ubuntu	22.04	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-radosbench}	4
fail	7775271	2024-06-27 14:09:07	2024-06-27 18:17:46	2024-06-27 18:33:05	0:15:19	0:04:29	0:10:50	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag}	2
Failure Reason: Command failed on smithi086 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775272	2024-06-27 14:09:09	2024-06-27 18:18:07	2024-06-27 18:40:26	0:22:19	0:11:53	0:10:26	smithi	main	ubuntu	22.04	rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775273	2024-06-27 14:09:10	2024-06-27 18:18:37	2024-06-27 18:35:28	0:16:51	0:07:38	0:09:13	smithi	main	centos	9.stream	rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_latest}}	1
pass	7775274	2024-06-27 14:09:11	2024-06-27 18:18:38	2024-06-27 18:46:56	0:28:18	0:15:42	0:12:36	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/dedup-io-snaps}	4
pass	7775275	2024-06-27 14:09:12	2024-06-27 18:20:18	2024-06-27 18:51:02	0:30:44	0:20:09	0:10:35	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	2
pass	7775276	2024-06-27 14:09:13	2024-06-27 18:21:09	2024-06-27 18:43:19	0:22:10	0:11:57	0:10:13	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_ca_signed_key}	2
pass	7775277	2024-06-27 14:09:14	2024-06-27 18:22:39	2024-06-27 19:00:01	0:37:22	0:25:40	0:11:42	smithi	main	centos	9.stream	rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}	4
pass	7775278	2024-06-27 14:09:15	2024-06-27 18:24:50	2024-06-27 19:09:12	0:44:22	0:25:07	0:19:15	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}	4
pass	7775279	2024-06-27 14:09:16	2024-06-27 18:34:34	2024-06-27 19:05:16	0:30:42	0:19:55	0:10:47	smithi	main	ubuntu	22.04	rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775280	2024-06-27 14:09:17	2024-06-27 18:34:35	2024-06-27 19:15:30	0:40:55	0:30:19	0:10:36	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects}	4
pass	7775281	2024-06-27 14:09:19	2024-06-27 18:35:35	2024-06-27 19:02:34	0:26:59	0:15:40	0:11:19	smithi	main	ubuntu	22.04	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/libcephsqlite}	2
fail	7775282	2024-06-27 14:09:20	2024-06-27 18:36:26	2024-06-27 20:33:53	1:57:27	1:47:33	0:09:54	smithi	main	centos	9.stream	rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind}	2
Failure Reason: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun
pass	7775283	2024-06-27 14:09:21	2024-06-27 18:36:36	2024-06-27 19:12:34	0:35:58	0:25:53	0:10:05	smithi	main	ubuntu	22.04	rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}	2
pass	7775284	2024-06-27 14:09:22	2024-06-27 18:36:47	2024-06-27 19:01:01	0:24:14	0:13:22	0:10:52	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_rw}	1
pass	7775285	2024-06-27 14:09:23	2024-06-27 18:37:27	2024-06-27 19:07:07	0:29:40	0:20:20	0:09:20	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775286	2024-06-27 14:09:24	2024-06-27 18:37:27	2024-06-27 19:02:42	0:25:15	0:14:06	0:11:09	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}	4
fail	7775287	2024-06-27 14:09:25	2024-06-27 18:38:58	2024-06-27 21:15:34	2:36:36	2:26:59	0:09:37	smithi	main	ubuntu	22.04	rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub}	1
Failure Reason: Command failed (workunit test scrub/osd-scrub-test.sh) on smithi143 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh'
pass	7775288	2024-06-27 14:09:26	2024-06-27 18:39:48	2024-06-27 19:20:12	0:40:24	0:29:20	0:11:04	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/rados_api_tests}	4
pass	7775289	2024-06-27 14:09:27	2024-06-27 18:40:09	2024-06-27 19:29:37	0:49:28	0:41:09	0:08:19	smithi	main	ubuntu	22.04	rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775290	2024-06-27 14:09:28	2024-06-27 18:40:09	2024-06-27 19:09:35	0:29:26	0:19:24	0:10:02	smithi	main	ubuntu	22.04	rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm}	1
pass	7775291	2024-06-27 14:09:29	2024-06-27 18:40:10	2024-06-27 19:36:06	0:55:56	0:44:45	0:11:11	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}	4
pass	7775292	2024-06-27 14:09:30	2024-06-27 18:02:32	2024-06-27 18:22:16	0:19:44	0:10:39	0:09:05	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775293	2024-06-27 14:09:31	2024-06-27 18:42:30	2024-06-27 19:00:30	0:18:00	0:08:21	0:09:39	smithi	main	centos	9.stream	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} tasks/{1-install 2-ceph 3-mgrmodules 4-units/workunits}}	2
pass	7775294	2024-06-27 14:09:32	2024-06-27 18:43:21	2024-06-27 19:25:07	0:41:46	0:26:26	0:15:20	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency}	4
pass	7775295	2024-06-27 14:09:33	2024-06-27 18:46:42	2024-06-27 19:28:48	0:42:06	0:31:25	0:10:41	smithi	main	ubuntu	22.04	rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_workunits}	2
pass	7775297	2024-06-27 14:09:34	2024-06-27 18:47:03	2024-06-27 19:19:49	0:32:46	0:20:12	0:12:34	smithi	main	ubuntu	22.04	rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait}	2
pass	7775299	2024-06-27 14:09:35	2024-06-27 18:47:53	2024-06-27 19:05:03	0:17:10	0:07:23	0:09:47	smithi	main	centos	9.stream	rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}	1
pass	7775300	2024-06-27 14:09:37	2024-06-27 18:47:54	2024-06-27 19:08:44	0:20:50	0:07:48	0:13:02	smithi	main	centos	9.stream	rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews}	3
pass	7775303	2024-06-27 14:09:38	2024-06-27 18:59:16	2024-06-27 19:35:12	0:35:56	0:25:36	0:10:20	smithi	main	centos	9.stream	rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}	3
pass	7775305	2024-06-27 14:09:39	2024-06-27 19:00:06	2024-06-27 19:19:04	0:18:58	0:10:32	0:08:26	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_latest}}	1
pass	7775307	2024-06-27 14:09:40	2024-06-27 19:00:37	2024-06-27 20:49:34	1:48:57	1:37:00	0:11:57	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/radosbench}	4
pass	7775309	2024-06-27 14:09:41	2024-06-27 19:01:48	2024-06-27 19:16:32	0:14:44	0:06:12	0:08:32	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos}	1
pass	7775311	2024-06-27 14:09:42	2024-06-27 19:02:49	2024-06-27 19:34:34	0:31:45	0:20:40	0:11:05	smithi	main	centos	9.stream	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-balanced}	4
pass	7775313	2024-06-27 14:09:43	2024-06-27 19:03:40	2024-06-27 19:46:12	0:42:32	0:32:34	0:09:58	smithi	main	ubuntu	22.04	rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775315	2024-06-27 14:09:44	2024-06-27 19:04:41	2024-06-27 19:28:30	0:23:49	0:14:01	0:09:48	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write}	1
pass	7775316	2024-06-27 14:09:45		2024-06-27 19:41:34		1566		smithi	main	centos	9.stream	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/rados_api_tests}	2
pass	7775318	2024-06-27 14:09:46	2024-06-27 19:07:32	2024-06-27 19:37:49	0:30:17	0:20:09	0:10:08	smithi	main	centos	9.stream	rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api}	3
fail	7775320	2024-06-27 14:09:47	2024-06-27 19:08:53	2024-06-27 19:24:15	0:15:22	0:04:40	0:10:42	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate}	2
Failure Reason: Command failed on smithi060 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775322	2024-06-27 14:09:48	2024-06-27 19:09:34	2024-06-27 19:35:39	0:26:05	0:13:08	0:12:57	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect}	4
pass	7775324	2024-06-27 14:09:49	2024-06-27 19:12:35	2024-06-27 19:34:08	0:21:33	0:11:57	0:09:36	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	1
fail	7775326	2024-06-27 14:09:51	2024-06-27 19:13:16	2024-06-27 19:51:45	0:38:29	0:28:29	0:10:00	smithi	main	ubuntu	22.04	rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}	1
Failure Reason: Command failed (workunit test cephtool/test.sh) on smithi113 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
pass	7775328	2024-06-27 14:09:52	2024-06-27 19:13:57	2024-06-27 19:50:43	0:36:46	0:25:33	0:11:13	smithi	main	centos	9.stream	rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}	4
pass	7775330	2024-06-27 14:09:53	2024-06-27 19:15:38	2024-06-27 19:59:25	0:43:47	0:33:22	0:10:25	smithi	main	ubuntu	22.04	rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775333	2024-06-27 14:09:55	2024-06-27 19:16:09	2024-06-27 19:54:33	0:38:24	0:21:58	0:16:26	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush}	4
pass	7775335	2024-06-27 14:09:56	2024-06-27 19:21:35	2024-06-27 19:47:45	0:26:10	0:12:56	0:13:14	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests}	4
fail	7775337	2024-06-27 14:09:57	2024-06-27 19:24:17	2024-06-27 20:07:05	0:42:48	0:32:02	0:10:46	smithi	main	centos	9.stream	rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/dashboard}	2
Failure Reason: Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)
fail	7775339	2024-06-27 14:09:58	2024-06-27 19:25:17	2024-06-27 19:52:53	0:27:36	0:18:07	0:09:29	smithi	main	ubuntu	22.04	rados/encoder/{0-start 1-tasks supported-random-distro$/{ubuntu_latest}}	1
Failure Reason: Command failed (workunit test dencoder/test-dencoder.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/dencoder/test-dencoder.sh'
pass	7775341	2024-06-27 14:09:59	2024-06-27 19:30:26	2024-06-27 19:49:45	0:19:19	0:09:05	0:10:14	smithi	main	centos	9.stream	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} tasks/{1-install 2-ceph 3-mgrmodules 4-units/cli}}	2
pass	7775343	2024-06-27 14:10:00	2024-06-27 19:33:18	2024-06-27 20:21:41	0:48:23	0:38:46	0:09:37	smithi	main	ubuntu	22.04	rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}}	1
pass	7775345	2024-06-27 14:10:01	2024-06-27 19:34:09	2024-06-27 19:57:45	0:23:36	0:12:55	0:10:41	smithi	main	ubuntu	22.04	rados/rest/{mgr-restful supported-random-distro$/{ubuntu_latest}}	1
pass	7775347	2024-06-27 14:10:02	2024-06-27 19:34:39	2024-06-27 19:58:02	0:23:23	0:14:25	0:08:58	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775352	2024-06-27 14:10:06	2024-06-27 19:35:41	2024-06-27 20:06:28	0:30:47	0:20:25	0:10:22	smithi	main	centos	9.stream	rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest}	1
pass	7775354	2024-06-27 14:10:07	2024-06-27 19:35:42	2024-06-27 19:57:28	0:21:46	0:12:09	0:09:37	smithi	main	centos	9.stream	rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados tasks/mon_recovery validater/lockdep}	2
fail	7775356	2024-06-27 14:10:08	2024-06-27 19:36:23	2024-06-27 19:52:12	0:15:49	0:04:36	0:11:13	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/deploy-raw}	2
Failure Reason: Command failed on smithi044 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775358	2024-06-27 14:10:09	2024-06-27 19:36:23	2024-06-27 19:59:32	0:23:09	0:11:32	0:11:37	smithi	main	centos	9.stream	rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}	4
pass	7775360	2024-06-27 14:10:11	2024-06-27 19:44:13	2024-06-27 20:03:20	0:19:07	0:10:32	0:08:35	smithi	main	ubuntu	22.04	rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775362	2024-06-27 14:10:12	2024-06-27 19:47:49	2024-06-27 20:18:15	0:30:26	0:17:06	0:13:20	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object}	4
pass	7775363	2024-06-27 14:10:13	2024-06-27 19:50:49	2024-06-27 20:12:17	0:21:28	0:11:53	0:09:35	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	1
fail	7775365	2024-06-27 14:10:14	2024-06-27 19:51:20	2024-06-27 20:07:28	0:16:08	0:04:29	0:11:39	smithi	main	centos	9.stream	rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start}	2
Failure Reason: Command failed on smithi044 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775367	2024-06-27 14:10:15	2024-06-27 19:51:51	2024-06-27 20:15:29	0:23:38	0:14:02	0:09:36	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read}	1
pass	7775369	2024-06-27 14:10:16	2024-06-27 19:52:55	2024-06-27 20:38:57	0:46:02	0:36:03	0:09:59	smithi	main	ubuntu	22.04	rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects}	2
pass	7775371	2024-06-27 14:10:17	2024-06-27 19:54:36	2024-06-27 20:15:34	0:20:58	0:11:31	0:09:27	smithi	main	centos	9.stream	rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}}	1
pass	7775373	2024-06-27 14:10:18	2024-06-27 19:54:57	2024-06-27 20:26:52	0:31:55	0:19:51	0:12:04	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read}	4
pass	7775374	2024-06-27 14:10:19	2024-06-27 19:56:07	2024-06-27 20:28:12	0:32:05	0:21:51	0:10:14	smithi	main	ubuntu	22.04	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_cls_all}	2
pass	7775376	2024-06-27 14:10:20	2024-06-27 19:57:38	2024-06-27 20:25:09	0:27:31	0:19:02	0:08:29	smithi	main	ubuntu	22.04	rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final}	1
pass	7775378	2024-06-27 14:10:21	2024-06-27 19:58:09	2024-06-27 20:29:02	0:30:53	0:21:37	0:09:16	smithi	main	centos	9.stream	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read}	4
pass	7775380	2024-06-27 14:10:22	2024-06-27 20:01:25	2024-06-27 20:20:57	0:19:32	0:10:07	0:09:25	smithi	main	ubuntu	22.04	rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews}	3
pass	7775382	2024-06-27 14:10:23	2024-06-27 20:01:26	2024-06-27 20:23:46	0:22:20	0:10:41	0:11:39	smithi	main	centos	9.stream	rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}	3
pass	7775384	2024-06-27 14:10:24	2024-06-27 20:04:57	2024-06-27 20:26:16	0:21:19	0:11:54	0:09:25	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775386	2024-06-27 14:10:25	2024-06-27 20:06:08	2024-06-27 20:41:32	0:35:24	0:23:44	0:11:40	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites}	4
pass	7775389	2024-06-27 14:10:28	2024-06-27 20:07:19	2024-06-27 20:32:55	0:25:36	0:15:07	0:10:29	smithi	main	ubuntu	22.04	rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775391	2024-06-27 14:10:29	2024-06-27 20:09:30	2024-06-27 20:43:40	0:34:10	0:23:02	0:11:08	smithi	main	ubuntu	22.04	rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain}	3
pass	7775393	2024-06-27 14:10:30	2024-06-27 20:13:13	2024-06-27 20:32:12	0:18:59	0:09:45	0:09:14	smithi	main	centos	9.stream	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-snappy} tasks/{1-install 2-ceph 3-mgrmodules 4-units/crash}}	2
pass	7775395	2024-06-27 14:10:31	2024-06-27 20:13:14	2024-06-27 20:31:59	0:18:45	0:09:08	0:09:37	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_latest}}	1
dead	7775397	2024-06-27 14:10:32	2024-06-27 20:13:14	2024-06-27 20:19:35	0:06:21			smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized}	4
Failure Reason: Error reimaging machines: Expected smithi082's OS to be ubuntu 22.04 but found centos 9.stream
fail	7775399	2024-06-27 14:10:33	2024-06-27 20:13:14	2024-06-27 20:27:59	0:14:45	0:04:31	0:10:14	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all}	2
Failure Reason: Command failed on smithi082 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775401	2024-06-27 14:10:34	2024-06-27 20:13:15	2024-06-27 20:36:24	0:23:09	0:13:12	0:09:57	smithi	main	ubuntu	22.04	rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775403	2024-06-27 14:10:35	2024-06-27 20:13:15	2024-06-27 20:54:51	0:41:36	0:29:07	0:12:29	smithi	main	centos	9.stream	rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}	4
pass	7775405	2024-06-27 14:10:36	2024-06-27 20:13:15	2024-06-27 20:49:17	0:36:02	0:23:39	0:12:23	smithi	main	ubuntu	22.04	rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/crush}	1
pass	7775407	2024-06-27 14:10:37	2024-06-27 20:13:16	2024-06-27 20:49:10	0:35:54	0:22:03	0:13:51	smithi	main	centos	9.stream	rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}	4
dead	7775409	2024-06-27 14:10:38	2024-06-27 20:13:16	2024-06-27 20:13:17	0:00:01			smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read}	1
pass	7775412	2024-06-27 14:10:40	2024-06-27 20:13:17	2024-06-27 20:32:33	0:19:16	0:07:35	0:11:41	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_latest}}	1
fail	7775414	2024-06-27 14:10:41	2024-06-27 20:13:17	2024-06-27 20:23:09	0:09:52			smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects}	4
Failure Reason: Command failed on smithi069 with status 1: 'sudo fuser -v /var/lib/dpkg/lock-frontend'
fail	7775416	2024-06-27 14:10:42	2024-06-27 20:13:17	2024-06-27 20:54:26	0:41:09	0:29:53	0:11:16	smithi	main	centos	9.stream	rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps}	3
Failure Reason: "2024-06-27T20:41:20.685039+0000 osd.6 (osd.6) 12 : cluster [ERR] 2.2 scrub : stat mismatch, got 128/129 objects, 5/5 clones, 128/129 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 1/1 whiteouts, 359971498/363198475 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log
fail	7775418	2024-06-27 14:10:43	2024-06-27 20:13:18	2024-06-27 21:11:18	0:58:00	0:45:33	0:12:27	smithi	main	centos	9.stream	rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}	2
Failure Reason: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun
pass	7775420	2024-06-27 14:10:45	2024-06-27 20:13:18	2024-06-27 20:40:54	0:27:36	0:17:16	0:10:20	smithi	main	ubuntu	22.04	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/rados_python}	2
pass	7775422	2024-06-27 14:10:46	2024-06-27 20:13:18	2024-06-27 20:35:48	0:22:30	0:12:44	0:09:46	smithi	main	ubuntu	22.04	rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775424	2024-06-27 14:10:47	2024-06-27 20:13:19	2024-06-27 20:37:05	0:23:46	0:11:29	0:12:17	smithi	main	centos	9.stream	rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}	4
pass	7775426	2024-06-27 14:10:48	2024-06-27 20:13:19	2024-06-27 20:45:18	0:31:59	0:20:14	0:11:45	smithi	main	ubuntu	22.04	rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add}	2
pass	7775428	2024-06-27 14:10:49	2024-06-27 20:13:20	2024-06-27 20:54:45	0:41:25	0:29:58	0:11:27	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced}	4
pass	7775430	2024-06-27 14:10:50	2024-06-27 20:13:20	2024-06-27 20:34:10	0:20:50	0:10:20	0:10:30	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775432	2024-06-27 14:10:51	2024-06-27 20:13:20	2024-06-27 20:40:48	0:27:28	0:16:18	0:11:10	smithi	main	ubuntu	22.04	rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775435	2024-06-27 14:10:52	2024-06-27 20:13:21	2024-06-27 20:34:27	0:21:06	0:09:47	0:11:19	smithi	main	ubuntu	22.04	rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}}	1
pass	7775436	2024-06-27 14:10:53	2024-06-27 20:13:21	2024-06-27 20:45:31	0:32:10	0:18:48	0:13:22	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic}	3
pass	7775438	2024-06-27 14:10:54	2024-06-27 20:15:22	2024-06-27 20:43:17	0:27:55	0:17:05	0:10:50	smithi	main	centos	9.stream	rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete}	2
pass	7775440	2024-06-27 14:10:55	2024-06-27 20:15:42	2024-06-27 20:37:58	0:22:16	0:11:26	0:10:50	smithi	main	centos	9.stream	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/devicehealth}}	2
pass	7775442	2024-06-27 14:10:56	2024-06-27 20:17:03	2024-06-27 20:49:34	0:32:31	0:21:02	0:11:29	smithi	main	centos	9.stream	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-many-deletes}	4
pass	7775444	2024-06-27 14:10:58	2024-06-27 20:18:23	2024-06-27 20:59:14	0:40:51	0:26:47	0:14:04	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized}	4
pass	7775446	2024-06-27 14:10:59	2024-06-27 20:21:44	2024-06-27 20:44:46	0:23:02	0:11:00	0:12:02	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
fail	7775448	2024-06-27 14:11:00	2024-06-27 20:22:55	2024-06-27 20:37:43	0:14:48	0:04:37	0:10:11	smithi	main	centos	9.stream	rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}	2
Failure Reason: Command failed on smithi184 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775449	2024-06-27 14:11:01	2024-06-27 20:22:55	2024-06-27 20:56:22	0:33:27	0:22:37	0:10:50	smithi	main	centos	9.stream	rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/mon_recovery}	3
pass	7775451	2024-06-27 14:11:02	2024-06-27 20:22:55	2024-06-27 21:03:01	0:40:06	0:28:13	0:11:53	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}	3
pass	7775453	2024-06-27 14:11:03	2024-06-27 20:25:16	2024-06-27 20:48:44	0:23:28	0:13:29	0:09:59	smithi	main	centos	9.stream	rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}}	1
pass	7775455	2024-06-27 14:11:04	2024-06-27 20:25:47	2024-06-27 20:49:21	0:23:34	0:13:59	0:09:35	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read}	1
pass	7775457	2024-06-27 14:11:05	2024-06-27 20:25:47	2024-06-27 21:09:33	0:43:46	0:30:45	0:13:01	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects}	4
pass	7775459	2024-06-27 14:11:06	2024-06-27 20:26:58	2024-06-27 21:00:25	0:33:27	0:20:58	0:12:29	smithi	main	ubuntu	22.04	rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite}	3
pass	7775460	2024-06-27 14:11:07	2024-06-27 20:27:28	2024-06-27 20:44:33	0:17:05	0:07:37	0:09:28	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{centos_latest}}	1
pass	7775462	2024-06-27 14:11:08	2024-06-27 20:27:49	2024-06-27 20:53:13	0:25:24	0:14:56	0:10:28	smithi	main	centos	9.stream	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch}	2
pass	7775464	2024-06-27 14:11:09	2024-06-27 20:28:09	2024-06-27 20:51:19	0:23:10	0:14:30	0:08:40	smithi	main	centos	9.stream	rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}}	1
pass	7775466	2024-06-27 14:11:10	2024-06-27 20:28:09	2024-06-27 21:00:01	0:31:52	0:21:15	0:10:37	smithi	main	centos	9.stream	rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites}	4
pass	7775468	2024-06-27 14:11:11	2024-06-27 20:29:10	2024-06-27 20:57:22	0:28:12	0:15:08	0:13:04	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed}	4
pass	7775470	2024-06-27 14:11:12	2024-06-27 20:32:21	2024-06-27 22:03:15	1:30:54	1:22:52	0:08:02	smithi	main	centos	9.stream	rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code}	1
pass	7775472	2024-06-27 14:11:13	2024-06-27 20:32:21	2024-06-27 21:08:37	0:36:16	0:25:01	0:11:15	smithi	main	centos	9.stream	rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}	4
fail	7775474	2024-06-27 14:11:14	2024-06-27 20:33:02	2024-06-27 20:46:19	0:13:17	0:04:25	0:08:52	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag}	2
Failure Reason: Command failed on smithi094 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775476	2024-06-27 14:11:15	2024-06-27 20:33:22	2024-06-27 20:53:32	0:20:10	0:10:22	0:09:48	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775478	2024-06-27 14:11:16	2024-06-27 20:34:02	2024-06-27 21:05:04	0:31:02	0:19:15	0:11:47	smithi	main	centos	9.stream	rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush}	4
fail	7775480	2024-06-27 14:11:18	2024-06-27 20:35:53	2024-06-27 21:11:06	0:35:13	0:26:48	0:08:25	smithi	main	centos	9.stream	rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}}	1
Failure Reason: Command failed (workunit test cephtool/test.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
pass	7775482	2024-06-27 14:11:18	2024-06-27 20:36:34	2024-06-27 21:11:11	0:34:37	0:22:29	0:12:08	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests}	4
pass	7775484	2024-06-27 14:11:19	2024-06-27 20:37:14	2024-06-27 21:02:54	0:25:40	0:15:59	0:09:41	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations}	3
pass	7775486	2024-06-27 14:11:21	2024-06-27 20:38:05	2024-06-27 21:24:02	0:45:57	0:35:44	0:10:13	smithi	main	ubuntu	22.04	rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775488	2024-06-27 14:11:22	2024-06-27 20:38:05	2024-06-27 21:05:47	0:27:42	0:17:33	0:10:09	smithi	main	centos	9.stream	rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep}	2
pass	7775490	2024-06-27 14:11:23	2024-06-27 20:38:05	2024-06-27 21:04:45	0:26:40	0:15:39	0:11:01	smithi	main	ubuntu	22.04	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-hybrid} tasks/{1-install 2-ceph 3-mgrmodules 4-units/failover}}	2
pass	7775492	2024-06-27 14:11:24	2024-06-27 20:40:21	2024-06-27 21:06:33	0:26:12	0:14:16	0:11:56	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}	4
pass	7775494	2024-06-27 14:11:25	2024-06-27 20:41:42	2024-06-27 21:00:06	0:18:24	0:09:48	0:08:36	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}}	1
pass	7775496	2024-06-27 14:11:26	2024-06-27 20:41:42	2024-06-27 21:05:10	0:23:28	0:14:40	0:08:48	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read}	1
fail	7775498	2024-06-27 14:11:27	2024-06-27 20:41:42	2024-06-27 20:59:00	0:17:18	0:04:27	0:12:51	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait}	2
Failure Reason: Command failed on smithi157 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775500	2024-06-27 14:11:28	2024-06-27 20:43:23	2024-06-27 21:11:46	0:28:23	0:17:20	0:11:03	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-big}	4
pass	7775502	2024-06-27 14:11:29	2024-06-27 20:43:44	2024-06-27 21:05:36	0:21:52	0:11:55	0:09:57	smithi	main	centos	9.stream	rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest}}	1
fail	7775504	2024-06-27 14:11:30	2024-06-27 20:44:34	2024-06-27 21:20:39	0:36:05	0:25:39	0:10:26	smithi	main	centos	9.stream	rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} tasks/e2e}	2
Failure Reason: "2024-06-27T21:05:15.543838+0000 mon.a (mon.0) 298 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
pass	7775506	2024-06-27 14:11:31	2024-06-27 20:45:24	2024-06-27 21:17:07	0:31:43	0:21:02	0:10:41	smithi	main	centos	9.stream	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects}	4
fail	7775508	2024-06-27 14:11:32	2024-06-27 20:45:35	2024-06-27 22:36:08	1:50:33	1:40:33	0:10:00	smithi	main	centos	9.stream	rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench}	3
Failure Reason: "2024-06-27T21:20:00.000105+0000 mon.a (mon.0) 1522 : cluster [WRN] Health detail: HEALTH_WARN Degraded data redundancy: 2161/22251 objects degraded (9.712%), 1 pg degraded, 1 pg undersized" in cluster log
pass	7775510	2024-06-27 14:11:33	2024-06-27 20:46:26	2024-06-27 21:10:21	0:23:55	0:12:36	0:11:19	smithi	main	ubuntu	22.04	rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_5925}	2
pass	7775512	2024-06-27 14:11:34	2024-06-27 20:48:46	2024-06-27 21:24:53	0:36:07	0:25:47	0:10:20	smithi	main	ubuntu	22.04	rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start}	2
pass	7775514	2024-06-27 14:11:35	2024-06-27 20:49:17	2024-06-27 21:12:19	0:23:02	0:11:20	0:11:42	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775516	2024-06-27 14:11:36	2024-06-27 20:49:17	2024-06-27 21:17:50	0:28:33	0:16:38	0:11:55	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-agent-small}	4
pass	7775518	2024-06-27 14:11:37	2024-06-27 20:49:38	2024-06-27 21:10:12	0:20:34	0:07:53	0:12:41	smithi	main	centos	9.stream	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} tasks/rados_striper}	2
pass	7775520	2024-06-27 14:11:38	2024-06-27 20:49:38	2024-06-27 21:08:14	0:18:36	0:08:54	0:09:42	smithi	main	centos	9.stream	rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest}}	1
pass	7775522	2024-06-27 14:11:39	2024-06-27 20:49:38	2024-06-27 21:09:49	0:20:11	0:07:57	0:12:14	smithi	main	centos	9.stream	rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews}	2
pass	7775524	2024-06-27 14:11:41	2024-06-27 20:49:39	2024-06-27 21:29:08	0:39:29	0:27:41	0:11:48	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}	3
pass	7775526	2024-06-27 14:11:42	2024-06-27 20:51:29	2024-06-27 21:14:13	0:22:44	0:12:38	0:10:06	smithi	main	centos	9.stream	rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final}	1
fail	7775528	2024-06-27 14:11:43	2024-06-27 20:54:42	2024-06-27 21:10:16	0:15:34	0:07:05	0:08:29	smithi	main	ubuntu	22.04	rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}}	1
Failure Reason: 502 Server Error: Bad Gateway for url: https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f51acc18cc950586c82c58d666661d4391fc1122
pass	7775530	2024-06-27 14:11:44	2024-06-27 20:54:45	2024-06-27 21:34:22	0:39:37	0:30:05	0:09:32	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy}	4
pass	7775532	2024-06-27 14:11:45	2024-06-27 20:54:46	2024-06-27 21:13:29	0:18:43	0:09:37	0:09:06	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_latest}}	1
fail	7775534	2024-06-27 14:11:46	2024-06-27 20:54:46	2024-06-27 21:12:27	0:17:41	0:06:51	0:10:50	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key}	2
Failure Reason: 502 Server Error: Bad Gateway for url: https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f51acc18cc950586c82c58d666661d4391fc1122
pass	7775536	2024-06-27 14:11:47	2024-06-27 20:54:46	2024-06-27 21:14:56	0:20:10	0:10:37	0:09:33	smithi	main	centos	9.stream	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-snappy} tasks/{1-install 2-ceph 3-mgrmodules 4-units/insights}}	2
pass	7775538	2024-06-27 14:11:48	2024-06-27 20:56:10	2024-06-27 21:22:05	0:25:55	0:16:54	0:09:01	smithi	main	ubuntu	22.04	rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr}	1
pass	7775540	2024-06-27 14:11:49	2024-06-27 20:56:30	2024-06-27 21:24:43	0:28:13	0:18:36	0:09:37	smithi	main	ubuntu	22.04	rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}	2
pass	7775542	2024-06-27 14:11:50	2024-06-27 20:56:30	2024-06-27 21:18:40	0:22:10	0:12:10	0:10:00	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write}	1
pass	7775544	2024-06-27 14:11:51	2024-06-27 20:57:52	2024-06-27 21:29:08	0:31:16	0:21:18	0:09:58	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps}	4
pass	7775546	2024-06-27 14:11:52	2024-06-27 20:57:53	2024-06-27 21:35:00	0:37:07	0:25:38	0:11:29	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}	4
pass	7775548	2024-06-27 14:11:53	2024-06-27 20:59:24	2024-06-27 21:20:54	0:21:30	0:12:12	0:09:18	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775551	2024-06-27 14:11:55	2024-06-27 20:59:24	2024-06-27 21:36:23	0:36:59	0:25:00	0:11:59	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}	4
pass	7775553	2024-06-27 14:11:56	2024-06-27 21:00:04	2024-06-27 21:28:39	0:28:35	0:18:06	0:10:29	smithi	main	ubuntu	22.04	rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775555	2024-06-27 14:11:57	2024-06-27 21:00:05	2024-06-27 21:33:44	0:33:39	0:21:40	0:11:59	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps}	4
fail	7775557	2024-06-27 14:11:59	2024-06-27 21:00:35	2024-06-27 21:33:27	0:32:52	0:21:56	0:10:56	smithi	main	centos	9.stream	rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest}	1
Failure Reason: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun
fail	7775559	2024-06-27 14:12:00	2024-06-27 21:02:56	2024-06-27 21:55:57	0:53:01	0:43:45	0:09:16	smithi	main	centos	9.stream	rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind}	2
Failure Reason: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun
pass	7775561	2024-06-27 14:12:01	2024-06-27 21:02:57	2024-06-27 21:45:58	0:43:01	0:31:00	0:12:01	smithi	main	centos	9.stream	rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}	4
pass	7775563	2024-06-27 14:12:02	2024-06-27 21:05:55	2024-06-27 21:27:05	0:21:10	0:11:20	0:09:50	smithi	main	centos	9.stream	rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}	4
fail	7775565	2024-06-27 14:12:03	2024-06-27 21:05:55	2024-06-27 21:28:55	0:23:00	0:13:43	0:09:17	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm}	1
Failure Reason: Command failed (workunit test cephadm/test_cephadm.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_cephadm.sh'
pass	7775567	2024-06-27 14:12:04	2024-06-27 21:05:55	2024-06-27 21:35:51	0:29:56	0:19:30	0:10:26	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775569	2024-06-27 14:12:06	2024-06-27 21:05:56	2024-06-27 21:49:40	0:43:44	0:32:53	0:10:51	smithi	main	ubuntu	22.04	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_big}	2
pass	7775571	2024-06-27 14:12:07	2024-06-27 21:05:56	2024-06-27 21:58:27	0:52:31	0:42:16	0:10:15	smithi	main	centos	9.stream	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}	4
pass	7775573	2024-06-27 14:12:08	2024-06-27 21:06:37	2024-06-27 21:23:34	0:16:57	0:08:03	0:08:54	smithi	main	centos	9.stream	rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}	1
pass	7775574	2024-06-27 14:12:09	2024-06-27 21:06:37	2024-06-27 21:32:17	0:25:40	0:12:22	0:13:18	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache}	4
pass	7775576	2024-06-27 14:12:10	2024-06-27 21:08:48	2024-06-27 21:38:38	0:29:50	0:19:45	0:10:05	smithi	main	ubuntu	22.04	rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/deploy-raw}	2
pass	7775578	2024-06-27 14:12:11	2024-06-27 21:08:58	2024-06-27 21:39:26	0:30:28	0:20:09	0:10:19	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	2
pass	7775580	2024-06-27 14:12:12	2024-06-27 21:08:59	2024-06-27 21:49:45	0:40:46	0:30:02	0:10:44	smithi	main	ubuntu	22.04	rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}	2
pass	7775582	2024-06-27 14:12:13	2024-06-27 21:08:59	2024-06-27 21:45:58	0:36:59	0:26:14	0:10:45	smithi	main	ubuntu	22.04	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} tasks/{1-install 2-ceph 3-mgrmodules 4-units/module_selftest}}	2
fail	7775584	2024-06-27 14:12:14	2024-06-27 21:10:20	2024-06-27 21:30:43	0:20:23	0:08:30	0:11:53	smithi	main	ubuntu	22.04	rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos}	1
Failure Reason: Command failed (workunit test cephadm/test_repos.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'
pass	7775586	2024-06-27 14:12:16	2024-06-27 21:10:20	2024-06-27 21:34:21	0:24:01	0:13:53	0:10:08	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/dedup-io-mixed}	4
pass	7775588	2024-06-27 14:12:17	2024-06-27 21:11:11	2024-06-27 21:44:50	0:33:39	0:23:07	0:10:32	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write}	1
pass	7775590	2024-06-27 14:12:18	2024-06-27 21:11:21	2024-06-27 21:38:55	0:27:34	0:17:44	0:09:50	smithi	main	centos	9.stream	rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}}	1
pass	7775592	2024-06-27 14:12:19	2024-06-27 21:11:21	2024-06-27 21:31:20	0:19:59	0:10:01	0:09:58	smithi	main	ubuntu	22.04	rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews}	2
pass	7775594	2024-06-27 14:12:20	2024-06-27 21:11:22	2024-06-27 21:33:34	0:22:12	0:11:05	0:11:07	smithi	main	centos	9.stream	rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}	3
pass	7775596	2024-06-27 14:12:21	2024-06-27 21:11:52	2024-06-27 21:42:28	0:30:36	0:20:19	0:10:17	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	1
fail	7775598	2024-06-27 14:12:22	2024-06-27 21:11:52	2024-06-27 21:27:08	0:15:16	0:04:25	0:10:51	smithi	main	centos	9.stream	rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}	2
Failure Reason: Command failed on smithi115 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775600	2024-06-27 14:12:23	2024-06-27 21:11:53	2024-06-27 21:44:41	0:32:48	0:20:24	0:12:24	smithi	main	centos	9.stream	rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls}	3
fail	7775602	2024-06-27 14:12:24	2024-06-27 21:12:33	2024-06-27 22:35:22	1:22:49	1:10:54	0:11:55	smithi	main	ubuntu	22.04	rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc}	1
Failure Reason: Command failed (workunit test misc/ver-health.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh'
pass	7775604	2024-06-27 14:12:25	2024-06-27 21:13:34	2024-06-27 21:38:51	0:25:17	0:14:02	0:11:15	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/dedup-io-snaps}	4
pass	7775606	2024-06-27 14:12:26	2024-06-27 21:15:45	2024-06-27 22:01:16	0:45:31	0:32:57	0:12:34	smithi	main	ubuntu	22.04	rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}	2
fail	7775608	2024-06-27 14:12:28	2024-06-27 21:17:15	2024-06-27 21:32:17	0:15:02	0:04:42	0:10:20	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all}	2
Failure Reason: Command failed on smithi087 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775610	2024-06-27 14:12:29	2024-06-27 21:17:15	2024-06-27 21:57:59	0:40:44	0:29:34	0:11:10	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}	4
pass	7775612	2024-06-27 14:12:30	2024-06-27 21:17:56	2024-06-27 21:38:15	0:20:19	0:10:35	0:09:44	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775614	2024-06-27 14:12:31	2024-06-27 21:18:46	2024-06-27 21:50:01	0:31:15	0:19:08	0:12:07	smithi	main	centos	9.stream	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mix}	2
pass	7775616	2024-06-27 14:12:32	2024-06-27 21:21:59	2024-06-27 22:00:36	0:38:37	0:28:35	0:10:02	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects}	4
pass	7775618	2024-06-27 14:12:33	2024-06-27 21:22:59	2024-06-27 22:11:21	0:48:22	0:37:52	0:10:30	smithi	main	ubuntu	22.04	rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}}	1
pass	7775620	2024-06-27 14:12:34	2024-06-27 21:23:40	2024-06-27 21:47:08	0:23:28	0:12:32	0:10:56	smithi	main	centos	9.stream	rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}}	1
pass	7775622	2024-06-27 14:12:35	2024-06-27 21:24:10	2024-06-27 21:56:56	0:32:46	0:21:09	0:11:37	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush}	4
pass	7775624	2024-06-27 14:12:36	2024-06-27 21:24:51	2024-06-27 21:50:05	0:25:14	0:12:51	0:12:23	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_extra_daemon_features}	2
pass	7775625	2024-06-27 14:12:37	2024-06-27 21:27:11	2024-06-27 22:00:22	0:33:11	0:24:15	0:08:56	smithi	main	centos	9.stream	rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep}	2
pass	7775627	2024-06-27 14:12:38	2024-06-27 21:27:12	2024-06-27 22:08:34	0:41:22	0:30:32	0:10:50	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/rados_api_tests}	4
pass	7775629	2024-06-27 14:12:40	2024-06-27 21:30:13	2024-06-27 21:51:05	0:20:52	0:11:30	0:09:22	smithi	main	centos	9.stream	rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}	4
pass	7775631	2024-06-27 14:12:41	2024-06-27 21:30:14	2024-06-27 21:54:03	0:23:49	0:12:58	0:10:51	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775633	2024-06-27 14:12:42	2024-06-27 21:30:14	2024-06-27 21:53:57	0:23:43	0:14:08	0:09:35	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio}	1
pass	7775635	2024-06-27 14:12:43	2024-06-27 21:30:14	2024-06-27 21:50:31	0:20:17	0:08:14	0:12:03	smithi	main	centos	9.stream	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-comp-zstd} tasks/{1-install 2-ceph 3-mgrmodules 4-units/per_module_finisher_stats}}	2
pass	7775636	2024-06-27 14:12:44	2024-06-27 21:30:45	2024-06-27 22:08:12	0:37:27	0:24:22	0:13:05	smithi	main	centos	9.stream	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}	4
fail	7775638	2024-06-27 14:12:45	2024-06-27 21:32:25	2024-06-27 21:57:39	0:25:14	0:16:02	0:09:12	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain}	3
Failure Reason: "2024-06-27T21:53:39.333676+0000 mon.a (mon.0) 697 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log
pass	7775640	2024-06-27 14:12:46	2024-06-27 21:32:26	2024-06-27 21:55:56	0:23:30	0:15:08	0:08:22	smithi	main	ubuntu	22.04	rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}}	1
fail	7775642	2024-06-27 14:12:47	2024-06-27 21:32:26	2024-06-27 22:09:10	0:36:44	0:26:13	0:10:31	smithi	main	centos	9.stream	rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}}	1
Failure Reason: Command failed (workunit test cephtool/test.sh) on smithi076 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
pass	7775644	2024-06-27 14:12:48	2024-06-27 21:33:37	2024-06-27 22:10:07	0:36:30	0:26:08	0:10:22	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/radosbench-high-concurrency}	4
fail	7775646	2024-06-27 14:12:49	2024-06-27 21:33:47	2024-06-27 21:48:45	0:14:58	0:04:43	0:10:15	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add}	2
Failure Reason: Command failed on smithi032 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775648	2024-06-27 14:12:51	2024-06-27 21:33:47	2024-06-27 21:54:19	0:20:32	0:09:27	0:11:05	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_latest}}	1
pass	7775650	2024-06-27 14:12:52	2024-06-27 21:33:48	2024-06-27 22:06:31	0:32:43	0:20:58	0:11:45	smithi	main	centos	9.stream	rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites}	4
pass	7775652	2024-06-27 14:12:53	2024-06-27 21:34:28	2024-06-27 22:41:12	1:06:44	0:56:38	0:10:06	smithi	main	ubuntu	22.04	rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune}	2
pass	7775654	2024-06-27 14:12:54	2024-06-27 21:34:29	2024-06-27 21:56:49	0:22:20	0:11:13	0:11:07	smithi	main	ubuntu	22.04	rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}	2
pass	7775656	2024-06-27 14:12:55	2024-06-27 21:34:29	2024-06-27 23:01:47	1:27:18	1:15:38	0:11:40	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench}	4
pass	7775658	2024-06-27 14:12:56	2024-06-27 21:35:10	2024-06-27 21:59:39	0:24:29	0:13:14	0:11:15	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}}	1
pass	7775660	2024-06-27 14:12:57	2024-06-27 21:36:00	2024-06-27 22:00:13	0:24:13	0:13:40	0:10:33	smithi	main	centos	9.stream	rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} tasks/mon_recovery}	3
pass	7775662	2024-06-27 14:12:58	2024-06-27 21:36:30	2024-06-27 22:13:41	0:37:11	0:24:50	0:12:21	smithi	main	centos	9.stream	rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}	3
pass	7775664	2024-06-27 14:12:59	2024-06-27 21:38:41	2024-06-27 22:20:34	0:41:53	0:32:35	0:09:18	smithi	main	ubuntu	22.04	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mostlyread}	2
fail	7775666	2024-06-27 14:13:01	2024-06-27 21:39:01	2024-06-27 22:20:09	0:41:08	0:31:27	0:09:41	smithi	main	centos	9.stream	rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zlib} tasks/dashboard}	2
Failure Reason: Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)
pass	7775668	2024-06-27 14:13:02	2024-06-27 21:39:02	2024-06-27 22:00:06	0:21:04	0:11:36	0:09:28	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}}	1
pass	7775670	2024-06-27 14:13:03	2024-06-27 21:39:02	2024-06-27 22:14:23	0:35:21	0:26:23	0:08:58	smithi	main	ubuntu	22.04	rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch}	1
pass	7775673	2024-06-27 14:13:05	2024-06-27 21:39:02	2024-06-27 22:01:21	0:22:19	0:12:02	0:10:17	smithi	main	ubuntu	22.04	rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775675	2024-06-27 14:13:06	2024-06-27 21:39:33	2024-06-27 22:11:58	0:32:25	0:19:55	0:12:30	smithi	main	ubuntu	22.04	rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag}	2
pass	7775677	2024-06-27 14:13:07	2024-06-27 21:42:34	2024-06-27 22:09:06	0:26:32	0:13:35	0:12:57	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/redirect}	4
pass	7775679	2024-06-27 14:13:08	2024-06-27 21:44:54	2024-06-27 22:07:43	0:22:49	0:12:32	0:10:17	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench}	1
pass	7775681	2024-06-27 14:13:09	2024-06-27 21:46:05	2024-06-27 22:17:36	0:31:31	0:23:01	0:08:30	smithi	main	ubuntu	22.04	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-comp-lz4} tasks/{1-install 2-ceph 3-mgrmodules 4-units/progress}}	2
pass	7775683	2024-06-27 14:13:10	2024-06-27 21:46:05	2024-06-27 22:04:45	0:18:40	0:09:48	0:08:52	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{centos_latest}}	1
pass	7775685	2024-06-27 14:13:12	2024-06-27 21:46:06	2024-06-27 22:25:53	0:39:47	0:25:14	0:14:33	smithi	main	centos	9.stream	rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}	4
fail	7775687	2024-06-27 14:13:13	2024-06-27 21:48:47	2024-06-27 22:05:37	0:16:50	0:04:33	0:12:17	smithi	main	centos	9.stream	rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start}	2
Failure Reason: Command failed on smithi039 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775689	2024-06-27 14:13:14	2024-06-27 21:49:47	2024-06-27 22:13:22	0:23:35	0:13:06	0:10:29	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests}	4
pass	7775691	2024-06-27 14:13:15	2024-06-27 21:50:08	2024-06-27 22:11:48	0:21:40	0:10:52	0:10:48	smithi	main	ubuntu	22.04	rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}	1
fail	7775693	2024-06-27 14:13:16	2024-06-27 21:50:08	2024-06-27 22:38:23	0:48:15	0:36:21	0:11:54	smithi	main	centos	9.stream	rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects}	3
Failure Reason: "2024-06-27T22:20:00.000143+0000 mon.a (mon.0) 1433 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 2 osds down; Degraded data redundancy: 84/348 objects degraded (24.138%), 5 pgs degraded, 2 pgs undersized" in cluster log
pass	7775695	2024-06-27 14:13:17	2024-06-27 21:50:38	2024-06-27 22:25:02	0:34:24	0:24:09	0:10:15	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}	4
pass	7775697	2024-06-27 14:13:18	2024-06-27 21:51:09	2024-06-27 22:30:27	0:39:18	0:25:15	0:14:03	smithi	main	centos	9.stream	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1}	4
pass	7775699	2024-06-27 14:13:19	2024-06-27 21:54:20	2024-06-27 22:16:05	0:21:45	0:11:18	0:10:27	smithi	main	centos	9.stream	rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final}	1
pass	7775701	2024-06-27 14:13:20	2024-06-27 21:55:10	2024-06-27 22:16:09	0:20:59	0:11:22	0:09:37	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{centos_latest}}	1
pass	7775703	2024-06-27 14:13:22	2024-06-27 21:55:20	2024-06-27 22:19:20	0:24:00	0:12:42	0:11:18	smithi	main	centos	9.stream	rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}	4
fail	7775705	2024-06-27 14:13:23	2024-06-27 21:56:01	2024-06-27 23:50:05	1:54:04	1:42:47	0:11:17	smithi	main	centos	9.stream	rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind}	2
Failure Reason: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun
pass	7775707	2024-06-27 14:13:24	2024-06-27 21:56:51	2024-06-27 22:25:11	0:28:20	0:18:37	0:09:43	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect_set_object}	4
pass	7775709	2024-06-27 14:13:25	2024-06-27 21:57:02	2024-06-28 00:22:53	2:25:51	2:15:49	0:10:02	smithi	main	centos	9.stream	rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}	2
fail	7775711	2024-06-27 14:13:26	2024-06-27 21:57:42	2024-06-27 22:27:29	0:29:47	0:18:38	0:11:09	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic}	3
Failure Reason: "2024-06-27T22:19:56.010339+0000 mon.a (mon.0) 736 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log
pass	7775713	2024-06-27 14:13:27	2024-06-27 21:58:03	2024-06-28 00:51:51	2:53:48	2:28:20	0:25:28	smithi	main	ubuntu	22.04	rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{ubuntu_latest}}	1
pass	7775715	2024-06-27 14:13:28	2024-06-27 21:58:03	2024-06-27 22:20:19	0:22:16	0:11:18	0:10:58	smithi	main	centos	9.stream	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/readwrite}	2
pass	7775717	2024-06-27 14:13:29	2024-06-27 21:58:34	2024-06-27 22:20:14	0:21:40	0:11:02	0:10:38	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775719	2024-06-27 14:13:30	2024-06-27 21:58:34	2024-06-27 22:27:25	0:28:51	0:16:24	0:12:27	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read}	4
pass	7775721	2024-06-27 14:13:31	2024-06-27 22:00:15	2024-06-27 22:23:49	0:23:34	0:14:40	0:08:54	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read}	1
pass	7775723	2024-06-27 14:13:33	2024-06-27 22:00:15	2024-06-27 22:48:10	0:47:55	0:37:28	0:10:27	smithi	main	ubuntu	22.04	rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}	2
fail	7775725	2024-06-27 14:13:34	2024-06-27 22:00:15	2024-06-27 22:16:09	0:15:54	0:04:25	0:11:29	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait}	2
Failure Reason: Command failed on smithi043 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775727	2024-06-27 14:13:35	2024-06-27 22:00:26	2024-06-27 22:40:52	0:40:26	0:30:18	0:10:08	smithi	main	ubuntu	22.04	rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits}	2
pass	7775729	2024-06-27 14:13:36	2024-06-27 22:00:46	2024-06-27 22:24:50	0:24:04	0:13:10	0:10:54	smithi	main	ubuntu	22.04	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/{1-install 2-ceph 3-mgrmodules 4-units/prometheus}}	2
fail	7775731	2024-06-27 14:13:37	2024-06-27 22:00:47	2024-06-27 22:24:46	0:23:59	0:12:14	0:11:45	smithi	main	centos	9.stream	rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon}	1
Failure Reason: Command failed (workunit test mon/mkfs.sh) on smithi155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mkfs.sh'
pass	7775733	2024-06-27 14:13:38	2024-06-27 22:01:17	2024-06-27 22:34:07	0:32:50	0:19:39	0:13:11	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/small-objects-balanced}	4
pass	7775735	2024-06-27 14:13:39	2024-06-27 22:04:48	2024-06-27 22:26:27	0:21:39	0:09:50	0:11:49	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775737	2024-06-27 14:13:40	2024-06-27 22:05:38	2024-06-27 22:26:11	0:20:33	0:10:25	0:10:08	smithi	main	ubuntu	22.04	rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews}	3
pass	7775739	2024-06-27 14:13:41	2024-06-27 22:06:39	2024-06-27 22:43:00	0:36:21	0:24:39	0:11:42	smithi	main	centos	9.stream	rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}	3
pass	7775741	2024-06-27 14:13:42	2024-06-27 22:07:49	2024-06-27 22:32:03	0:24:14	0:13:32	0:10:42	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_rgw_multisite}	3
pass	7775743	2024-06-27 14:13:44	2024-06-27 22:08:20	2024-06-27 22:39:40	0:31:20	0:20:14	0:11:06	smithi	main	centos	9.stream	rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}}	2
pass	7775745	2024-06-27 14:13:45	2024-06-27 22:08:40	2024-06-27 22:44:00	0:35:20	0:24:17	0:11:03	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites}	4
pass	7775747	2024-06-27 14:13:46	2024-06-27 22:09:11	2024-06-27 22:43:28	0:34:17	0:23:31	0:10:46	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized}	4
fail	7775749	2024-06-27 14:13:47	2024-06-27 22:09:11	2024-06-27 22:22:51	0:13:40	0:04:36	0:09:04	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate}	2
Failure Reason: Command failed on smithi002 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775751	2024-06-27 14:13:48	2024-06-27 22:10:12	2024-06-27 22:29:40	0:19:28	0:08:05	0:11:23	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{centos_latest}}	1
pass	7775753	2024-06-27 14:13:49	2024-06-27 22:10:12	2024-06-27 22:47:46	0:37:34	0:26:28	0:11:06	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}	4
pass	7775755	2024-06-27 14:13:50	2024-06-27 22:12:03	2024-06-27 22:42:23	0:30:20	0:19:17	0:11:03	smithi	main	centos	9.stream	rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest}}	2
pass	7775757	2024-06-27 14:13:51	2024-06-27 22:13:23	2024-06-27 22:47:17	0:33:54	0:22:44	0:11:10	smithi	main	ubuntu	22.04	rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_set_mon_crush_locations}	3
pass	7775759	2024-06-27 14:13:52	2024-06-27 22:13:24	2024-06-27 22:48:06	0:34:42	0:23:26	0:11:16	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects}	4
fail	7775761	2024-06-27 14:13:53	2024-06-27 22:14:24	2024-06-27 22:26:45	0:12:21			smithi	main	centos	9.stream	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}	8
Failure Reason: too many values to unpack (expected 1)
pass	7775763	2024-06-27 14:13:54	2024-06-27 22:19:26	2024-06-27 22:46:59	0:27:33	0:18:37	0:08:56	smithi	main	ubuntu	22.04	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/repair_test}	2
pass	7775765	2024-06-27 14:13:55	2024-06-27 22:19:26	2024-06-27 22:52:57	0:33:31	0:22:04	0:11:27	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush}	4
pass	7775767	2024-06-27 14:13:56	2024-06-27 22:20:27	2024-06-27 22:43:52	0:23:25	0:14:22	0:09:03	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw}	1
pass	7775769	2024-06-27 14:13:57	2024-06-27 22:20:27	2024-06-27 22:39:37	0:19:10	0:09:01	0:10:09	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_latest}}	1
pass	7775771	2024-06-27 14:13:58	2024-06-27 22:20:37	2024-06-27 22:47:29	0:26:52	0:14:27	0:12:25	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}	4
pass	7775773	2024-06-27 14:13:59	2024-06-27 22:23:58	2024-06-27 23:03:04	0:39:06	0:26:38	0:12:28	smithi	main	ubuntu	22.04	rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}	2
fail	7775775	2024-06-27 14:14:00	2024-06-27 22:24:59	2024-06-27 22:55:49	0:30:50	0:20:55	0:09:55	smithi	main	centos	9.stream	rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest}	1
Failure Reason: Command failed on smithi193 with status 32: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-1'
pass	7775777	2024-06-27 14:14:01	2024-06-27 22:24:59	2024-06-27 22:45:46	0:20:47	0:12:00	0:08:47	smithi	main	centos	9.stream	rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/mon_recovery validater/lockdep}	2
pass	7775779	2024-06-27 14:14:02	2024-06-27 22:25:09	2024-06-27 22:44:29	0:19:20	0:10:35	0:08:45	smithi	main	ubuntu	22.04	rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775781	2024-06-27 14:14:03	2024-06-27 22:25:10	2024-06-27 23:05:35	0:40:25	0:29:23	0:11:02	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced}	4
pass	7775783	2024-06-27 14:14:04	2024-06-27 22:25:20	2024-06-27 22:45:55	0:20:35	0:08:27	0:12:08	smithi	main	centos	9.stream	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} tasks/{1-install 2-ceph 3-mgrmodules 4-units/workunits}}	2
pass	7775785	2024-06-27 14:14:06	2024-06-27 22:26:01	2024-06-27 22:54:41	0:28:40	0:17:47	0:10:53	smithi	main	centos	9.stream	rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api}	3
pass	7775787	2024-06-27 14:14:06	2024-06-27 22:26:01	2024-06-27 22:45:55	0:19:54	0:09:55	0:09:59	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
fail	7775789	2024-06-27 14:14:07	2024-06-27 22:26:01	2024-06-27 22:43:56	0:17:55	0:04:35	0:13:20	smithi	main	centos	9.stream	rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw}	2
Failure Reason: Command failed on smithi033 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775791	2024-06-27 14:14:08	2024-06-27 22:26:02	2024-06-27 22:47:13	0:21:11	0:09:51	0:11:20	smithi	main	ubuntu	22.04	rados/singleton/{all/admin-socket mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775793	2024-06-27 14:14:10	2024-06-27 22:26:02	2024-06-27 23:05:06	0:39:04	0:27:25	0:11:39	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized}	4
fail	7775794	2024-06-27 14:14:11	2024-06-27 22:26:03	2024-06-27 23:19:29	0:53:26	0:41:41	0:11:45	smithi	main	ubuntu	22.04	rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd-backfill}	1
Failure Reason: Command failed (workunit test osd-backfill/osd-backfill-space.sh) on smithi172 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh'
fail	7775796	2024-06-27 14:14:12	2024-06-27 22:26:03	2024-06-27 23:04:55	0:38:52	0:28:44	0:10:08	smithi	main	ubuntu	22.04	rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}	1
Failure Reason: Command failed (workunit test cephtool/test.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f51acc18cc950586c82c58d666661d4391fc1122 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
pass	7775798	2024-06-27 14:14:13	2024-06-27 22:26:03	2024-06-27 22:53:22	0:27:19	0:14:48	0:12:31	smithi	main	centos	9.stream	rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete}	2
pass	7775800	2024-06-27 14:14:14	2024-06-27 22:26:04	2024-06-27 22:55:26	0:29:22	0:17:12	0:12:10	smithi	main	ubuntu	22.04	rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_ca_signed_key}	2
pass	7775802	2024-06-27 14:14:15	2024-06-27 22:26:04	2024-06-27 22:46:57	0:20:53	0:08:51	0:12:02	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{centos_latest}}	1
pass	7775804	2024-06-27 14:14:16	2024-06-27 22:26:34	2024-06-27 22:46:37	0:20:03	0:08:58	0:11:05	smithi	main	ubuntu	22.04	rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}}	1
pass	7775806	2024-06-27 14:14:17	2024-06-27 22:27:35	2024-06-27 23:07:57	0:40:22	0:27:33	0:12:49	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects}	4
pass	7775808	2024-06-27 14:14:18	2024-06-27 22:29:45	2024-06-27 22:47:49	0:18:04	0:07:59	0:10:05	smithi	main	centos	9.stream	rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews}	2
pass	7775810	2024-06-27 14:14:19	2024-06-27 22:30:36	2024-06-27 22:56:19	0:25:43	0:14:01	0:11:42	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}	3
pass	7775812	2024-06-27 14:14:20	2024-06-27 22:32:07	2024-06-27 23:02:19	0:30:12	0:18:09	0:12:03	smithi	main	centos	9.stream	rados/singleton/{all/backfill-toofull mon_election/connectivity msgr-failures/none msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest}}	1
pass	7775814	2024-06-27 14:14:21	2024-06-27 22:32:08	2024-06-27 22:55:57	0:23:49	0:13:03	0:10:46	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read}	1
pass	7775816	2024-06-27 14:14:22	2024-06-27 22:32:08	2024-06-27 22:57:27	0:25:19	0:14:09	0:11:10	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm}	1
pass	7775818	2024-06-27 14:14:23	2024-06-27 22:34:09	2024-06-27 22:55:35	0:21:26	0:10:25	0:11:01	smithi	main	centos	9.stream	rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/scrub_test}	2
pass	7775820	2024-06-27 14:14:24	2024-06-27 22:34:09	2024-06-27 22:53:02	0:18:53	0:10:01	0:08:52	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}}	1
pass	7775822	2024-06-27 14:14:25	2024-06-27 22:34:09	2024-06-27 23:01:59	0:27:50	0:16:39	0:11:11	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/write_fadvise_dontneed}	4
pass	7775824	2024-06-27 14:14:26	2024-06-27 22:37:14	2024-06-27 23:09:00	0:31:46	0:18:55	0:12:51	smithi	main	ubuntu	22.04	rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all}	2
pass	7775826	2024-06-27 14:14:27	2024-06-27 22:38:24	2024-06-27 22:59:36	0:21:12	0:10:54	0:10:18	smithi	main	centos	9.stream	rados/singleton/{all/deduptool mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest}}	1
pass	7775828	2024-06-27 14:14:29	2024-06-27 22:38:24	2024-06-27 23:17:58	0:39:34	0:27:53	0:11:41	smithi	main	centos	9.stream	rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}	4
pass	7775830	2024-06-27 14:14:30	2024-06-27 22:40:55	2024-06-28 00:00:27	1:19:32	1:08:09	0:11:23	smithi	main	ubuntu	22.04	rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-radosbench}	4
pass	7775833	2024-06-27 14:14:32	2024-06-27 22:42:26	2024-06-27 23:04:50	0:22:24	0:11:44	0:10:40	smithi	main	ubuntu	22.04	rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/cli}}	2
pass	7775835	2024-06-27 14:14:33	2024-06-27 22:43:06	2024-06-27 23:40:34	0:57:28	0:44:45	0:12:43	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites}	4
pass	7775837	2024-06-27 14:14:34	2024-06-27 22:43:37	2024-06-27 23:18:26	0:34:49	0:24:34	0:10:15	smithi	main	ubuntu	22.04	rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests}	4
pass	7775839	2024-06-27 14:14:35	2024-06-27 22:43:57	2024-06-27 23:01:22	0:17:25	0:08:41	0:08:44	smithi	main	centos	9.stream	rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}}	1
fail	7775841	2024-06-27 14:14:36	2024-06-27 22:43:58	2024-06-27 23:00:35	0:16:37	0:04:29	0:12:08	smithi	main	centos	9.stream	rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start}	2
Failure Reason: Command failed on smithi066 with status 234: 'sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 \| sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn'
pass	7775842	2024-06-27 14:14:37	2024-06-27 22:44:08	2024-06-27 23:18:53	0:34:45	0:23:04	0:11:41	smithi	main	centos	9.stream	rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}	4
pass	7775844	2024-06-27 14:14:38	2024-06-27 22:45:49	2024-06-27 23:09:48	0:23:59	0:12:46	0:11:13	smithi	main	ubuntu	22.04	rados/singleton/{all/divergent_priors mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775846	2024-06-27 14:14:39	2024-06-27 22:45:49	2024-06-27 23:10:14	0:24:25	0:13:57	0:10:28	smithi	main	ubuntu	22.04	rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}	4
fail	7775848	2024-06-27 14:14:40	2024-06-27 22:47:51	2024-06-27 23:43:11	0:55:20	0:45:55	0:09:25	smithi	main	centos	9.stream	rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}	2
Failure Reason: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun
pass	7775850	2024-06-27 14:14:41	2024-06-27 22:47:52	2024-06-27 23:09:31	0:21:39	0:12:38	0:09:01	smithi	main	centos	9.stream	rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream} 1-start 2-services/rgw 3-final}	1
pass	7775852	2024-06-27 14:14:42	2024-06-27 22:47:52	2024-06-27 23:15:55	0:28:03	0:17:46	0:10:17	smithi	main	centos	9.stream	rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-big}	4
pass	7775854	2024-06-27 14:14:43	2024-06-27 22:47:52	2024-06-27 23:07:47	0:19:55	0:10:42	0:09:13	smithi	main	ubuntu	22.04	rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775856	2024-06-27 14:14:44	2024-06-27 22:47:53	2024-06-27 23:11:24	0:23:31	0:13:52	0:09:39	smithi	main	ubuntu	22.04	rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw}	1
pass	7775858	2024-06-27 14:14:45	2024-06-27 22:47:53	2024-06-27 23:09:24	0:21:31	0:12:15	0:09:16	smithi	main	ubuntu	22.04	rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}	1
pass	7775860	2024-06-27 14:14:46	2024-06-27 22:47:53	2024-06-28 01:38:11	2:50:18	2:40:33	0:09:45	smithi	main	ubuntu	22.04	rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd}	1
pass	7775862	2024-06-27 14:14:48	2024-06-27 22:47:54	2024-06-27 23:03:54	0:16:00	0:06:13	0:09:47	smithi	main	centos	9.stream	rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos}	1
dead		2024-06-27 14:08:51		2024-06-27 14:08:51									—