Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7908922 2024-09-17 15:02:40 2024-09-17 15:03:30 2024-09-17 15:18:26 0:14:56 0:05:26 0:09:30 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
pass 7908923 2024-09-17 15:02:41 2024-09-17 15:03:30 2024-09-17 15:45:05 0:41:35 0:25:01 0:16:34 smithi main ubuntu 22.04 rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-balanced} 4
pass 7908924 2024-09-17 15:02:42 2024-09-17 15:03:31 2024-09-17 15:42:33 0:39:02 0:25:18 0:13:44 smithi main ubuntu 22.04 rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} 4
pass 7908925 2024-09-17 15:02:43 2024-09-17 15:03:31 2024-09-17 15:27:14 0:23:43 0:14:09 0:09:34 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} 1
pass 7908926 2024-09-17 15:02:44 2024-09-17 15:03:32 2024-09-17 15:46:14 0:42:42 0:29:59 0:12:43 smithi main ubuntu 22.04 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests} 2
pass 7908927 2024-09-17 15:02:45 2024-09-17 15:03:32 2024-09-17 15:30:45 0:27:13 0:17:57 0:09:16 smithi main centos 9.stream rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} 3
pass 7908928 2024-09-17 15:02:46 2024-09-17 15:03:32 2024-09-17 15:25:16 0:21:44 0:12:29 0:09:15 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7908929 2024-09-17 15:02:48 2024-09-17 15:03:33 2024-09-17 15:48:46 0:45:13 0:33:50 0:11:23 smithi main ubuntu 22.04 rados/singleton/{all/lost-unfound-delete mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7908930 2024-09-17 15:02:49 2024-09-17 15:03:33 2024-09-17 15:22:33 0:19:00 0:08:45 0:10:15 smithi main centos 9.stream rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7908931 2024-09-17 15:02:50 2024-09-17 15:03:34 2024-09-17 16:13:28 1:09:54 0:55:00 0:14:54 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/careful_host thrashosds-health workloads/radosbench-high-concurrency} 4
pass 7908932 2024-09-17 15:02:51 2024-09-17 15:03:34 2024-09-17 15:53:52 0:50:18 0:35:00 0:15:18 smithi main ubuntu 22.04 rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}} 1
pass 7908933 2024-09-17 15:02:52 2024-09-17 15:03:35 2024-09-17 15:28:04 0:24:29 0:13:00 0:11:29 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_extra_daemon_features} 2
pass 7908934 2024-09-17 15:02:54 2024-09-17 15:03:35 2024-09-17 15:55:47 0:52:12 0:37:14 0:14:58 smithi main centos 9.stream rados/singleton/{all/lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest}} 1
fail 7908935 2024-09-17 15:02:55 2024-09-17 15:03:35 2024-09-17 15:52:05 0:48:30 0:32:30 0:16:00 smithi main centos 9.stream rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/dashboard} 2
Failure Reason:

Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)

fail 7908936 2024-09-17 15:02:56 2024-09-17 15:03:36 2024-09-17 15:29:58 0:26:22 0:13:48 0:12:34 smithi main centos 9.stream rados/encoder/{0-start 1-tasks supported-random-distro$/{centos_latest}} 1
Failure Reason:

Command failed (workunit test dencoder/test-dencoder.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/dencoder/test-dencoder.sh'

pass 7908937 2024-09-17 15:02:57 2024-09-17 15:03:36 2024-09-17 15:26:11 0:22:35 0:12:41 0:09:54 smithi main ubuntu 22.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore/bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/cli}} 2
pass 7908938 2024-09-17 15:02:58 2024-09-17 15:03:36 2024-09-17 16:05:07 1:01:31 0:49:48 0:11:43 smithi main ubuntu 22.04 rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}} 1
pass 7908939 2024-09-17 15:02:59 2024-09-17 15:03:37 2024-09-17 15:24:17 0:20:40 0:09:27 0:11:13 smithi main centos 9.stream rados/rest/{mgr-restful supported-random-distro$/{centos_latest}} 1
pass 7908940 2024-09-17 15:03:00 2024-09-17 15:03:37 2024-09-17 15:34:36 0:30:59 0:15:21 0:15:38 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7908941 2024-09-17 15:03:02 2024-09-17 15:03:37 2024-09-17 15:34:45 0:31:08 0:16:21 0:14:47 smithi main ubuntu 22.04 rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c} 1
fail 7908942 2024-09-17 15:03:03 2024-09-17 15:03:38 2024-09-17 15:51:36 0:47:58 0:35:32 0:12:26 smithi main centos 9.stream rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

Command failed (workunit test cls/test_cls_rbd.sh) on smithi102 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

pass 7908943 2024-09-17 15:03:04 2024-09-17 15:03:38 2024-09-17 15:31:05 0:27:27 0:15:28 0:11:59 smithi main centos 9.stream rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest} 1
pass 7908944 2024-09-17 15:03:05 2024-09-17 15:03:39 2024-09-17 15:31:56 0:28:17 0:12:17 0:16:00 smithi main centos 9.stream rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados tasks/mon_recovery validater/lockdep} 2
pass 7908945 2024-09-17 15:03:06 2024-09-17 15:03:39 2024-09-17 16:38:26 1:34:47 1:20:22 0:14:25 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/radosbench} 4
fail 7908946 2024-09-17 15:03:07 2024-09-17 15:03:40 2024-09-17 15:33:00 0:29:20 0:14:39 0:14:41 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

reached maximum tries (120) after waiting for 120 seconds

pass 7908947 2024-09-17 15:03:08 2024-09-17 15:03:40 2024-09-17 15:34:40 0:31:00 0:15:18 0:15:42 smithi main ubuntu 22.04 rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 7908948 2024-09-17 15:03:10 2024-09-17 15:03:40 2024-09-17 15:46:54 0:43:14 0:28:25 0:14:49 smithi main ubuntu 22.04 rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7908949 2024-09-17 15:03:11 2024-09-17 15:03:41 2024-09-17 15:26:58 0:23:17 0:11:05 0:12:12 smithi main ubuntu 22.04 rados/singleton/{all/max-pg-per-osd.from-mon mon_election/connectivity msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7908950 2024-09-17 15:03:12 2024-09-17 15:03:41 2024-09-17 15:25:03 0:21:22 0:09:09 0:12:13 smithi main centos 9.stream rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7908951 2024-09-17 15:03:13 2024-09-17 15:03:42 2024-09-17 15:30:11 0:26:29 0:14:11 0:12:18 smithi main centos 9.stream rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7908952 2024-09-17 15:03:14 2024-09-17 15:03:42 2024-09-17 15:38:22 0:34:40 0:20:25 0:14:15 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/redirect} 4
pass 7908953 2024-09-17 15:03:15 2024-09-17 15:03:42 2024-09-17 15:30:44 0:27:02 0:14:24 0:12:38 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} 1
pass 7908954 2024-09-17 15:03:17 2024-09-17 15:03:43 2024-09-17 15:53:41 0:49:58 0:36:35 0:13:23 smithi main ubuntu 22.04 rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} 2
pass 7908955 2024-09-17 15:03:18 2024-09-17 15:03:43 2024-09-17 15:32:26 0:28:43 0:15:37 0:13:06 smithi main centos 9.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} tasks/rados_cls_all} 2
pass 7908956 2024-09-17 15:03:19 2024-09-17 15:03:43 2024-09-17 15:27:09 0:23:26 0:14:31 0:08:55 smithi main ubuntu 22.04 rados/singleton/{all/max-pg-per-osd.from-primary mon_election/classic msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7908957 2024-09-17 15:03:20 2024-09-17 15:03:44 2024-09-17 15:25:06 0:21:22 0:10:23 0:10:59 smithi main centos 9.stream rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/basic 3-final} 1
pass 7908958 2024-09-17 15:03:21 2024-09-17 15:03:44 2024-09-17 15:45:13 0:41:29 0:27:54 0:13:35 smithi main ubuntu 22.04 rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} 4
pass 7908959 2024-09-17 15:03:22 2024-09-17 15:03:44 2024-09-17 15:39:34 0:35:50 0:20:53 0:14:57 smithi main centos 9.stream rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-small-objects-fast-read} 4
pass 7908960 2024-09-17 15:03:23 2024-09-17 15:03:45 2024-09-17 15:30:50 0:27:05 0:11:22 0:15:43 smithi main ubuntu 22.04 rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} 3
pass 7908961 2024-09-17 15:03:24 2024-09-17 15:03:45 2024-09-17 15:31:28 0:27:43 0:12:52 0:14:51 smithi main centos 9.stream rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} 3
pass 7908962 2024-09-17 15:03:26 2024-09-17 15:03:46 2024-09-17 15:27:53 0:24:07 0:09:16 0:14:51 smithi main centos 9.stream rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_latest}} 1
pass 7908963 2024-09-17 15:03:27 2024-09-17 15:03:46 2024-09-17 15:43:31 0:39:45 0:25:19 0:14:26 smithi main ubuntu 22.04 rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} 4
pass 7908964 2024-09-17 15:03:28 2024-09-17 15:03:46 2024-09-17 15:35:46 0:32:00 0:16:43 0:15:17 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests} 4
pass 7908965 2024-09-17 15:03:29 2024-09-17 15:03:47 2024-09-17 15:44:13 0:40:26 0:25:00 0:15:26 smithi main ubuntu 22.04 rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_host_drain} 3
pass 7908966 2024-09-17 15:03:30 2024-09-17 15:03:47 2024-09-17 15:30:42 0:26:55 0:14:40 0:12:15 smithi main ubuntu 22.04 rados/singleton/{all/max-pg-per-osd.from-replica mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7908967 2024-09-17 15:03:31 2024-09-17 15:03:47 2024-09-17 15:26:27 0:22:40 0:09:35 0:13:05 smithi main centos 9.stream rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-options/write$/{write_v2}} tasks/{1-install 2-ceph 3-mgrmodules 4-units/crash}} 2
pass 7908968 2024-09-17 15:03:32 2024-09-17 15:03:48 2024-09-17 15:25:42 0:21:54 0:08:55 0:12:59 smithi main centos 9.stream rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
dead 7908969 2024-09-17 15:03:34 2024-09-17 15:03:48 2024-09-17 15:10:13 0:06:25 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/redirect_set_object} 4
Failure Reason:

Error reimaging machines: Expected smithi161's OS to be ubuntu 22.04 but found centos 9.stream

dead 7908970 2024-09-17 15:03:35 2024-09-17 15:03:48 2024-09-17 15:03:50 0:00:02 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/repave-all} 2
Failure Reason:

Error reimaging machines: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/193/task

pass 7908971 2024-09-17 15:03:36 2024-09-17 15:28:57 669 smithi main centos 9.stream rados/singleton/{all/mon-auth-caps mon_election/classic msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest}} 1
pass 7908972 2024-09-17 15:03:37 2024-09-17 15:03:49 2024-09-17 15:33:29 0:29:40 0:19:55 0:09:45 smithi main centos 9.stream rados/standalone/{supported-random-distro$/{centos_latest} workloads/crush} 1
pass 7908973 2024-09-17 15:03:39 2024-09-17 15:26:11 765 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read} 1
pass 7908974 2024-09-17 15:03:40 2024-09-17 15:03:50 2024-09-17 15:43:15 0:39:25 0:24:26 0:14:59 smithi main centos 9.stream rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7908975 2024-09-17 15:03:41 2024-09-17 15:03:50 2024-09-17 15:25:15 0:21:25 0:12:05 0:09:20 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7908976 2024-09-17 15:03:42 2024-09-17 15:26:38 451 smithi main centos 9.stream rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_latest}} 1
pass 7908977 2024-09-17 15:03:44 2024-09-17 15:31:29 898 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/set-chunks-read} 4
pass 7908978 2024-09-17 15:03:45 2024-09-17 15:03:51 2024-09-17 15:48:24 0:44:33 0:28:56 0:15:37 smithi main centos 9.stream rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps} 3
pass 7908979 2024-09-17 15:03:46 2024-09-17 15:04:42 2024-09-17 15:39:05 0:34:23 0:13:23 0:21:00 smithi main centos 9.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} tasks/rados_python} 2
pass 7908980 2024-09-17 15:03:48 2024-09-17 15:46:44 628 smithi main centos 9.stream rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 7908981 2024-09-17 15:03:49 2024-09-17 15:25:15 2024-09-17 15:44:26 0:19:11 0:09:46 0:09:25 smithi main centos 9.stream rados/singleton/{all/mon-config-key-caps mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest}} 1
pass 7908982 2024-09-17 15:03:51 2024-09-17 15:57:59 1263 smithi main ubuntu 22.04 rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
pass 7908983 2024-09-17 15:03:52 2024-09-17 15:25:26 2024-09-17 15:42:29 0:17:03 0:07:39 0:09:24 smithi main centos 9.stream rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7908984 2024-09-17 15:03:53 2024-09-17 15:25:37 2024-09-17 15:55:42 0:30:05 0:19:08 0:10:57 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/small-objects-balanced} 4
pass 7908985 2024-09-17 15:03:54 2024-09-17 15:26:27 2024-09-17 16:02:24 0:35:57 0:23:11 0:12:46 smithi main ubuntu 22.04 rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} 4
pass 7908986 2024-09-17 15:03:56 2024-09-17 15:27:18 2024-09-17 15:46:53 0:19:35 0:10:16 0:09:19 smithi main ubuntu 22.04 rados/objectstore/{backends/fusestore supported-random-distro$/{ubuntu_latest}} 1
fail 7908987 2024-09-17 15:03:57 2024-09-17 15:27:28 2024-09-17 15:55:17 0:27:49 0:18:27 0:09:22 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} 3
Failure Reason:

Command failed on smithi001 with status 5: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 92235a74-750b-11ef-bceb-c7b262605968 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v1/status\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts | jq -e \'"\'"\'.data | .[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n\''

pass 7908988 2024-09-17 15:03:58 2024-09-17 15:28:18 2024-09-17 15:51:27 0:23:09 0:13:18 0:09:51 smithi main centos 9.stream rados/singleton/{all/mon-config-keys mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest}} 1
pass 7908989 2024-09-17 15:03:59 2024-09-17 15:28:19 2024-09-17 15:55:24 0:27:05 0:16:06 0:10:59 smithi main centos 9.stream rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/sync-many workloads/pool-create-delete} 2
pass 7908990 2024-09-17 15:04:00 2024-09-17 15:30:19 2024-09-17 15:54:05 0:23:46 0:14:25 0:09:21 smithi main ubuntu 22.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore/bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/devicehealth}} 2
pass 7908991 2024-09-17 15:04:02 2024-09-17 15:30:30 2024-09-17 16:01:56 0:31:26 0:21:08 0:10:18 smithi main centos 9.stream rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-many-deletes} 4
pass 7908992 2024-09-17 15:04:03 2024-09-17 15:31:00 2024-09-17 15:52:24 0:21:24 0:11:54 0:09:30 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7908993 2024-09-17 15:04:04 2024-09-17 15:31:01 2024-09-17 15:54:26 0:23:25 0:14:43 0:08:42 smithi main centos 9.stream rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
pass 7908994 2024-09-17 15:04:05 2024-09-17 15:31:11 2024-09-17 16:07:36 0:36:25 0:24:50 0:11:35 smithi main ubuntu 22.04 rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} 3
pass 7908995 2024-09-17 15:04:07 2024-09-17 15:31:42 2024-09-17 16:09:28 0:37:46 0:27:15 0:10:31 smithi main ubuntu 22.04 rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} 3
pass 7908996 2024-09-17 15:04:08 2024-09-17 15:31:42 2024-09-17 16:04:21 0:32:39 0:20:22 0:12:17 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized} 4
pass 7908997 2024-09-17 15:04:09 2024-09-17 15:32:13 2024-09-17 15:55:33 0:23:20 0:13:52 0:09:28 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read} 1
pass 7908998 2024-09-17 15:04:10 2024-09-17 15:32:13 2024-09-17 15:55:43 0:23:30 0:13:27 0:10:03 smithi main centos 9.stream rados/singleton/{all/mon-config mon_election/connectivity msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest}} 1
dead 7908999 2024-09-17 15:04:12 2024-09-17 15:32:44 2024-09-17 23:43:28 8:10:44 smithi main ubuntu 22.04 rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} 3
Failure Reason:

hit max job timeout

pass 7909000 2024-09-17 15:04:13 2024-09-17 15:33:14 2024-09-17 15:53:10 0:19:56 0:10:35 0:09:21 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909001 2024-09-17 15:04:14 2024-09-17 15:33:45 2024-09-17 16:08:40 0:34:55 0:23:56 0:10:59 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow_host thrashosds-health workloads/small-objects} 4
pass 7909002 2024-09-17 15:04:15 2024-09-17 15:34:55 2024-09-17 15:57:44 0:22:49 0:13:54 0:08:55 smithi main centos 9.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} tasks/rados_stress_watch} 2
pass 7909003 2024-09-17 15:04:17 2024-09-17 15:34:56 2024-09-17 16:07:31 0:32:35 0:21:55 0:10:40 smithi main centos 9.stream rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7909004 2024-09-17 15:04:18 2024-09-17 15:35:56 2024-09-17 16:08:24 0:32:28 0:20:22 0:12:06 smithi main centos 9.stream rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} 4
pass 7909005 2024-09-17 15:04:19 2024-09-17 15:38:37 2024-09-17 16:07:47 0:29:10 0:20:03 0:09:07 smithi main centos 9.stream rados/singleton/{all/mon-stretch-pool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest}} 1
pass 7909006 2024-09-17 15:04:20 2024-09-17 15:38:38 2024-09-17 17:11:52 1:33:14 1:24:12 0:09:02 smithi main centos 9.stream rados/standalone/{supported-random-distro$/{centos_latest} workloads/erasure-code} 1
pass 7909007 2024-09-17 15:04:22 2024-09-17 15:38:38 2024-09-17 16:03:20 0:24:42 0:12:40 0:12:02 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7909008 2024-09-17 15:04:23 2024-09-17 15:39:18 2024-09-17 16:01:14 0:21:56 0:11:13 0:10:43 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909009 2024-09-17 15:04:24 2024-09-17 15:39:49 2024-09-17 16:23:16 0:43:27 0:34:45 0:08:42 smithi main ubuntu 22.04 rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909010 2024-09-17 15:04:25 2024-09-17 15:39:49 2024-09-17 16:22:51 0:43:02 0:29:28 0:13:34 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} 4
pass 7909011 2024-09-17 15:04:26 2024-09-17 15:42:50 2024-09-17 16:12:20 0:29:30 0:20:24 0:09:06 smithi main ubuntu 22.04 rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909012 2024-09-17 15:04:28 2024-09-17 15:42:50 2024-09-17 16:08:50 0:26:00 0:15:23 0:10:37 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_set_mon_crush_locations} 3
pass 7909013 2024-09-17 15:04:29 2024-09-17 15:43:31 2024-09-17 16:10:40 0:27:09 0:16:27 0:10:42 smithi main centos 9.stream rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-lz4} rados tasks/rados_cls_all validater/lockdep} 2
pass 7909014 2024-09-17 15:04:30 2024-09-17 15:43:31 2024-09-17 16:09:21 0:25:50 0:16:13 0:09:37 smithi main ubuntu 22.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore/bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/failover}} 2
pass 7909015 2024-09-17 15:04:31 2024-09-17 15:43:42 2024-09-17 16:05:39 0:21:57 0:10:55 0:11:02 smithi main centos 9.stream rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 7909016 2024-09-17 15:04:32 2024-09-17 15:44:32 2024-09-17 16:16:33 0:32:01 0:21:07 0:10:54 smithi main centos 9.stream rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} 4
pass 7909017 2024-09-17 15:04:34 2024-09-17 15:45:23 2024-09-17 16:07:02 0:21:39 0:12:38 0:09:01 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909018 2024-09-17 15:04:35 2024-09-17 15:45:23 2024-09-17 16:08:48 0:23:25 0:13:42 0:09:43 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read} 1
pass 7909019 2024-09-17 15:04:36 2024-09-17 15:45:23 2024-09-17 16:08:25 0:23:02 0:12:14 0:10:48 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7909020 2024-09-17 15:04:37 2024-09-17 15:45:34 2024-09-17 16:30:43 0:45:09 0:35:19 0:09:50 smithi main ubuntu 22.04 rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909021 2024-09-17 15:04:38 2024-09-17 15:45:34 2024-09-17 16:23:39 0:38:05 0:27:37 0:10:28 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/careful_host thrashosds-health workloads/snaps-few-objects-localized} 4
fail 7909022 2024-09-17 15:04:40 2024-09-17 15:46:15 2024-09-17 16:07:04 0:20:49 0:11:26 0:09:23 smithi main centos 9.stream rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-hybrid} tasks/e2e} 2
Failure Reason:

Command failed on smithi007 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2'

pass 7909023 2024-09-17 15:04:41 2024-09-17 15:46:15 2024-09-17 16:17:24 0:31:09 0:20:43 0:10:26 smithi main centos 9.stream rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} 4
fail 7909024 2024-09-17 15:04:42 2024-09-17 15:46:26 2024-09-17 17:28:13 1:41:47 1:31:14 0:10:33 smithi main centos 9.stream rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} 3
Failure Reason:

"2024-09-17T16:20:00.000196+0000 mon.a (mon.0) 1905 : cluster [WRN] pg 4.b is stuck undersized for 76s, current state active+recovering+undersized+remapped, last acting [4,0]" in cluster log

pass 7909025 2024-09-17 15:04:43 2024-09-17 15:47:06 2024-09-17 16:06:38 0:19:32 0:09:56 0:09:36 smithi main centos 9.stream rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} 2
pass 7909026 2024-09-17 15:04:44 2024-09-17 15:47:16 2024-09-17 16:24:41 0:37:25 0:25:26 0:11:59 smithi main ubuntu 22.04 rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
pass 7909027 2024-09-17 15:04:46 2024-09-17 15:48:47 2024-09-17 16:08:09 0:19:22 0:08:09 0:11:13 smithi main centos 9.stream rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} 1
pass 7909028 2024-09-17 15:04:47 2024-09-17 15:48:47 2024-09-17 16:13:22 0:24:35 0:11:54 0:12:41 smithi main ubuntu 22.04 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper} 2
pass 7909029 2024-09-17 15:04:48 2024-09-17 15:51:48 2024-09-17 16:31:13 0:39:25 0:29:17 0:10:08 smithi main ubuntu 22.04 rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909030 2024-09-17 15:04:49 2024-09-17 15:51:49 2024-09-17 16:31:57 0:40:08 0:26:53 0:13:15 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/snaps-few-objects} 4
pass 7909031 2024-09-17 15:04:51 2024-09-17 15:52:39 2024-09-17 16:13:51 0:21:12 0:10:18 0:10:54 smithi main ubuntu 22.04 rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} 2
pass 7909032 2024-09-17 15:04:52 2024-09-17 15:54:00 2024-09-17 16:27:21 0:33:21 0:23:34 0:09:47 smithi main centos 9.stream rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} 3
pass 7909033 2024-09-17 15:04:53 2024-09-17 15:54:20 2024-09-17 16:15:14 0:20:54 0:11:41 0:09:13 smithi main centos 9.stream rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_9.stream_runc} 1-start 2-services/rgw 3-final} 1
pass 7909034 2024-09-17 15:04:54 2024-09-17 15:54:20 2024-09-17 16:30:52 0:36:32 0:23:44 0:12:48 smithi main centos 9.stream rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7909035 2024-09-17 15:04:55 2024-09-17 15:55:31 2024-09-17 16:16:05 0:20:34 0:11:41 0:08:53 smithi main centos 9.stream rados/objectstore/{backends/keyvaluedb supported-random-distro$/{centos_latest}} 1
pass 7909036 2024-09-17 15:04:57 2024-09-17 15:55:31 2024-09-17 16:12:35 0:17:04 0:07:57 0:09:07 smithi main centos 9.stream rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7909037 2024-09-17 15:04:58 2024-09-17 15:55:42 2024-09-17 16:17:21 0:21:39 0:11:35 0:10:04 smithi main ubuntu 22.04 rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909038 2024-09-17 15:04:59 2024-09-17 15:55:42 2024-09-17 16:16:41 0:20:59 0:10:53 0:10:06 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} 2
pass 7909039 2024-09-17 15:05:00 2024-09-17 15:55:53 2024-09-17 16:19:34 0:23:41 0:14:27 0:09:14 smithi main ubuntu 22.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore/bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/insights}} 2
pass 7909040 2024-09-17 15:05:01 2024-09-17 15:56:03 2024-09-17 16:19:07 0:23:04 0:14:22 0:08:42 smithi main centos 9.stream rados/standalone/{supported-random-distro$/{centos_latest} workloads/mgr} 1
pass 7909041 2024-09-17 15:05:03 2024-09-17 15:56:03 2024-09-17 16:20:34 0:24:31 0:13:35 0:10:56 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/write_fadvise_dontneed} 4
pass 7909042 2024-09-17 15:05:04 2024-09-17 15:56:54 2024-09-17 16:19:28 0:22:34 0:12:34 0:10:00 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_write} 1
pass 7909043 2024-09-17 15:05:05 2024-09-17 15:57:55 2024-09-17 16:19:49 0:21:54 0:12:46 0:09:08 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909044 2024-09-17 15:05:06 2024-09-17 15:58:05 2024-09-17 16:33:41 0:35:36 0:20:55 0:14:41 smithi main ubuntu 22.04 rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7909045 2024-09-17 15:05:08 2024-09-17 16:02:06 2024-09-17 16:27:01 0:24:55 0:15:22 0:09:33 smithi main centos 9.stream rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest}} 2
pass 7909046 2024-09-17 15:05:09 2024-09-17 16:02:16 2024-09-17 16:31:48 0:29:32 0:19:58 0:09:34 smithi main centos 9.stream rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow_host thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} 4
pass 7909047 2024-09-17 15:05:10 2024-09-17 16:02:47 2024-09-17 16:36:39 0:33:52 0:20:01 0:13:51 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests} 4
pass 7909048 2024-09-17 15:05:11 2024-09-17 16:04:38 2024-09-17 16:35:00 0:30:22 0:17:47 0:12:35 smithi main centos 9.stream rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} 1
pass 7909049 2024-09-17 15:05:12 2024-09-17 16:04:38 2024-09-17 16:41:43 0:37:05 0:28:20 0:08:45 smithi main centos 9.stream rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados tasks/mon_recovery validater/valgrind} 2
pass 7909050 2024-09-17 15:05:14 2024-09-17 16:04:38 2024-09-17 16:45:06 0:40:28 0:29:02 0:11:26 smithi main centos 9.stream rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} 4
pass 7909051 2024-09-17 15:05:15 2024-09-17 16:05:59 2024-09-17 16:33:17 0:27:18 0:15:00 0:12:18 smithi main ubuntu 22.04 rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 7909052 2024-09-17 15:05:16 2024-09-17 16:07:20 2024-09-17 16:30:00 0:22:40 0:13:32 0:09:08 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm} 1
pass 7909053 2024-09-17 15:05:17 2024-09-17 16:07:20 2024-09-17 16:29:45 0:22:25 0:14:55 0:07:30 smithi main centos 9.stream rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7909054 2024-09-17 15:05:18 2024-09-17 16:07:21 2024-09-17 16:35:01 0:27:40 0:18:28 0:09:12 smithi main ubuntu 22.04 rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909055 2024-09-17 15:05:20 2024-09-17 16:07:21 2024-09-17 16:46:23 0:39:02 0:28:20 0:10:42 smithi main centos 9.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_big} 2
pass 7909056 2024-09-17 15:05:21 2024-09-17 16:07:21 2024-09-17 17:03:34 0:56:13 0:45:31 0:10:42 smithi main ubuntu 22.04 rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} 4
pass 7909057 2024-09-17 15:05:22 2024-09-17 16:07:42 2024-09-17 17:20:26 1:12:44 1:01:04 0:11:40 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/cache-agent-big} 4
fail 7909058 2024-09-17 15:05:23 2024-09-17 16:08:02 2024-09-17 16:48:19 0:40:17 0:29:40 0:10:37 smithi main ubuntu 22.04 rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

reached maximum tries (120) after waiting for 120 seconds

pass 7909059 2024-09-17 15:05:24 2024-09-17 16:08:43 2024-09-17 16:35:50 0:27:07 0:17:22 0:09:45 smithi main centos 9.stream rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{centos_latest}} 2
pass 7909060 2024-09-17 15:05:25 2024-09-17 16:08:43 2024-09-17 16:48:35 0:39:52 0:29:53 0:09:59 smithi main ubuntu 22.04 rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} 2
pass 7909061 2024-09-17 15:05:27 2024-09-17 16:08:53 2024-09-17 16:30:29 0:21:36 0:10:58 0:10:38 smithi main ubuntu 22.04 rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909062 2024-09-17 15:05:28 2024-09-17 16:08:54 2024-09-17 16:44:34 0:35:40 0:27:06 0:08:34 smithi main ubuntu 22.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-options/write$/{write_v2}} tasks/{1-install 2-ceph 3-mgrmodules 4-units/module_selftest}} 2
fail 7909063 2024-09-17 15:05:29 2024-09-17 16:09:04 2024-09-17 16:29:49 0:20:45 0:09:39 0:11:06 smithi main ubuntu 22.04 rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} 1
Failure Reason:

Command failed (workunit test cephadm/test_repos.sh) on smithi029 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

pass 7909064 2024-09-17 15:05:30 2024-09-17 16:09:05 2024-09-17 16:47:17 0:38:12 0:27:35 0:10:37 smithi main ubuntu 22.04 rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow_host thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7909065 2024-09-17 15:05:31 2024-09-17 16:09:35 2024-09-17 16:42:24 0:32:49 0:22:52 0:09:57 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write} 1
pass 7909066 2024-09-17 15:05:33 2024-09-17 16:09:45 2024-09-17 16:38:42 0:28:57 0:17:17 0:11:40 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-small} 4
pass 7909067 2024-09-17 15:05:34 2024-09-17 16:10:56 2024-09-17 16:29:44 0:18:48 0:07:00 0:11:48 smithi main centos 9.stream rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v1only no_pools objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} 2
pass 7909068 2024-09-17 15:05:35 2024-09-17 16:12:47 2024-09-17 16:34:43 0:21:56 0:10:29 0:11:27 smithi main centos 9.stream rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} 3
pass 7909069 2024-09-17 15:05:36 2024-09-17 16:13:47 2024-09-17 16:40:51 0:27:04 0:17:35 0:09:29 smithi main centos 9.stream rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7909070 2024-09-17 15:05:37 2024-09-17 16:13:47 2024-09-17 16:41:11 0:27:24 0:17:11 0:10:13 smithi main ubuntu 22.04 rados/singleton/{all/radostool mon_election/classic msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909071 2024-09-17 15:05:38 2024-09-17 16:13:48 2024-09-17 16:37:22 0:23:34 0:14:18 0:09:16 smithi main centos 9.stream rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
fail 7909072 2024-09-17 15:05:40 2024-09-17 16:14:08 2024-09-17 16:45:17 0:31:09 0:18:35 0:12:34 smithi main centos 9.stream rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} 3
Failure Reason:

Command failed on smithi193 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror'"

fail 7909073 2024-09-17 15:05:41 2024-09-17 16:16:19 2024-09-17 17:09:46 0:53:27 0:43:11 0:10:16 smithi main centos 9.stream rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} 1
Failure Reason:

Command failed (workunit test misc/test-ceph-helpers.sh) on smithi104 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh'

pass 7909074 2024-09-17 15:05:42 2024-09-17 16:16:50 2024-09-17 16:46:14 0:29:24 0:18:03 0:11:21 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} 4
pass 7909075 2024-09-17 15:05:43 2024-09-17 16:17:00 2024-09-17 16:48:32 0:31:32 0:21:03 0:10:29 smithi main centos 9.stream rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} 4
pass 7909076 2024-09-17 15:05:45 2024-09-17 16:17:41 2024-09-17 16:38:40 0:20:59 0:11:42 0:09:17 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7909077 2024-09-17 15:05:46 2024-09-17 16:17:41 2024-09-17 17:02:58 0:45:17 0:33:57 0:11:20 smithi main centos 9.stream rados/singleton/{all/random-eio mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest}} 2
pass 7909078 2024-09-17 15:05:47 2024-09-17 16:19:42 2024-09-17 16:36:51 0:17:09 0:08:07 0:09:02 smithi main centos 9.stream rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{centos_latest}} 1
pass 7909079 2024-09-17 15:05:48 2024-09-17 16:19:52 2024-09-17 16:53:45 0:33:53 0:23:34 0:10:19 smithi main ubuntu 22.04 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} tasks/rados_workunit_loadgen_mix} 2
pass 7909080 2024-09-17 15:05:50 2024-09-17 16:20:12 2024-09-17 17:11:18 0:51:06 0:38:48 0:12:18 smithi main ubuntu 22.04 rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}} 1
pass 7909081 2024-09-17 15:05:51 2024-09-17 16:20:43 2024-09-17 16:41:41 0:20:58 0:12:02 0:08:56 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_extra_daemon_features} 2
pass 7909082 2024-09-17 15:05:52 2024-09-17 16:20:43 2024-09-17 16:54:38 0:33:55 0:20:42 0:13:13 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps} 4
pass 7909083 2024-09-17 15:05:53 2024-09-17 16:23:14 2024-09-17 16:44:18 0:21:04 0:11:56 0:09:08 smithi main centos 9.stream rados/singleton/{all/rebuild-mondb mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest}} 1
pass 7909084 2024-09-17 15:05:54 2024-09-17 16:23:14 2024-09-17 16:45:25 0:22:11 0:11:11 0:11:00 smithi main centos 9.stream rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 7909085 2024-09-17 15:05:56 2024-09-17 16:23:55 2024-09-17 16:48:49 0:24:54 0:16:34 0:08:20 smithi main centos 9.stream rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7909086 2024-09-17 15:05:57 2024-09-17 16:23:55 2024-09-17 16:48:39 0:24:44 0:14:43 0:10:01 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio} 1
pass 7909087 2024-09-17 15:05:58 2024-09-17 16:24:56 2024-09-17 16:44:14 0:19:18 0:07:42 0:11:36 smithi main centos 9.stream rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore-options/write$/{write_v2}} tasks/{1-install 2-ceph 3-mgrmodules 4-units/per_module_finisher_stats}} 2
pass 7909088 2024-09-17 15:05:59 2024-09-17 16:27:17 2024-09-17 17:03:00 0:35:43 0:25:21 0:10:22 smithi main centos 9.stream rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} 4
pass 7909089 2024-09-17 15:06:00 2024-09-17 16:27:37 2024-09-17 16:54:52 0:27:15 0:15:37 0:11:38 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_host_drain} 3
pass 7909090 2024-09-17 15:06:02 2024-09-17 16:29:58 2024-09-17 17:16:51 0:46:53 0:36:04 0:10:49 smithi main ubuntu 22.04 rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909091 2024-09-17 15:06:03 2024-09-17 16:30:08 2024-09-17 17:10:50 0:40:42 0:29:15 0:11:27 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow_host thrashosds-health workloads/cache-snaps} 4
pass 7909092 2024-09-17 15:06:04 2024-09-17 16:31:09 2024-09-17 17:10:56 0:39:47 0:30:42 0:09:05 smithi main ubuntu 22.04 rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7909093 2024-09-17 15:06:05 2024-09-17 16:31:29 2024-09-17 16:52:30 0:21:01 0:11:21 0:09:40 smithi main centos 9.stream rados/singleton/{all/recovery-preemption mon_election/connectivity msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest}} 1
pass 7909094 2024-09-17 15:06:06 2024-09-17 16:32:00 2024-09-17 16:55:08 0:23:08 0:12:57 0:10:11 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
pass 7909095 2024-09-17 15:06:08 2024-09-17 16:32:10 2024-09-17 16:53:38 0:21:28 0:12:18 0:09:10 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909096 2024-09-17 15:06:09 2024-09-17 16:32:10 2024-09-17 17:09:09 0:36:59 0:20:22 0:16:37 smithi main centos 9.stream rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} 4
pass 7909097 2024-09-17 15:06:10 2024-09-17 16:32:11 2024-09-17 17:41:33 1:09:22 0:57:26 0:11:56 smithi main ubuntu 22.04 rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/many workloads/rados_mon_osdmap_prune} 2
pass 7909098 2024-09-17 15:06:11 2024-09-17 16:33:31 2024-09-17 16:59:46 0:26:15 0:14:56 0:11:19 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache} 4
pass 7909099 2024-09-17 15:06:12 2024-09-17 16:34:02 2024-09-17 16:55:52 0:21:50 0:12:07 0:09:43 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7909100 2024-09-17 15:06:14 2024-09-17 16:35:02 2024-09-17 16:56:35 0:21:33 0:11:39 0:09:54 smithi main ubuntu 22.04 rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest}} 2
pass 7909101 2024-09-17 15:06:15 2024-09-17 16:35:03 2024-09-17 17:03:48 0:28:45 0:17:08 0:11:37 smithi main ubuntu 22.04 rados/multimon/{clusters/9 mon_election/connectivity msgr-failures/many msgr/async-v2only no_pools objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} 3
pass 7909102 2024-09-17 15:06:16 2024-09-17 16:36:03 2024-09-17 17:11:00 0:34:57 0:23:47 0:11:10 smithi main centos 9.stream rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} 3
pass 7909103 2024-09-17 15:06:17 2024-09-17 16:36:54 2024-09-17 17:13:21 0:36:27 0:24:20 0:12:07 smithi main centos 9.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest} tasks/rados_workunit_loadgen_mostlyread} 2
fail 7909104 2024-09-17 15:06:18 2024-09-17 16:36:54 2024-09-17 17:18:31 0:41:37 0:31:43 0:09:54 smithi main centos 9.stream rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/dashboard} 2
Failure Reason:

Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest)

pass 7909105 2024-09-17 15:06:20 2024-09-17 16:37:35 2024-09-17 16:58:48 0:21:13 0:11:28 0:09:45 smithi main centos 9.stream rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7909106 2024-09-17 15:06:21 2024-09-17 16:37:35 2024-09-17 17:10:57 0:33:22 0:22:32 0:10:50 smithi main centos 9.stream rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon-stretch} 1
fail 7909107 2024-09-17 15:06:22 2024-09-17 16:38:26 2024-09-17 17:24:12 0:45:46 0:34:33 0:11:13 smithi main centos 9.stream rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} 2
Failure Reason:

Command failed (workunit test cls/test_cls_rbd.sh) on smithi031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

pass 7909108 2024-09-17 15:06:23 2024-09-17 16:38:26 2024-09-17 17:09:32 0:31:06 0:21:28 0:09:38 smithi main centos 9.stream rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/careful_host thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} 4
pass 7909109 2024-09-17 15:06:24 2024-09-17 16:38:26 2024-09-17 17:08:11 0:29:45 0:19:36 0:10:09 smithi main ubuntu 22.04 rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7909110 2024-09-17 15:06:25 2024-09-17 16:38:27 2024-09-17 16:59:56 0:21:29 0:12:33 0:08:56 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench} 1
pass 7909111 2024-09-17 15:06:27 2024-09-17 16:38:27 2024-09-17 17:07:28 0:29:01 0:15:02 0:13:59 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} thrashers/careful_host thrashosds-health workloads/dedup-io-mixed} 4
pass 7909112 2024-09-17 15:06:28 2024-09-17 16:41:58 2024-09-17 17:03:32 0:21:34 0:12:53 0:08:41 smithi main ubuntu 22.04 rados/singleton/{all/test-crash mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909113 2024-09-17 15:06:29 2024-09-17 16:41:58 2024-09-17 17:09:38 0:27:40 0:16:49 0:10:51 smithi main centos 9.stream rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore-options/write$/{write_v2}} tasks/{1-install 2-ceph 3-mgrmodules 4-units/progress}} 2
pass 7909114 2024-09-17 15:06:30 2024-09-17 16:42:39 2024-09-17 17:05:59 0:23:20 0:12:29 0:10:51 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909115 2024-09-17 15:06:31 2024-09-17 16:44:30 2024-09-17 17:07:49 0:23:19 0:13:26 0:09:53 smithi main centos 9.stream rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 2
fail 7909116 2024-09-17 15:06:33 2024-09-17 16:44:40 2024-09-17 17:28:23 0:43:43 0:33:34 0:10:09 smithi main centos 9.stream rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} 3
Failure Reason:

"2024-09-17T17:10:00.000155+0000 mon.a (mon.0) 1203 : cluster [WRN] osd.5 (root=default,host=smithi123) is down" in cluster log

pass 7909117 2024-09-17 15:06:34 2024-09-17 16:45:21 2024-09-17 17:07:00 0:21:39 0:11:35 0:10:04 smithi main ubuntu 22.04 rados/singleton/{all/test-noautoscale-flag mon_election/classic msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909118 2024-09-17 15:06:35 2024-09-17 16:45:21 2024-09-17 17:11:03 0:25:42 0:16:57 0:08:45 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/dedup-io-snaps} 4
pass 7909119 2024-09-17 15:06:36 2024-09-17 16:45:31 2024-09-17 17:23:52 0:38:21 0:27:51 0:10:30 smithi main ubuntu 22.04 rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} 4
pass 7909120 2024-09-17 15:06:37 2024-09-17 16:45:42 2024-09-17 17:15:12 0:29:30 0:19:21 0:10:09 smithi main ubuntu 22.04 rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/basic 3-final} 1
pass 7909121 2024-09-17 15:06:38 2024-09-17 16:45:42 2024-09-17 17:08:16 0:22:34 0:12:09 0:10:25 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909122 2024-09-17 15:06:40 2024-09-17 16:46:33 2024-09-17 17:07:47 0:21:14 0:10:50 0:10:24 smithi main centos 9.stream rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 7909123 2024-09-17 15:06:41 2024-09-17 16:46:33 2024-09-17 18:48:43 2:02:10 1:52:01 0:10:09 smithi main centos 9.stream rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados tasks/rados_cls_all validater/valgrind} 2
pass 7909124 2024-09-17 15:06:42 2024-09-17 16:47:34 2024-09-17 17:26:10 0:38:36 0:28:03 0:10:33 smithi main ubuntu 22.04 rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful_host thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7909125 2024-09-17 15:06:43 2024-09-17 16:48:34 2024-09-17 18:49:52 2:01:18 1:51:12 0:10:06 smithi main centos 9.stream rados/singleton/{all/thrash-backfill-full mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest}} 2
fail 7909126 2024-09-17 15:06:44 2024-09-17 16:48:45 2024-09-17 17:15:50 0:27:05 0:18:08 0:08:57 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic} 3
Failure Reason:

Command failed on smithi040 with status 5: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dfdb0ca2-7516-11ef-bceb-c7b262605968 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v1/status\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts | jq -e \'"\'"\'.data | .[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n\''

pass 7909127 2024-09-17 15:06:46 2024-09-17 16:48:45 2024-09-17 18:37:20 1:48:35 1:40:26 0:08:09 smithi main centos 9.stream rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_latest}} 1
fail 7909128 2024-09-17 15:06:47 2024-09-17 16:48:45 2024-09-17 17:38:28 0:49:43 0:33:27 0:16:16 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/pool-snaps-few-objects} 4
Failure Reason:

Found coredumps on ubuntu@smithi161.front.sepia.ceph.com

pass 7909129 2024-09-17 15:06:48 2024-09-17 16:52:46 2024-09-17 17:14:57 0:22:11 0:10:33 0:11:38 smithi main centos 9.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} tasks/readwrite} 2
pass 7909130 2024-09-17 15:06:49 2024-09-17 16:53:57 2024-09-17 17:15:30 0:21:33 0:11:42 0:09:51 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909131 2024-09-17 15:06:50 2024-09-17 16:54:07 2024-09-17 17:19:54 0:25:47 0:14:54 0:10:53 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read} 1
pass 7909132 2024-09-17 15:06:52 2024-09-17 16:54:48 2024-09-17 17:16:17 0:21:29 0:11:49 0:09:40 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
pass 7909133 2024-09-17 15:06:53 2024-09-17 16:54:58 2024-09-17 17:36:40 0:41:42 0:32:18 0:09:24 smithi main ubuntu 22.04 rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_mon_workunits} 2
pass 7909134 2024-09-17 15:06:54 2024-09-17 16:55:09 2024-09-17 17:26:30 0:31:21 0:20:50 0:10:31 smithi main centos 9.stream rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} 4
pass 7909135 2024-09-17 15:06:55 2024-09-17 16:55:29 2024-09-17 17:48:26 0:52:57 0:42:08 0:10:49 smithi main ubuntu 22.04 rados/singleton/{all/thrash-eio mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest}} 2
pass 7909136 2024-09-17 15:06:56 2024-09-17 16:56:50 2024-09-17 17:41:03 0:44:13 0:31:36 0:12:37 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/rados_api_tests} 4
pass 7909137 2024-09-17 15:06:58 2024-09-17 17:00:13 2024-09-17 17:26:50 0:26:37 0:13:41 0:12:56 smithi main ubuntu 22.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/disable mon_election/classic random-objectstore$/{bluestore/bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/prometheus}} 2
pass 7909138 2024-09-17 15:06:59 2024-09-17 17:03:13 2024-09-17 18:28:58 1:25:45 1:16:39 0:09:06 smithi main ubuntu 22.04 rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon} 1
pass 7909139 2024-09-17 15:07:00 2024-09-17 17:03:14 2024-09-17 17:20:11 0:16:57 0:07:00 0:09:57 smithi main centos 9.stream rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7909140 2024-09-17 15:07:01 2024-09-17 17:03:14 2024-09-17 17:23:39 0:20:25 0:10:41 0:09:44 smithi main ubuntu 22.04 rados/multimon/{clusters/21 mon_election/classic msgr-failures/few msgr/async no_pools objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_no_skews} 3
pass 7909141 2024-09-17 15:07:02 2024-09-17 17:03:45 2024-09-17 17:42:00 0:38:15 0:28:33 0:09:42 smithi main ubuntu 22.04 rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-hybrid} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} 3
fail 7909142 2024-09-17 15:07:04 2024-09-17 17:03:55 2024-09-17 17:25:10 0:21:15 0:12:12 0:09:03 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_rgw_multisite} 3
Failure Reason:

Command failed on smithi094 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid f0076786-7518-11ef-bceb-c7b262605968 -- bash -c \'set -e\nset -x\nwhile true; do TOKEN=$(ceph rgw realm tokens | jq -r \'"\'"\'.[0].token\'"\'"\'); echo $TOKEN; if [ "$TOKEN" != "master zone has no endpoint" ]; then break; fi; sleep 5; done\nTOKENS=$(ceph rgw realm tokens)\necho $TOKENS | jq --exit-status \'"\'"\'.[0].realm == "myrealm1"\'"\'"\'\necho $TOKENS | jq --exit-status \'"\'"\'.[0].token\'"\'"\'\nTOKEN_JSON=$(ceph rgw realm tokens | jq -r \'"\'"\'.[0].token\'"\'"\' | base64 --decode)\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.realm_name == "myrealm1"\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.endpoint | test("http://.+:\\\\d+")\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.realm_id | test("^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.access_key\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.secret\'"\'"\'\n\''

pass 7909143 2024-09-17 15:07:05 2024-09-17 17:04:05 2024-09-17 17:41:23 0:37:18 0:23:59 0:13:19 smithi main ubuntu 22.04 rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-overwrites} 4
pass 7909144 2024-09-17 15:07:06 2024-09-17 17:07:46 2024-09-17 17:37:58 0:30:12 0:19:03 0:11:09 smithi main centos 9.stream rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/connectivity msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest}} 2
pass 7909145 2024-09-17 15:07:07 2024-09-17 17:07:47 2024-09-17 18:19:55 1:12:08 1:01:01 0:11:07 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/radosbench-high-concurrency} 4
pass 7909146 2024-09-17 15:07:08 2024-09-17 17:08:07 2024-09-17 17:31:21 0:23:14 0:12:34 0:10:40 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7909147 2024-09-17 15:07:10 2024-09-17 17:08:08 2024-09-17 17:29:56 0:21:48 0:11:03 0:10:45 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/ceph-snapmapper mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909148 2024-09-17 15:07:11 2024-09-17 17:08:08 2024-09-17 17:35:42 0:27:34 0:18:40 0:08:54 smithi main centos 9.stream rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest}} 2
pass 7909149 2024-09-17 15:07:12 2024-09-17 17:08:28 2024-09-17 17:43:04 0:34:36 0:22:23 0:12:13 smithi main ubuntu 22.04 rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_set_mon_crush_locations} 3
pass 7909150 2024-09-17 15:07:13 2024-09-17 17:09:19 2024-09-17 17:44:52 0:35:33 0:24:17 0:11:16 smithi main centos 9.stream rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7909151 2024-09-17 15:07:15 2024-09-17 17:09:50 2024-09-17 17:39:05 0:29:15 0:20:19 0:08:56 smithi main centos 9.stream rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} 4
pass 7909152 2024-09-17 15:07:16 2024-09-17 17:10:00 2024-09-17 17:39:33 0:29:33 0:19:29 0:10:04 smithi main ubuntu 22.04 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest} tasks/repair_test} 2
pass 7909153 2024-09-17 15:07:17 2024-09-17 17:10:00 2024-09-17 18:11:52 1:01:52 0:51:23 0:10:29 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/radosbench} 4
pass 7909154 2024-09-17 15:07:18 2024-09-17 17:10:01 2024-09-17 17:33:30 0:23:29 0:14:42 0:08:47 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} 1
pass 7909155 2024-09-17 15:07:19 2024-09-17 17:10:01 2024-09-17 17:30:34 0:20:33 0:08:27 0:12:06 smithi main centos 9.stream rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7909156 2024-09-17 15:07:20 2024-09-17 17:11:12 2024-09-17 17:33:15 0:22:03 0:11:13 0:10:50 smithi main centos 9.stream rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 7909157 2024-09-17 15:07:22 2024-09-17 17:11:12 2024-09-17 17:46:38 0:35:26 0:26:34 0:08:52 smithi main ubuntu 22.04 rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/on fixed-2 mon_election/connectivity start} 2
fail 7909158 2024-09-17 15:07:23 2024-09-17 17:11:13 2024-09-17 17:40:12 0:28:59 0:19:47 0:09:12 smithi main centos 9.stream rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest} 1
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 7909159 2024-09-17 15:07:24 2024-09-17 17:11:13 2024-09-17 17:32:03 0:20:50 0:11:44 0:09:06 smithi main centos 9.stream rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados tasks/mon_recovery validater/lockdep} 2
pass 7909160 2024-09-17 15:07:25 2024-09-17 17:11:23 2024-09-17 17:28:21 0:16:58 0:07:44 0:09:14 smithi main centos 9.stream rados/singleton/{all/watch-notify-same-primary mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest}} 1
pass 7909161 2024-09-17 15:07:26 2024-09-17 17:11:24 2024-09-17 17:36:13 0:24:49 0:12:01 0:12:48 smithi main ubuntu 22.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/enable mon_election/connectivity random-objectstore$/{bluestore/bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/workunits}} 2
pass 7909162 2024-09-17 15:07:28 2024-09-17 17:13:34 2024-09-17 17:46:48 0:33:14 0:18:31 0:14:43 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect} 4
pass 7909163 2024-09-17 15:07:29 2024-09-17 17:15:35 2024-09-17 17:43:20 0:27:45 0:16:20 0:11:25 smithi main centos 9.stream rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/none thrashosds-health workloads/test_rbd_api} 3
pass 7909164 2024-09-17 15:07:30 2024-09-17 17:16:06 2024-09-17 17:32:54 0:16:48 0:07:09 0:09:39 smithi main centos 9.stream rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_latest}} 1
pass 7909165 2024-09-17 15:07:31 2024-09-17 17:16:06 2024-09-17 17:52:11 0:36:05 0:23:58 0:12:07 smithi main centos 9.stream rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} 4
fail 7909166 2024-09-17 15:07:32 2024-09-17 17:18:47 2024-09-17 17:44:52 0:26:05 0:15:05 0:11:00 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw} 2
Failure Reason:

reached maximum tries (120) after waiting for 120 seconds

pass 7909167 2024-09-17 15:07:34 2024-09-17 17:19:37 2024-09-17 17:45:32 0:25:55 0:14:42 0:11:13 smithi main centos 9.stream rados/singleton/{all/3-az-stretch-cluster-netsplit mon_election/classic msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest}} 4
pass 7909168 2024-09-17 15:07:35 2024-09-17 17:20:18 2024-09-17 20:02:04 2:41:46 2:31:04 0:10:42 smithi main centos 9.stream rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill} 1
pass 7909169 2024-09-17 15:07:36 2024-09-17 17:20:18 2024-09-17 18:01:09 0:40:51 0:31:28 0:09:23 smithi main centos 9.stream rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}} 1
pass 7909170 2024-09-17 15:07:37 2024-09-17 17:20:19 2024-09-17 17:50:18 0:29:59 0:19:12 0:10:47 smithi main ubuntu 22.04 rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/sync-many workloads/pool-create-delete} 2
pass 7909171 2024-09-17 15:07:38 2024-09-17 17:20:19 2024-09-17 17:51:45 0:31:26 0:17:00 0:14:26 smithi main ubuntu 22.04 rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/off mon_election/classic task/test_ca_signed_key} 2
pass 7909172 2024-09-17 15:07:40 2024-09-17 17:23:50 2024-09-17 17:52:20 0:28:30 0:17:19 0:11:11 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests} 4
pass 7909173 2024-09-17 15:07:41 2024-09-17 17:24:11 2024-09-17 17:43:39 0:19:28 0:11:09 0:08:19 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909174 2024-09-17 15:07:42 2024-09-17 17:24:11 2024-09-17 19:05:54 1:41:43 1:32:12 0:09:31 smithi main ubuntu 22.04 rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{ubuntu_latest}} 1
pass 7909175 2024-09-17 15:07:43 2024-09-17 17:24:31 2024-09-17 17:43:08 0:18:37 0:07:00 0:11:37 smithi main centos 9.stream rados/multimon/{clusters/3 mon_election/connectivity msgr-failures/many msgr/async-v1only no_pools objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} tasks/mon_clock_with_skews} 2
pass 7909176 2024-09-17 15:07:44 2024-09-17 17:25:22 2024-09-17 17:52:07 0:26:45 0:15:20 0:11:25 smithi main ubuntu 22.04 rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} 3
pass 7909177 2024-09-17 15:07:45 2024-09-17 17:26:22 2024-09-17 17:43:29 0:17:07 0:07:13 0:09:54 smithi main centos 9.stream rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest}} 1
pass 7909178 2024-09-17 15:07:47 2024-09-17 17:26:23 2024-09-17 17:49:50 0:23:27 0:13:42 0:09:45 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read} 1
pass 7909179 2024-09-17 15:07:48 2024-09-17 17:26:23 2024-09-17 17:49:22 0:22:59 0:13:30 0:09:29 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_cephadm} 1
pass 7909180 2024-09-17 15:07:49 2024-09-17 17:26:34 2024-09-17 17:50:14 0:23:40 0:14:04 0:09:36 smithi main ubuntu 22.04 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test} 2
pass 7909181 2024-09-17 15:07:50 2024-09-17 17:26:44 2024-09-17 17:50:09 0:23:25 0:13:17 0:10:08 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{centos_latest} thrashers/pggrow_host thrashosds-health workloads/redirect_set_object} 4
pass 7909182 2024-09-17 15:07:51 2024-09-17 17:27:05 2024-09-17 17:47:14 0:20:09 0:09:12 0:10:57 smithi main centos 9.stream rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{centos_latest}} 1
pass 7909183 2024-09-17 15:07:53 2024-09-17 17:28:25 2024-09-17 18:03:41 0:35:16 0:25:09 0:10:07 smithi main centos 9.stream rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/default_host thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7909184 2024-09-17 15:07:54 2024-09-17 17:28:36 2024-09-17 18:00:02 0:31:26 0:22:32 0:08:54 smithi main ubuntu 22.04 rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909185 2024-09-17 15:07:55 2024-09-17 17:28:36 2024-09-17 17:57:45 0:29:09 0:18:42 0:10:27 smithi main ubuntu 22.04 rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/repave-all} 2
pass 7909186 2024-09-17 15:07:56 2024-09-17 17:30:07 2024-09-17 18:55:51 1:25:44 1:13:01 0:12:43 smithi main centos 9.stream rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-bitmap} rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-radosbench} 4
fail 7909187 2024-09-17 15:07:57 2024-09-17 17:32:17 2024-09-17 17:54:08 0:21:51 0:11:00 0:10:51 smithi main centos 9.stream rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/e2e} 2
Failure Reason:

Command failed on smithi134 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2'

pass 7909188 2024-09-17 15:07:58 2024-09-17 17:33:08 2024-09-17 17:58:09 0:25:01 0:12:40 0:12:21 smithi main ubuntu 22.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore/bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/cli}} 2
pass 7909189 2024-09-17 15:08:00 2024-09-17 17:33:28 2024-09-17 18:27:16 0:53:48 0:41:14 0:12:34 smithi main centos 9.stream rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} 4
pass 7909190 2024-09-17 15:08:01 2024-09-17 17:35:59 2024-09-17 18:04:59 0:29:00 0:17:15 0:11:45 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/set-chunks-read} 4
pass 7909191 2024-09-17 15:08:02 2024-09-17 17:37:00 2024-09-17 17:53:53 0:16:53 0:08:07 0:08:46 smithi main centos 9.stream rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7909192 2024-09-17 15:08:03 2024-09-17 17:37:00 2024-09-17 18:02:56 0:25:56 0:14:22 0:11:34 smithi main centos 9.stream rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/classic start} 2
pass 7909193 2024-09-17 15:08:04 2024-09-17 17:38:11 2024-09-17 18:02:13 0:24:02 0:13:57 0:10:05 smithi main ubuntu 22.04 rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909194 2024-09-17 15:08:06 2024-09-17 17:38:41 2024-09-17 18:02:58 0:24:17 0:14:25 0:09:52 smithi main ubuntu 22.04 rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-low-osd-mem-target} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 7909195 2024-09-17 15:08:07 2024-09-17 17:39:22 2024-09-17 18:13:33 0:34:11 0:24:26 0:09:45 smithi main ubuntu 22.04 rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush} 4
pass 7909196 2024-09-17 15:08:08 2024-09-17 17:39:52 2024-09-17 18:09:28 0:29:36 0:20:09 0:09:27 smithi main ubuntu 22.04 rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_22.04} 1-start 2-services/rgw 3-final} 1
pass 7909197 2024-09-17 15:08:09 2024-09-17 17:39:53 2024-09-17 18:16:50 0:36:57 0:25:35 0:11:22 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-bitmap} rados supported-random-distro$/{ubuntu_latest} thrashers/careful_host thrashosds-health workloads/small-objects-balanced} 4
pass 7909198 2024-09-17 15:08:10 2024-09-17 17:41:13 2024-09-17 18:02:48 0:21:35 0:11:32 0:10:03 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909199 2024-09-17 15:08:11 2024-09-17 17:41:14 2024-09-17 18:00:08 0:18:54 0:08:48 0:10:06 smithi main centos 9.stream rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest}} 1
pass 7909200 2024-09-17 15:08:13 2024-09-17 17:41:14 2024-09-17 18:04:49 0:23:35 0:13:52 0:09:43 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw} 1
fail 7909201 2024-09-17 15:08:14 2024-09-17 17:41:14 2024-09-17 18:36:55 0:55:41 0:44:18 0:11:23 smithi main centos 9.stream rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} 1
Failure Reason:

Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh'

pass 7909202 2024-09-17 15:08:15 2024-09-17 17:41:15 2024-09-17 17:56:02 0:14:47 0:05:23 0:09:24 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos} 1
pass 7909203 2024-09-17 15:08:16 2024-09-17 17:41:15 2024-09-17 18:02:38 0:21:23 0:11:34 0:09:49 smithi main centos 9.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest} tasks/libcephsqlite} 2
pass 7909204 2024-09-17 15:08:17 2024-09-17 17:41:16 2024-09-17 18:00:13 0:18:57 0:09:48 0:09:09 smithi main centos 9.stream rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925} 2
pass 7909205 2024-09-17 15:08:19 2024-09-17 17:41:16 2024-09-17 18:01:16 0:20:00 0:09:05 0:10:55 smithi main centos 9.stream rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}} 1
pass 7909206 2024-09-17 15:08:20 2024-09-17 17:42:16 2024-09-17 18:15:07 0:32:51 0:20:12 0:12:39 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/small-objects-localized} 4
pass 7909207 2024-09-17 15:08:21 2024-09-17 17:43:17 2024-09-17 18:02:32 0:19:15 0:08:56 0:10:19 smithi main centos 9.stream rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest}} 1
pass 7909208 2024-09-17 15:08:22 2024-09-17 17:43:17 2024-09-17 18:04:40 0:21:23 0:12:05 0:09:18 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add} 2
fail 7909209 2024-09-17 15:08:23 2024-09-17 17:43:18 2024-09-17 18:22:49 0:39:31 0:28:25 0:11:06 smithi main centos 9.stream rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/cache-snaps} 3
Failure Reason:

"2024-09-17T18:10:00.000152+0000 mon.a (mon.0) 1475 : cluster [WRN] osd.3 (root=default,host=smithi040) is down" in cluster log

pass 7909210 2024-09-17 15:08:25 2024-09-17 17:43:38 2024-09-17 18:02:45 0:19:07 0:08:58 0:10:09 smithi main centos 9.stream rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore/bluestore-bitmap} tasks/{1-install 2-ceph 3-mgrmodules 4-units/crash}} 2
pass 7909211 2024-09-17 15:08:26 2024-09-17 17:43:49 2024-09-17 18:10:29 0:26:40 0:14:44 0:11:56 smithi main ubuntu 22.04 rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery} 2
pass 7909212 2024-09-17 15:08:27 2024-09-17 17:45:09 2024-09-17 18:18:15 0:33:06 0:24:19 0:08:47 smithi main centos 9.stream rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} 3
pass 7909213 2024-09-17 15:08:28 2024-09-17 17:45:10 2024-09-17 18:28:54 0:43:44 0:31:25 0:12:19 smithi main ubuntu 22.04 rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7909214 2024-09-17 15:08:29 2024-09-17 17:45:50 2024-09-17 18:16:24 0:30:34 0:19:30 0:11:04 smithi main ubuntu 22.04 rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_extra_daemon_features} 2
pass 7909215 2024-09-17 15:08:31 2024-09-17 17:46:51 2024-09-17 18:16:22 0:29:31 0:20:51 0:08:40 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909216 2024-09-17 15:08:32 2024-09-17 17:46:51 2024-09-17 18:10:28 0:23:37 0:12:27 0:11:10 smithi main ubuntu 22.04 rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909217 2024-09-17 15:08:33 2024-09-17 17:47:02 2024-09-17 18:24:40 0:37:38 0:24:45 0:12:53 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/upmap-read mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/default_host thrashosds-health workloads/small-objects} 4
pass 7909218 2024-09-17 15:08:34 2024-09-17 17:47:32 2024-09-17 18:23:48 0:36:16 0:24:09 0:12:07 smithi main ubuntu 22.04 rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-lz4} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-balanced} 4
pass 7909219 2024-09-17 15:08:35 2024-09-17 17:50:13 2024-09-17 18:31:24 0:41:11 0:31:06 0:10:05 smithi main centos 9.stream rados/objectstore/{backends/objectstore-memstore supported-random-distro$/{centos_latest}} 1
pass 7909220 2024-09-17 15:08:37 2024-09-17 17:50:24 2024-09-17 18:11:19 0:20:55 0:12:19 0:08:36 smithi main centos 9.stream rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag} 2
pass 7909221 2024-09-17 15:08:38 2024-09-17 17:50:24 2024-09-17 18:13:59 0:23:35 0:14:03 0:09:32 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write} 1
pass 7909222 2024-09-17 15:08:39 2024-09-17 17:50:24 2024-09-17 18:21:59 0:31:35 0:20:00 0:11:35 smithi main centos 9.stream rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/mapgap_host thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=2-crush} 4
pass 7909223 2024-09-17 15:08:40 2024-09-17 17:50:35 2024-09-17 18:14:44 0:24:09 0:14:21 0:09:48 smithi main centos 9.stream rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{centos_latest}} 1
pass 7909224 2024-09-17 15:08:41 2024-09-17 17:51:45 2024-09-17 18:18:34 0:26:49 0:17:40 0:09:09 smithi main centos 9.stream rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{centos_latest}} 2
pass 7909225 2024-09-17 15:08:43 2024-09-17 17:51:46 2024-09-17 18:33:52 0:42:06 0:31:42 0:10:24 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced} 4
pass 7909226 2024-09-17 15:08:44 2024-09-17 17:51:46 2024-09-17 18:15:50 0:24:04 0:13:37 0:10:27 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_host_drain} 3
pass 7909227 2024-09-17 15:08:45 2024-09-17 17:52:27 2024-09-17 18:27:27 0:35:00 0:25:07 0:09:53 smithi main centos 9.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} tasks/rados_api_tests} 2
pass 7909228 2024-09-17 15:08:46 2024-09-17 17:52:27 2024-09-17 18:18:15 0:25:48 0:14:09 0:11:39 smithi main ubuntu 22.04 rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 7909229 2024-09-17 15:08:47 2024-09-17 17:54:28 2024-09-17 18:24:53 0:30:25 0:16:47 0:13:38 smithi main centos 9.stream rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-stupid} rados tasks/rados_cls_all validater/lockdep} 2
pass 7909230 2024-09-17 15:08:49 2024-09-17 17:57:59 2024-09-17 18:55:45 0:57:46 0:47:52 0:09:54 smithi main ubuntu 22.04 rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909231 2024-09-17 15:08:50 2024-09-17 17:57:59 2024-09-17 18:26:21 0:28:22 0:17:38 0:10:44 smithi main centos 9.stream rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_latest}} 1
pass 7909232 2024-09-17 15:08:51 2024-09-17 17:58:20 2024-09-17 18:40:10 0:41:50 0:30:28 0:11:22 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zstd} rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/snaps-few-objects-localized} 4
pass 7909233 2024-09-17 15:08:52 2024-09-17 18:00:30 2024-09-17 18:50:18 0:49:48 0:37:09 0:12:39 smithi main ubuntu 22.04 rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} 4
pass 7909234 2024-09-17 15:08:54 2024-09-17 18:02:11 2024-09-17 18:25:16 0:23:05 0:14:02 0:09:03 smithi main centos 9.stream rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/connectivity start} 2
pass 7909235 2024-09-17 15:08:55 2024-09-17 18:02:12 2024-09-17 18:25:46 0:23:34 0:14:28 0:09:06 smithi main ubuntu 22.04 rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{ubuntu_latest} mgr_ttl_cache/disable mon_election/connectivity random-objectstore$/{bluestore-options/write$/{write_random}} tasks/{1-install 2-ceph 3-mgrmodules 4-units/devicehealth}} 2
fail 7909236 2024-09-17 15:08:56 2024-09-17 18:02:12 2024-09-17 19:39:50 1:37:38 1:28:03 0:09:35 smithi main ubuntu 22.04 rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} 1
Failure Reason:

Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1889a165b5128787ec693450a3d1c5df5d6dc203 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh'

pass 7909237 2024-09-17 15:08:57 2024-09-17 18:02:12 2024-09-17 18:21:41 0:19:29 0:10:07 0:09:22 smithi main ubuntu 22.04 rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/many msgr/async objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-low-osd-mem-target} rados supported-random-distro$/{ubuntu_latest}} 1
fail 7909238 2024-09-17 15:08:58 2024-09-17 18:02:13 2024-09-17 18:27:37 0:25:24 0:11:49 0:13:35 smithi main ubuntu 22.04 rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait} 2
Failure Reason:

Command failed on smithi005 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1889a165b5128787ec693450a3d1c5df5d6dc203 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d24425a0-7521-11ef-bceb-c7b262605968 -- ceph orch host add smithi179'

pass 7909239 2024-09-17 15:08:59 2024-09-17 18:02:13 2024-09-17 18:40:10 0:37:57 0:26:19 0:11:38 smithi main centos 9.stream rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-zlib} rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/mapgap_host thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 4
pass 7909240 2024-09-17 15:09:01 2024-09-17 18:03:14 2024-09-17 18:25:19 0:22:05 0:11:17 0:10:48 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909241 2024-09-17 15:09:02 2024-09-17 18:03:54 2024-09-17 18:38:54 0:35:00 0:25:55 0:09:05 smithi main centos 9.stream rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/force-sync-many workloads/rados_api_tests} 2
pass 7909242 2024-09-17 15:09:03 2024-09-17 18:03:55 2024-09-17 18:38:30 0:34:35 0:24:09 0:10:26 smithi main centos 9.stream rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-hybrid} rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects} 4
pass 7909243 2024-09-17 15:09:04 2024-09-17 18:05:05 2024-09-17 18:28:30 0:23:25 0:13:55 0:09:30 smithi main ubuntu 22.04 rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read} 1
pass 7909244 2024-09-17 15:09:05 2024-09-17 18:05:16 2024-09-17 18:26:05 0:20:49 0:12:28 0:08:21 smithi main centos 9.stream rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/on mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}} 1
pass 7909245 2024-09-17 15:09:07 2024-09-17 18:05:16 2024-09-17 18:23:17 0:18:01 0:06:51 0:11:10 smithi main centos 9.stream rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/many msgr/async no_pools objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-bitmap} rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews} 2
pass 7909246 2024-09-17 15:09:08 2024-09-17 18:05:16 2024-09-17 18:45:49 0:40:33 0:25:05 0:15:28 smithi main centos 9.stream rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-bitmap} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} 3
pass 7909247 2024-09-17 15:09:09 2024-09-17 18:11:42 2024-09-17 18:55:18 0:43:36 0:34:44 0:08:52 smithi main ubuntu 22.04 rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} 1
pass 7909248 2024-09-17 15:09:10 2024-09-17 18:11:43 2024-09-17 19:00:27 0:48:44 0:36:48 0:11:56 smithi main centos 9.stream rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-stupid} rados supported-random-distro$/{centos_latest}} 1
pass 7909249 2024-09-17 15:09:11 2024-09-17 18:12:03 2024-09-17 18:35:33 0:23:30 0:13:30 0:10:00 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1