Name Machine Type Up Locked Locked Since Locked By OS Type OS Version Arch Description
smithi038.front.sepia.ceph.com smithi True True 2024-03-18 21:32:36.596245 scheduled_pdonnell@teuthology centos 9 x86_64 /home/teuthworker/archive/pdonnell-2024-03-18_21:05:49-fs-wip-batrick-testing-20240318.181317-distro-default-smithi/7609142
Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
running 7609142 2024-03-18 21:06:22 2024-03-18 21:31:16 2024-03-19 06:53:28 9:23:19 smithi main centos 9.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/xfstests-dev} 2
fail 7608888 2024-03-18 14:21:23 2024-03-18 16:04:53 2239 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} 3
Failure Reason:

"2024-03-18T15:43:48.162136+0000 mon.a (mon.0) 683 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log

fail 7608826 2024-03-18 14:20:21 2024-03-18 14:21:16 2024-03-18 15:09:49 0:48:33 0:36:53 0:11:40 smithi main centos 9.stream fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/centos_latest mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} 2
Failure Reason:

Test failure: test_reading_conf (tasks.cephfs.test_cephfs_shell.TestShellOpts)

pass 7608691 2024-03-18 00:38:31 2024-03-18 01:54:51 2024-03-18 03:36:30 1:41:39 1:32:06 0:09:33 smithi main ubuntu 22.04 rgw/verify/{0-install accounts$/{tenant} clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} 2
fail 7608559 2024-03-17 23:19:31 2024-03-18 00:53:08 2024-03-18 01:46:28 0:53:20 0:41:42 0:11:38 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} 3
Failure Reason:

"2024-03-18T01:16:37.325508+0000 mon.a (mon.0) 212 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] is down (out of quorum)" in cluster log

fail 7608523 2024-03-17 23:18:52 2024-03-18 00:06:56 2024-03-18 00:52:33 0:45:37 0:35:35 0:10:02 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} 3
Failure Reason:

"2024-03-18T00:28:19.222977+0000 mon.a (mon.0) 212 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] is down (out of quorum)" in cluster log

fail 7608478 2024-03-17 23:18:04 2024-03-17 23:23:58 2024-03-18 00:05:48 0:41:50 0:30:28 0:11:22 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} 3
Failure Reason:

"2024-03-17T23:48:16.615702+0000 mon.a (mon.0) 212 : cluster [WRN] mon.b (rank 2) addr [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] is down (out of quorum)" in cluster log

pass 7606440 2024-03-16 15:08:28 2024-03-18 20:45:23 2024-03-18 21:32:26 0:47:03 0:32:40 0:14:23 smithi main centos 8.stream rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/ec features/readbalance msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_8} workloads/qemu_bonnie} 3
pass 7606396 2024-03-16 15:07:53 2024-03-18 20:17:02 2024-03-18 20:47:58 0:30:56 0:16:48 0:14:08 smithi main ubuntu 20.04 rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/none features/defaults msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_20.04} workloads/qemu_fsstress} 3
pass 7606303 2024-03-16 15:05:55 2024-03-18 19:07:42 2024-03-18 20:18:42 1:11:00 1:00:07 0:10:53 smithi main ubuntu 22.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} 5
fail 7606214 2024-03-16 15:04:43 2024-03-18 17:44:48 2024-03-18 18:58:51 1:14:03 0:56:30 0:17:33 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} 3
Failure Reason:

Command failed (workunit test suites/dbench.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=49bdc51142759622a281d3e5327eebd07cbb8bb3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

pass 7604909 2024-03-15 21:41:00 2024-03-18 12:07:02 2024-03-18 12:30:40 0:23:38 0:13:55 0:09:43 smithi main ubuntu 22.04 rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} 2
pass 7604881 2024-03-15 21:40:32 2024-03-18 11:43:38 2024-03-18 12:06:54 0:23:16 0:13:32 0:09:44 smithi main ubuntu 22.04 rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest} 1
fail 7604769 2024-03-15 21:09:39 2024-03-18 10:31:43 2024-03-18 11:33:28 1:01:45 0:48:35 0:13:10 smithi main ubuntu 22.04 orch/cephadm/thrash/{0-distro/ubuntu_22.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} 2
Failure Reason:

"2024-03-18T10:56:04.373520+0000 mon.a (mon.0) 179 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,b" in cluster log

pass 7604484 2024-03-15 20:15:18 2024-03-18 06:38:02 2024-03-18 10:33:49 3:55:47 3:46:27 0:09:20 smithi main centos 8.stream rbd/qemu/{cache/writethrough clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated features/journaling msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{centos_8} workloads/qemu_xfstests} 3
pass 7604434 2024-03-15 20:13:55 2024-03-18 06:02:57 2024-03-18 06:38:12 0:35:15 0:21:44 0:13:31 smithi main rhel 8.6 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds_creation_retry} 2
pass 7604394 2024-03-15 20:13:22 2024-03-18 05:29:25 2024-03-18 06:03:07 0:33:42 0:24:26 0:09:16 smithi main centos 8.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/damage} 2
pass 7604367 2024-03-15 20:13:00 2024-03-18 05:05:21 2024-03-18 05:29:21 0:24:00 0:14:11 0:09:49 smithi main centos 8.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/auto-repair} 2
pass 7604329 2024-03-15 20:12:28 2024-03-18 04:40:52 2024-03-18 05:06:15 0:25:23 0:14:38 0:10:45 smithi main centos 8.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/workunit/dir-max-entries} 2
pass 7604296 2024-03-15 20:12:01 2024-03-18 04:13:15 2024-03-18 04:41:14 0:27:59 0:16:45 0:11:14 smithi main ubuntu 22.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/sessionmap} 2