Name Machine Type Up Locked Locked Since Locked By OS Type OS Version Arch Description
smithi138.front.sepia.ceph.com smithi True True 2024-04-24 20:09:25.676039 scheduled_teuthology@teuthology centos 8 x86_64 /home/teuthworker/archive/teuthology-2024-04-24_01:16:02-upgrade:quincy-x-reef-distro-default-smithi/7671111
Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7671787 2024-04-24 15:49:46 2024-04-24 16:20:14 2024-04-24 17:35:43 1:15:29 1:08:04 0:07:25 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

reached maximum tries (51) after waiting for 300 seconds

pass 7671736 2024-04-24 15:14:34 2024-04-24 19:11:50 2024-04-24 20:07:16 0:55:26 0:41:16 0:14:10 smithi main centos 8.stream krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/upmap thrashosds-health workloads/rbd_fio} 3
fail 7671625 2024-04-24 14:12:37 2024-04-24 17:39:06 2024-04-24 18:58:13 1:19:07 1:10:56 0:08:11 smithi main rhel 8.6 rbd/pwl-cache/tmpfs/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{rhel_8} 4-cache-path 5-cache-mode/ssd 6-cache-size/5G 7-workloads/qemu_xfstests conf/{disable-pool-app}} 2
Failure Reason:

Command failed on smithi204 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success'

pass 7671466 2024-04-24 13:00:50 2024-04-24 13:35:26 2024-04-24 14:08:57 0:33:31 0:22:06 0:11:25 smithi main ubuntu 22.04 rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} 2
pass 7671434 2024-04-24 13:00:22 2024-04-24 13:10:40 2024-04-24 13:37:14 0:26:34 0:15:48 0:10:46 smithi main ubuntu 22.04 rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} 2
fail 7671259 2024-04-24 09:26:34 2024-04-24 09:52:29 2024-04-24 10:33:46 0:41:17 0:33:24 0:07:53 smithi main centos 9.stream rbd/iscsi/{0-single-container-host base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} workloads/cephadm_iscsi} 3
Failure Reason:

"2024-04-24T10:07:42.412256+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 7671216 2024-04-24 07:33:11 2024-04-24 08:04:31 2024-04-24 09:49:08 1:44:37 1:19:44 0:24:53 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} 3
Failure Reason:

"2024-04-24T08:55:08.800397+0000 mds.b (mds.0) 34 : cluster [WRN] Scrub error on inode 0x10000000215 (/volumes/qa/sv_0/e395bed6-5bba-41b8-a256-3505e4afcca2/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.b log and `damage ls` output for details" in cluster log

fail 7671197 2024-04-24 07:32:47 2024-04-24 07:34:18 2024-04-24 08:13:29 0:39:11 0:28:23 0:10:48 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} 3
Failure Reason:

Command failed (workunit test suites/fsx.sh) on smithi059 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f81fb397ae98da7563d451a78c61574c8f4e6e0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh'

running 7671111 2024-04-24 01:18:10 2024-04-24 20:07:25 2024-04-24 20:54:30 0:47:43 smithi main centos 8.stream upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd_api 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} 2
pass 7671068 2024-04-24 01:17:26 2024-04-24 14:08:52 2024-04-24 16:20:31 2:11:39 2:04:05 0:07:34 smithi main rhel 8.6 upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-import-export 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
pass 7671042 2024-04-24 01:16:59 2024-04-24 10:39:15 2024-04-24 13:11:26 2:32:11 2:25:14 0:06:57 smithi main rhel 8.6 upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
pass 7670947 2024-04-24 01:09:58 2024-04-24 04:46:39 2024-04-24 07:34:10 2:47:31 2:38:54 0:08:37 smithi main centos 8.stream upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools_crun 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/classic} 2
pass 7670905 2024-04-24 01:09:15 2024-04-24 02:29:03 2024-04-24 04:46:48 2:17:45 2:05:33 0:12:12 smithi main centos 8.stream upgrade:pacific-x/stress-split/{0-distro/centos_8.stream_container_tools 0-roles 1-start 2-first-half-tasks/rbd-cls 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/rbd-import-export mon_election/connectivity} 2
pass 7670649 2024-04-23 21:41:28 2024-04-24 02:06:38 2024-04-24 02:31:16 0:24:38 0:13:40 0:10:58 smithi main ubuntu 22.04 rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest} 2
pass 7670565 2024-04-23 21:20:05 2024-04-24 00:56:47 2024-04-24 02:07:25 1:10:38 1:01:37 0:09:01 smithi main centos 9.stream rbd/migration/{1-base/install 2-clusters/{fixed-3 openstack} 3-objectstore/bluestore-comp-zstd 4-supported-random-distro$/{centos_latest} 5-data-pool/replicated 6-prepare/qcow2-http 7-io-workloads/qemu_xfstests 8-migrate-workloads/execute 9-cleanup/cleanup conf/{disable-pool-app}} 3
pass 7670538 2024-04-23 21:19:39 2024-04-24 00:23:23 2024-04-24 00:57:50 0:34:27 0:23:24 0:11:03 smithi main ubuntu 22.04 rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/ec extra-conf/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} workloads/python_api_tests} 3
pass 7670504 2024-04-23 21:19:05 2024-04-23 23:47:14 2024-04-24 00:23:43 0:36:29 0:23:26 0:13:03 smithi main ubuntu 22.04 rbd/librbd/{cache/writethrough clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/ec extra-conf/copy-on-read min-compat-client/octopus msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{ubuntu_latest} workloads/python_api_tests_with_journaling} 3
pass 7670442 2024-04-23 21:18:06 2024-04-23 22:43:52 2024-04-23 23:47:41 1:03:49 0:53:21 0:10:28 smithi main ubuntu 22.04 rbd/pwl-cache/home/{1-base/install 2-cluster/{fix-2 openstack} 3-supported-random-distro$/{ubuntu_latest} 4-cache-path 5-cache-mode/ssd 6-cache-size/8G 7-workloads/c_api_tests_with_defaults conf/{disable-pool-app}} 2
pass 7670185 2024-04-23 20:18:55 2024-04-23 20:46:44 2024-04-23 22:43:50 1:57:06 1:45:54 0:11:12 smithi main ubuntu 22.04 rbd/maintenance/{base/install clusters/{fixed-3 openstack} conf/{disable-pool-app} objectstore/bluestore-low-osd-mem-target qemu/xfstests supported-random-distro$/{ubuntu_latest} workloads/rebuild_object_map} 3
pass 7670137 2024-04-23 20:18:07 2024-04-23 20:19:18 2024-04-23 20:47:53 0:28:35 0:17:50 0:10:45 smithi main centos 9.stream rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} conf/{disable-pool-app} data-pool/replicated extra-conf/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_latest} workloads/fsx} 3