Name Machine Type Up Locked Locked Since Locked By OS Type OS Version Arch Description
smithi137.front.sepia.ceph.com smithi True True 2024-04-25 14:32:38.970031 scheduled_leonidus@teuthology centos 9 x86_64 /home/teuthworker/archive/leonidus-2024-04-25_13:50:46-fs-wip-lusov-quiescer-distro-default-smithi/7673510
Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
running 7673510 2024-04-25 13:51:45 2024-04-25 14:32:08 2024-04-25 15:03:42 0:32:04 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/iozone}} 3
fail 7673457 2024-04-25 12:18:03 2024-04-25 12:48:12 2024-04-25 13:11:45 0:23:33 0:13:16 0:10:17 smithi main ubuntu 22.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/with-quiesce 2-workunit/suites/iozone}} 2
Failure Reason:

'NoneType' object has no attribute 'get'

fail 7673360 2024-04-25 11:14:49 2024-04-25 12:01:38 2024-04-25 12:40:40 0:39:02 0:32:48 0:06:14 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/direct_io}} 3
Failure Reason:

'cluster'

pass 7673159 2024-04-25 10:02:29 2024-04-25 13:19:28 2024-04-25 14:32:30 1:13:02 1:06:57 0:06:05 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} 3
pass 7673121 2024-04-25 10:01:50 2024-04-25 10:59:41 2024-04-25 12:02:43 1:03:02 0:55:49 0:07:13 smithi main centos 9.stream fs/volumes/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/volumes/{overrides test/clone}} 2
fail 7673058 2024-04-25 09:43:41 2024-04-25 09:56:41 2024-04-25 10:46:38 0:49:57 0:38:56 0:11:01 smithi main centos 9.stream fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/admin} 2
Failure Reason:

Test failure: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)

fail 7673021 2024-04-25 09:10:41 2024-04-25 09:10:46 2024-04-25 09:51:48 0:41:02 0:34:11 0:06:51 smithi main centos 9.stream fs:nfs/{cluster/{1-node} conf/{client mds mgr mon osd} overrides/{ignorelist_health pg_health} supported-random-distros$/{centos_latest} tasks/nfs} 1
Failure Reason:

"2024-04-25T09:27:19.767852+0000 mon.a (mon.0) 610 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON)" in cluster log

fail 7672994 2024-04-25 08:15:29 2024-04-25 08:20:36 2024-04-25 08:57:58 0:37:22 0:24:16 0:13:06 smithi main ubuntu 22.04 fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/scrub} 2
Failure Reason:

Test failure: test_scrub_when_mds_is_inactive (tasks.cephfs.test_scrub_checks.TestScrubControls)

fail 7672937 2024-04-25 07:22:00 2024-04-25 07:22:53 2024-04-25 08:12:26 0:49:33 0:40:13 0:09:20 smithi main ubuntu 22.04 fs:functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/admin} 2
Failure Reason:

Test failure: test_multifs_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize)

fail 7672814 2024-04-25 03:55:21 2024-04-25 04:58:40 2024-04-25 05:38:54 0:40:14 0:30:32 0:09:42 smithi main centos 9.stream orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-04-25T05:30:00.000162+0000 mon.smithi137 (mon.0) 496 : cluster [WRN] Health detail: HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid" in cluster log

pass 7672789 2024-04-25 03:54:56 2024-04-25 04:42:09 2024-04-25 04:59:25 0:17:16 0:05:39 0:11:37 smithi main centos 9.stream orch:cephadm/workunits/{0-distro/centos_9.stream agent/off mon_election/classic task/test_cephadm_repos} 1
pass 7672766 2024-04-25 03:54:33 2024-04-25 04:26:18 2024-04-25 04:46:26 0:20:08 0:13:01 0:07:07 smithi main centos 9.stream orch:cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rmdir-reactivate} 2
pass 7672747 2024-04-25 03:54:13 2024-04-25 04:08:32 2024-04-25 04:27:11 0:18:39 0:09:44 0:08:55 smithi main centos 9.stream orch:cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} 3
pass 7672463 2024-04-24 21:49:22 2024-04-25 03:18:15 2024-04-25 03:50:49 0:32:34 0:16:04 0:16:30 smithi main centos 8.stream krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_fallocate} 3
pass 7672434 2024-04-24 21:48:52 2024-04-25 02:48:40 2024-04-25 03:21:30 0:32:50 0:16:24 0:16:26 smithi main centos 8.stream krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 7672411 2024-04-24 21:48:29 2024-04-25 02:27:18 2024-04-25 02:52:19 0:25:01 0:13:39 0:11:22 smithi main centos 8.stream krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} 1
fail 7672351 2024-04-24 21:28:14 2024-04-25 01:40:53 2024-04-25 02:26:26 0:45:33 0:34:26 0:11:07 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} 3
Failure Reason:

error during scrub thrashing: rank damage found: {'backtrace'}

fail 7672175 2024-04-24 21:25:24 2024-04-24 22:47:51 2024-04-25 01:41:10 2:53:19 2:41:28 0:11:51 smithi main ubuntu 22.04 fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/fsstress validater/valgrind} 2
Failure Reason:

Command failed on smithi137 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf'

fail 7672123 2024-04-24 20:45:16 2024-04-24 21:28:35 2024-04-24 22:43:12 1:14:37 0:41:26 0:33:11 smithi main centos 9.stream rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/kafka/{0-install supported-distros/{centos_latest} test_kafka}} 2
Failure Reason:

Command failed (bucket notification tests against different endpoints) on smithi067 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test'

fail 7671847 2024-04-24 15:50:47 2024-04-24 17:08:04 2024-04-24 17:55:39 0:47:35 0:37:11 0:10:24 smithi main ubuntu 22.04 orch:cephadm/upgrade/{1-start-distro/1-start-ubuntu_22.04 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} 2
Failure Reason:

"2024-04-24T17:38:50.032070+0000 mon.a (mon.0) 874 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log