Name Machine Type Up Locked Locked Since Locked By OS Type OS Version Arch Description
smithi119.front.sepia.ceph.com smithi True True 2024-05-27 15:35:02.163203 scheduled_teuthology@teuthology ubuntu 22.04 x86_64 /home/teuthworker/archive/teuthology-2024-05-21_20:16:16-rbd-main-distro-default-smithi/7718811
Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 7727925 2024-05-27 13:52:10 2024-05-27 13:52:34 2024-05-27 14:15:48 0:23:14 0:14:20 0:08:54 smithi main centos 9.stream rgw:notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/amqp/{0-install centos_latest test_amqp}} 2
pass 7727777 2024-05-27 07:25:48 2024-05-27 08:11:19 2024-05-27 08:58:11 0:46:52 0:32:54 0:13:58 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/fs/norstats}} 3
fail 7727730 2024-05-27 06:48:18 2024-05-27 10:37:54 2024-05-27 11:33:31 0:55:37 0:44:35 0:11:02 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/blogbench}} 3
Failure Reason:

error during quiesce thrashing: Error quiescing set '5402aa82': 110 (ETIMEDOUT)

fail 7727710 2024-05-27 06:47:56 2024-05-27 09:45:01 2024-05-27 10:40:24 0:55:23 0:42:26 0:12:57 smithi main centos 9.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-05-27T10:30:00.000355+0000 mon.smithi119 (mon.0) 515 : cluster [WRN] osd.4 (root=default,host=smithi138) is down" in cluster log

fail 7727678 2024-05-27 06:47:21 2024-05-27 08:57:48 2024-05-27 09:49:15 0:51:27 0:41:46 0:09:41 smithi main centos 9.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

"2024-05-27T09:40:00.000322+0000 mon.smithi119 (mon.0) 563 : cluster [ERR] fs cephfs is degraded" in cluster log

fail 7727670 2024-05-27 05:54:19 2024-05-27 07:20:52 2024-05-27 08:14:54 0:54:02 0:43:40 0:10:22 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/fs/test_o_trunc}} 3
Failure Reason:

Command failed on smithi189 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid 3ed1cb70-1bfc-11ef-bc9b-c7b262605968 -- ceph daemon mds.c perf dump'

fail 7727632 2024-05-27 05:53:38 2024-05-27 06:38:21 2024-05-27 07:21:33 0:43:12 0:33:50 0:09:22 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/fs/norstats}} 3
Failure Reason:

Command failed on smithi119 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:38f9d900c83c63799fbdbe61acc9a11b0d3554a6 shell --fsid c5aa1a14-1bf5-11ef-bc9b-c7b262605968 -- ceph daemon mds.a perf dump'

dead 7727627 2024-05-27 05:53:33 2024-05-27 06:33:29 2024-05-27 06:35:53 0:02:24 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/postgres}} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi144

pass 7727556 2024-05-27 00:24:49 2024-05-27 03:04:50 2024-05-27 04:22:51 1:18:01 1:07:36 0:10:25 smithi main centos 8.stream upgrade:octopus-x/parallel/{0-distro/centos_8.stream_container_tools 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api}} 2
pass 7727515 2024-05-26 22:06:15 2024-05-27 13:02:51 2024-05-27 13:31:56 0:29:05 0:22:50 0:06:15 smithi main rhel 8.6 rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_cls_all} 2
pass 7727496 2024-05-26 22:05:56 2024-05-27 12:34:08 2024-05-27 13:02:53 0:28:45 0:18:25 0:10:20 smithi main ubuntu 20.04 rados/cephadm/smoke-singlehost/{0-random-distro$/{ubuntu_20.04} 1-start 2-services/rgw 3-final} 1
pass 7727434 2024-05-26 22:04:55 2024-05-27 12:02:52 2024-05-27 12:34:58 0:32:06 0:22:33 0:09:33 smithi main ubuntu 20.04 rados/singleton/{all/random-eio mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_20.04}} 2
pass 7727379 2024-05-26 22:04:00 2024-05-27 11:33:35 2024-05-27 12:03:01 0:29:26 0:20:06 0:09:20 smithi main ubuntu 22.04 rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{ubuntu_latest}} 1
pass 7727282 2024-05-26 22:02:22 2024-05-27 05:44:16 2024-05-27 06:34:41 0:50:25 0:43:35 0:06:50 smithi main rhel 8.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency} 2
pass 7727243 2024-05-26 22:01:42 2024-05-27 05:16:58 2024-05-27 05:45:29 0:28:31 0:15:40 0:12:51 smithi main centos 9.stream rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} tasks/rados_cls_all} 2
fail 7727173 2024-05-26 21:27:25 2024-05-26 23:52:42 2024-05-27 02:04:16 2:11:34 1:59:04 0:12:30 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} 3
Failure Reason:

error during scrub thrashing: rank damage found: {'backtrace'}

pass 7726940 2024-05-26 21:05:55 2024-05-27 02:29:34 2024-05-27 03:05:46 0:36:12 0:25:30 0:10:42 smithi main ubuntu 22.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench-high-concurrency} 2
dead 7726929 2024-05-26 21:05:45 2024-05-27 02:22:49 2024-05-27 02:24:53 0:02:04 smithi main ubuntu 22.04 rados/multimon/{clusters/9 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi119

pass 7726889 2024-05-26 21:05:06 2024-05-27 02:04:21 2024-05-27 02:23:46 0:19:25 0:10:59 0:08:26 smithi main ubuntu 22.04 rados/singleton/{all/resolve_stuck_peering mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}} 2
dead 7726318 2024-05-26 10:47:46 2024-05-26 11:39:38 2024-05-26 23:56:50 12:17:12 smithi main centos 9.stream fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} 3
Failure Reason:

hit max job timeout