Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7385760 2023-08-31 16:24:45 2023-08-31 18:26:19 2023-08-31 18:33:31 0:07:12 smithi main centos 9.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} 2
pass 7385761 2023-08-31 16:24:45 2023-08-31 18:26:20 2023-08-31 19:12:29 0:46:09 0:39:27 0:06:42 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} 3
fail 7385762 2023-08-31 16:24:46 2023-08-31 18:27:10 2023-08-31 18:52:36 0:25:26 0:15:22 0:10:04 smithi main ubuntu 22.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} 2
Failure Reason:

"2023-08-31T18:47:30.474023+0000 mon.a (mon.0) 233 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

pass 7385763 2023-08-31 16:24:47 2023-08-31 18:27:11 2023-08-31 19:11:20 0:44:09 0:32:55 0:11:14 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} 3
fail 7385764 2023-08-31 16:24:48 2023-08-31 18:28:48 2023-08-31 18:50:17 0:21:29 0:15:06 0:06:23 smithi main rhel 8.6 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} 2
Failure Reason:

Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

fail 7385765 2023-08-31 16:24:49 2023-08-31 18:28:48 2023-08-31 19:06:18 0:37:30 0:22:16 0:15:14 smithi main centos 8.stream fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} 2
Failure Reason:

Command failed on smithi094 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385766 2023-08-31 16:24:50 2023-08-31 18:29:08 2023-08-31 18:38:45 0:09:37 smithi main centos 9.stream fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
fail 7385767 2023-08-31 16:24:50 2023-08-31 18:31:29 2023-08-31 18:39:18 0:07:49 smithi main ubuntu 22.04 fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} 2
Failure Reason:

Host key for server 'smithi040.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIEilx+FPja+Zv7c7MDm8VeqUSgQ2KvYLlYI7U95ozla4', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDRdkyFa7Qw5ReQ+NZ/gigXfKrHP6vRTL1pKXdZyQDToWNDgsJZd1aDu57Hn8IhPAsojDReK5y4Wej5/rzEMaYvlqi3O6eERvujm1tVdlyJlsOzFrG5G7oLZajcnHoaucS+C374Jjoq2ObbTRFWriVVlG31ZqOuPr1mNoqMNEaiLSCn4wfuo4EM+8t2C8/VqFpfqnd42VKcSg9nAU/X9W9aPi/DgBsFxNeT+YkZ3zQ7HvBh1fORx3UEL37vlCYBMI2CUYcLjoqzPbfJzApIDHVjptpAYQ4bkdbCXY3Y6Fg74xpysEXLnSwqTETcfRvG8g7jx3bqvOGyJAglPPuwYTPzyR04UskQBkAM6+oDqXpxRIbfptRSdbsKU88IoI3DgIXWzH4AFpQKQSwHZRZ9l+ujoobIeJpfrjUMAC7rZdzRIJLGSQpkRLuedKDpAP/BvleQMoAYewOGuquhBB78ICls6f1uEFbOIRbCcChRN8cqSYQBOolaUPHDjv3PMO7m+zc='

fail 7385768 2023-08-31 16:24:51 2023-08-31 18:31:30 2023-08-31 19:13:39 0:42:09 0:26:09 0:16:00 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

Command failed on smithi087 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385769 2023-08-31 16:24:52 2023-08-31 18:32:20 2023-08-31 18:57:47 0:25:27 0:16:30 0:08:57 smithi main centos 9.stream fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
Failure Reason:

"2023-08-31T18:51:10.084918+0000 mon.a (mon.0) 226 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

pass 7385770 2023-08-31 16:24:53 2023-08-31 18:32:50 2023-08-31 19:22:35 0:49:45 0:39:00 0:10:45 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} 3
fail 7385771 2023-08-31 16:24:54 2023-08-31 18:41:10 2023-08-31 18:49:49 0:08:39 smithi main ubuntu 22.04 fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} 2
Failure Reason:

Host key for server 'smithi039.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILVUmDOxSDXpmUjPmXbdGPqfTZDS2NCe/XzdvJWcGXJ+', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDRvIrRQYC4QhHeTo9oPoaSk/G5/IF/ntU2TCjQet4aABbjWAPrLmtuz8WMv6JAQBIPNJzp5tX9GMwGhE/dnvNaHdC03ZkbIIpARTvoR0D9QWhM022uRWMfeWLS9Umiz5zxrqY+wnmbd68DRY2eW2y9qho5tME6w7qWfBd0Z11VvURhYSQOOSqW9CPgb/HOcEMPXfgW3PErCa1l22Tiq5q4ur/dxmumrqe1mYLs8hzbDo8Wtc2KQobhDtKFkxDA0wolW7G7XGt5mqBaLrLqYjQ3MPzSIpi7XMbvUGLV7eQoXhMY/2b0f1QB80re41LpmBkDcwU2syGcHwoj5+kyjly8+A1FLkQ7DkmQuryI8ck20DdqJxxhtDUcvSACMKLOOG6EXnNB42VqRKrYpOr9weqZOs2C1pGZ3YISpz2f+aqiO7mVCiVaV1SFRhBu9tlf7Z4dZ9upybH0DwLhsz411gQNFchUpWB0T+ofSR5xV5fj4QgiCqlk7OVJ+H2aTtm13pk='

fail 7385772 2023-08-31 16:24:54 2023-08-31 18:41:10 2023-08-31 18:51:36 0:10:26 smithi main ubuntu 22.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} 2
Failure Reason:

Host key for server 'smithi006.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIP5u3vyR0pPa+2+b76h1+f606vWFg392AeD4RMOhAZ6W', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDSvWYbqj+2rJRSiXk9+ASdhBdjS5QQMm6pu3HWomDUvPwdfxonVZUkJFAn87XVyINeGn41gtMzmND6U8OrguFV3fAlev42w+2L1UqrteVfVFmfdet7Rmsc05sHCwU4gQClkbDwUjy6Hctaj/wRMW7Nkfvp3HkPpx99TTZ02ltwQrTVB+2hAaa/FH3ytj4MbZT4EIyndI/cdVhyVKQmP5EdkyJTHv5h18M4HmCKu5gb1w8iMxkHadIffTf8ZmdY1K26jXckgtv4MP3aw7YsVM8V4raxs26FJ81ftr/qkBg5IDkFFiGyUQGrh4o5VKKhkZFi9ztZysfASdnv19T0cZPN0wwgYlMyo5DiZY7TFQWybQtQtDwtMgM6WPJQRj+gQjGK7jpqMqr2s1DNi5b+fHNu4PnljZWUSIfQIyqfXE9fG/CX9/jeGM2Cz8uxjVqFSp3zncaTwOH2Wm2X0bhRonNlhehu56MkgwJt/OmiH/hKkP4Lu0fo+E8d0YcdCFsHlt0='

fail 7385773 2023-08-31 16:24:55 2023-08-31 18:43:11 2023-08-31 18:51:49 0:08:38 smithi main centos 9.stream fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}} 3
fail 7385774 2023-08-31 16:24:56 2023-08-31 18:43:41 2023-08-31 18:51:22 0:07:41 smithi main ubuntu 22.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} 5
Failure Reason:

Host key for server 'smithi053.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIK234+DKAmG6EU4bwcJOPzPZtfFnTFxKfhZ/87klx9Fm', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDE3/etabK62/0YsVnvhgsjs/r9EW9yJeSFetiQQqtA2c25SPrmuNm7WEqdBaQxn7zSy12dPxDcMl+o3cYbVNV9ZVoiWQ8F5S6/G5l4dMBMxzpeb1L7YGm3uHauSk4Dttsjb0veoMOBhKPzTSdVgdB1uDLmEXvnQ5Va7oRJ/z0xWnr0aEL3A/JLFZXcCqIgsFV2q8ZDShPCouhXLhHKqziOkTQ3lQAAhUdg8ix3piJwHuSeP6zK9NSwkHKPZ0nR0SCAdsFs936CzlvVzjjEm01nRPJ5GUWyDSPUBfobDBFMHzzrw9Y5nSHJjPf2cncIlIjiKIRfVhV72j7VdZPdsGm5av+jwJYVyNo8xwPhvRIKI1UtWJgl8E1qDWnU5+5OzahHDeXZ8rLyykb8DUs8b5Bc3lERUKCXiXTQne3yiYf9QIDvFOv+5JsVu4OJ5POIYMp0bmbSdrZVxMe0Iwtf5BpEJMpjkgBCP3S1pihKCPeCQiJRPZzcKh22/X/xhsvIcjM='

dead 7385775 2023-08-31 16:24:57 2023-08-31 18:43:42 2023-09-01 06:52:34 12:08:52 smithi main rhel 8.6 fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} 2
Failure Reason:

hit max job timeout

fail 7385776 2023-08-31 16:24:58 2023-08-31 18:43:42 2023-08-31 18:51:35 0:07:53 smithi main centos 9.stream fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
fail 7385777 2023-08-31 16:24:58 2023-08-31 18:44:03 2023-08-31 19:27:01 0:42:58 0:26:32 0:16:26 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

Command failed on smithi012 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

pass 7385778 2023-08-31 16:24:59 2023-08-31 18:44:03 2023-08-31 19:17:01 0:32:58 0:27:12 0:05:46 smithi main rhel 8.6 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} 2
fail 7385779 2023-08-31 16:25:00 2023-08-31 18:44:04 2023-08-31 18:52:32 0:08:28 smithi main centos 9.stream fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
fail 7385780 2023-08-31 16:25:01 2023-08-31 18:45:24 2023-08-31 19:08:37 0:23:13 0:12:24 0:10:49 smithi main centos 9.stream fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} 2
Failure Reason:

Command failed (workunit test fs/test_python.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'

fail 7385781 2023-08-31 16:25:02 2023-08-31 18:45:34 2023-08-31 19:34:54 0:49:20 0:31:45 0:17:35 smithi main ubuntu 22.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} 2
Failure Reason:

Test failure: test_snapshot_remove (tasks.cephfs.test_strays.TestStrays)

fail 7385782 2023-08-31 16:25:03 2023-08-31 18:46:05 2023-08-31 19:22:14 0:36:09 0:24:46 0:11:23 smithi main ubuntu 20.04 fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} 2
Failure Reason:

saw valgrind issues

fail 7385783 2023-08-31 16:25:03 2023-08-31 18:46:05 2023-08-31 19:27:57 0:41:52 0:26:50 0:15:02 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

Command failed on smithi040 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385784 2023-08-31 16:25:04 2023-08-31 18:46:56 2023-08-31 18:55:22 0:08:26 smithi main ubuntu 22.04 fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} 2
Failure Reason:

Host key for server 'smithi181.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIKP+YUeq+szP5RhxQUo+wFX+38voGCwwhPMYU1VgjHQO', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDRlbWQIXAJ+6BdnF8RzS6zN5dJTJUdd9TCVjPiaP2P6JvN1X2+zZ7gtVeDh7uzkzV27qLSQ+q/CUSKt9wcgIIJCA8Fq14tp6uZl+nJlv9WpZk69xF4sCFa+hJx2TVTLiy8x1w6I2D4TDCTbaTLQpl6Grd/NI0O+IQ9KBV4LJJt15q15EbCtcH7qciS6bjFRjwyq6wxdbBt6ZgXaL9IGCgqVDbpwBJTkxrL67qNfrp4BX/pWCPDyII56RVlM8c5+Sh0wnsvg+R2jDHd+4xGtGMIEmpufhaxIwON8LUQIukqlrSvrgiQKJDPOssKOfmhbXuzOTsDq2dUt8zECB9VFSg/fl3YMd0lTxTJyuAs9L+QJoE7KrYRvxlUM874qYUpVT4ADw4u+2ftXrGld7VdhFZG4ssp2aJr9DW89Z6ir/nZi/6tzEocEFqISY7ncv8hTQfOoZgXlaxkFwxzAfDtHjh8wSviRYKdGtV/2b3/nMls6/q2qq1p5yGyaBe4dG5F5gE='

fail 7385785 2023-08-31 16:25:05 2023-08-31 18:47:46 2023-08-31 18:57:06 0:09:20 smithi main centos 9.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} 2
fail 7385786 2023-08-31 16:25:06 2023-08-31 18:49:37 2023-08-31 18:58:38 0:09:01 smithi main ubuntu 22.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} 4
Failure Reason:

Host key for server 'smithi138.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIHCAubYhee5fRENuM1NddpCLTwWJMYW98idIokDIeE2l', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCk/Vs+scxYc9dKvsc9FiDi8PExUbbqiczaEvJ1zjcfjPZPQg2QkaLQRJlRpojwXB9U350+U+Z6h4aRat/gPUP/6srydLk5nm9tqC6+Lalq0ZCQk11OkvkfrBgFafgzYWSlJ7OZEp8ciDiwsd2noRKAvEbEPyjq+bhxTKrjYEjWQzXQmRKyQZY6tUMqw5SrtAdLCrskz6SE2mYxYhQlwflKsBgX+UHF2IiyEo7DxSgGKU6jnGZ3gxOVK5yAFypCbH9lZLxDvky7ZKptj0fegZEf8eUq0IA1VjMDpILr7kzwhipBRAPzZd0196eolwsy6RsEZwVlm3+HgxtXbKUAHU0PtQAUFJhOLbq5IEumfwQbv+CDV80IzgGmf2k7BVwLRqcYh0YAkQPit+bIHvSDyGSsfI9tXoJImhep9p9LgmNV1JHCo0t3Hi+TfkY6YdWcFAXXt161Dn5wlaK4bkMeFngAsiJCSYXSllPauIJx3OnD0A1qv6RpVhFHSbQeCwqMOk0='

fail 7385787 2023-08-31 16:25:07 2023-08-31 18:53:32 2023-08-31 19:00:23 0:06:51 smithi main centos 9.stream fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} 2
fail 7385788 2023-08-31 16:25:07 2023-08-31 18:53:32 2023-08-31 19:01:09 0:07:37 smithi main ubuntu 22.04 fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

Host key for server 'smithi022.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIInSh7EpaRKGIrgUF4W9b/3cfi6VRsd7y80D+EfJFSaD', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDEmz7TnZnnnyaax2fptOqvotowYD0f1z+9c27TkjMBZYdgs/yuX85JP+Y1k7nuHVBWRZ1xqqfeX+mC5qUPoGkWwrFm1kOwn6T8YhIXm/nPy6nqqC34Hqy6v0nOmDeDQiPCbjIAuqqxU7MZ0euQV1xqjp6n3EtpwrUnpVZgmEQy/nSSxk0WZZdR0lclQ1w/A40izkLYAaQ14QdUqzbxwmwa//egDnh8qUzpfEnGJ5OWAOAxdSxiBaiCVjN/FDd6r6K3LacrDF1YWEXqFg+3MmMnuLCJJjcHgp6lYVyNCq0rJyqdjzVASIXu1xMCOmDfD7qYMpIskxvWz/mjW4Phmb/7biilaAlA7AO5Iw6ewG2a4iK7NWWbe4KdztAQt5rOPfUed2e9knv07VR1O13yMfO/7VfofvjPNLNOec/VjcwXa1cD6WPMP13v9PiAYMdq2WyJ2AJLgPPqefHF1GhNZmOS2NeA0GSP1j5TT20SVT4q5yyEyV0slgWfXDw6oOZO+80='

dead 7385789 2023-08-31 16:25:08 2023-08-31 18:53:32 2023-09-01 07:09:59 12:16:27 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} 3
Failure Reason:

hit max job timeout

dead 7385790 2023-08-31 16:25:09 2023-08-31 18:53:33 2023-09-01 07:04:08 12:10:35 smithi main rhel 8.6 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} 2
Failure Reason:

hit max job timeout

fail 7385791 2023-08-31 16:25:10 2023-08-31 18:53:33 2023-08-31 19:34:35 0:41:02 0:27:15 0:13:47 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

Command failed on smithi053 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385792 2023-08-31 16:25:11 2023-08-31 18:53:33 2023-08-31 19:00:48 0:07:15 smithi main centos 9.stream fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
fail 7385793 2023-08-31 16:25:11 2023-08-31 18:53:34 2023-08-31 19:01:14 0:07:40 smithi main centos 9.stream fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
fail 7385794 2023-08-31 16:25:12 2023-08-31 18:53:34 2023-08-31 19:17:13 0:23:39 0:15:39 0:08:00 smithi main rhel 8.6 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} 2
Failure Reason:

Command failed (workunit test fs/quota/quota.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2 CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh'

pass 7385795 2023-08-31 16:25:13 2023-08-31 18:54:05 2023-08-31 19:46:45 0:52:40 0:41:47 0:10:53 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} 3
fail 7385796 2023-08-31 16:25:14 2023-08-31 18:55:25 2023-08-31 19:03:26 0:08:01 smithi main ubuntu 22.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} 2
Failure Reason:

Host key for server 'smithi098.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIAcNfltreHCFkT3rx4tfIVepEtck0V35rZ9/vfsVOZQU', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCj5mhDdXQvkY2aZ3RQbRWhp7uU0gSVRersMVQh9Ht0+kfvOB4J6d9d3FwCSs2uqpflMjNZOMh0i13y2gT0vHxV9i8KZNga0BqYDqXDBY8CXp6C7yZ58UjA+fiR2gAxuD1Sv4UbLHrRhxrnhs/Y3NLm+ZiXGHCkenr3cRFnk+27CZic2BPu84eGeFYRQBTZSwD43LUVln4M1YUFckPQLzCK9anguoq1Mm2kLYdVizNAJdEPtLLXVtos24LGN+q4xCtJTyKn/12xZ6Nnsyg8Z40e+39cQcTFgFtaW0kFhsYnXUFEcG/FO8GEZ5bqpXNJk5TDO8hscrre6jVkXCbfwWEVjF05zgnt783+bSTuCABATnJbGzsycJeu1H9fFco6TBM9vpIGSsmLIwBKKRVHr0RbBQp3Ng76TfNIbk4MVfB++e825qMqUtRXn6uuodG4fKONTHjASGlQg2NnK3jtvwyipwvwtOGi7ahr8MDl2kZ7uyrDiNcxD3BEeuccaUzKNKk='

fail 7385797 2023-08-31 16:25:15 2023-08-31 18:55:26 2023-08-31 19:02:18 0:06:52 smithi main centos 9.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} 2
fail 7385798 2023-08-31 16:25:15 2023-08-31 18:55:26 2023-08-31 19:36:10 0:40:44 0:24:14 0:16:30 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

Command failed on smithi079 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

pass 7385799 2023-08-31 16:25:16 2023-08-31 18:56:57 2023-08-31 19:54:27 0:57:30 0:48:14 0:09:16 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} 3
fail 7385800 2023-08-31 16:25:17 2023-08-31 18:56:57 2023-08-31 19:05:00 0:08:03 smithi main ubuntu 22.04 fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
Failure Reason:

Host key for server 'smithi139.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAINdtGK+uES/7R8RbYRPnhDofx0ekRlZltxC0aQP3JwPh', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDZeUsf6mEAG2dxGx2PIVTJdhnDQ/sbqeiUvSA+dCdBZcqJEyNrwV3UnZYGeYsNbjgIKCfHy7AVe5k0QE1D1LPwoAlLFSAJkXqSHTPZfLkhVaHAPxoXJ54HA/qw/i2CIko9I9Ygb+pYOOMvn0qt5NIuF78agWsKJgGi6ZNRozagL8eS5VjdWuUJb/rHF4dh7+PRIH+xNnV1dA4zcVHyNQGZrZHszeOW4mU3EDbIhxoR3jBhPsfCkmfKnaowGrNL/oT9UJBMtNoI2f0IBHsL1N7aK3hS9IXt1SMBbS5aJNxiEpXlQhDIfJ63JslY/aYiwBwfI7mHqx7jh+tRVC4DhVmmqyyNdHUjomn9SAdfCvXd478B5NRBtH+yo+CGzkOMY15fDMDHhzDFGtYfKaE47UCacdDFElzTeeNMZCHsDDyAactOsIr28WaOv5xv+bV9ZbOZO97lxhYJPCCyFowFrG4Zcr6W+DGEoaNASEsjNAdUC3AfgRuuz6NVeOXFArg5lNs='

fail 7385801 2023-08-31 16:25:18 2023-08-31 18:57:07 2023-08-31 19:04:45 0:07:38 smithi main centos 9.stream fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} 2
pass 7385802 2023-08-31 16:25:19 2023-08-31 18:57:58 2023-08-31 19:27:37 0:29:39 0:16:31 0:13:08 smithi main ubuntu 20.04 fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
fail 7385803 2023-08-31 16:25:19 2023-08-31 18:58:48 2023-08-31 19:05:45 0:06:57 smithi main centos 9.stream fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} 1
fail 7385804 2023-08-31 16:25:20 2023-08-31 18:58:49 2023-08-31 19:06:22 0:07:33 smithi main ubuntu 22.04 fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} 1
Failure Reason:

Host key for server 'smithi138.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILYVxOzxcSmp6jEtj9JZlPUJfCPOhw273OR0kIqiWUPt', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCzziTVYzyd3/2BlCMhFA1faC8VSweAsRyBInCOhQyxZOdDusj3+yB/awCBvZA0+dgqyrJ3CPdc0CSmpwcIyDIt0Xl3Sb7aOsKrePSsPwt7ZDrqAa7doIGKa8WNoyEFF0BMhutSFBV/i0pVBnHUj3U/1XG9R2ilJaGypq0j3IKR7p2ShLKCm90e7pE2iJLc3uw+y64okr3ZOc3XCu1jPx5Z7ypn5QNmo5MVk3q4aK8kLGsR5nJ1ZKFP6kUoM/CCJ6qqGWz5DGB2wVuVuNEzDFQKzDQRrLcQknUcQEeS4um/Oy432nXn2Ng1rbAo2z/eZT/56TPtxTib+dPoHwJoRSActdOwOUOF3Qw6PkDVYvJ50YEXGyFWi3dLAEsmxRrEn8XF2kaOCG9rwIwqBN0IJ/wl5ZadBoAr8QHSMpeCxQD00j3IjB8Js7FyYMxC8Adkl/JsUw0i2m9OM+goV5Py0FuVTTgVS8tYgZFcwwE6Hw5a00mibsqD4vp/q+nEicEWZ7M='

fail 7385805 2023-08-31 16:25:21 2023-08-31 18:58:49 2023-08-31 19:07:56 0:09:07 smithi main ubuntu 22.04 fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} 2
Failure Reason:

Host key for server 'smithi114.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIEz5Xqq4aQNhjfUOw2pUCXWQUdCNqS5P5de/2TF/jrNe', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCt26F9VrhdZZif6fAuLU9AM3UUju4Hh9J1pQzvTxSEdikYS1W/tiMixZoFNlFZC2NrMRToHwIZ0ODiOaDZdAayibgEkOHV8hpOFohzoWfiLCvOVywUA2Df5nfTd79IIFV1Om365RszeTEkIuKNpAcXDrL6z9OjH7SZjJNiD8tjlhmjZBGrEikInBsVRDf+vTJWH1Dan25QrYxGBY5aip8nnqxRNnIONZYdMLX8tglkMlOpel8knWmkdC+ZaJZ4lFDPKn4NniwdHisNa392CRxQ2JoUz8Yfm2pRyoaZ3Yb3buCq9R27ZZoV6YY4AOJVOS9ZfmSSvzR9ki14+Y0A3AbzSTrrOPgo0BPVVnd1Co/ZHk2Ea3k1k1rxviA+bUe1AVPxioIRE3GcISWqr5NQ2BjQRvTGpwW+3GmUWP9X+eqCO6/cIOtcUNIR37WThfj8gdMkzulrfKpPjaO/XVjSX2yqPDq6TEOMNhSIidjxaut+loNaDZMH7onHYSxqoJh1tbk='

fail 7385806 2023-08-31 16:25:22 2023-08-31 19:00:30 2023-08-31 19:10:03 0:09:33 smithi main ubuntu 22.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} 5
Failure Reason:

Host key for server 'smithi022.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIO3S93IzQbwSip/ubhFIOZRH9a2C5nkTm5wa8uL72bMd', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC3H2sgU5DmZvXJ7qyVSvCoARHSQHg2xk0tnbrHqYjs8GCuC4GPFr0sRUbLaugOTPNtIpLHD6bStWTNtnHrdbpL0PS6+OSpQIBEo2NIzhBux++DWGtYdU+BYT1SetsnVnDjN1SQyt+Z1IQEa7KjK6vF04CJy79rQeosQRXoBO89ykEAqspr3KtnvfCvdjJlv2//bOU52h3cURYK9xK3NYG+yHyjGKdPOkn5fJRTgXvA2P30qdQOzoICY11H5RipFcMsZIcVwTQe1QtxeK3TIi1hIDD2OAWBdLKTEnXFIcwFLSm9oA9JeLxyI+0nlVcLn9rFLIBWTuGiOT6qTlThYrDbMwpfkH3szSEQiUDyX9ug7xh9nTRsldvYOSlXfjGWbqNXGO0tFQIw4jw0gh+DmQbnE+DeEJJxrld1wBNTFgjAYXKcPwbZPbvDN/DaAlBB+h1Pne9m94We1lgcF6R5DxyJ75aHUqhfUir7iimHiS8bspxOiTzv3GdngWZNyzxoUik='

fail 7385807 2023-08-31 16:25:23 2023-08-31 19:01:20 2023-08-31 19:09:34 0:08:14 smithi main centos 9.stream fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} 2
fail 7385808 2023-08-31 16:25:23 2023-08-31 19:02:21 2023-08-31 19:09:24 0:07:03 smithi main centos 9.stream fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
fail 7385809 2023-08-31 16:25:24 2023-08-31 19:02:31 2023-08-31 19:10:12 0:07:41 smithi main ubuntu 22.04 fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

Host key for server 'smithi148.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIC7s4UxZ/jRydnBvHYCNTscWaVg9ZgZXNn0xX7DmjrmS', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCWOm8GU0XieDcsLViEDe029DtlDts7ORo31WmZwcictF8wM+mCPqI2RoEjglzTEvvwij57xu/lyF0O+WF1MJIilvFBN1trBmsFfdZkSHJMVenYuYWyr2D3mpuGIQKGiXZ3B9iMeV2arPkiyEchqDSCeiXb3EoMf4kB7b3vFx+2V4AfJ+PydK3kRLIFlDTMXp4IYVHp1w8YpsxoSJBeay8fPKvDCPEoSbkcQ9TWO5XfZPJGXS4qx3E266Kby5lfYgl6phLzorF6YMws7EwyXmKDimKDJaQl3sx1wabt8ljmb+8LcuOjOf8tBugHctMrfr8Bk6HBn3LoMx7fSIBhuYTaEUJfUTq+DHQkca08gBqQBxKM//IDL3E8YQJlycmLYLseX8vKibt16vgGX8b6H9ZwXoO0h39HOYEFTQlJQKF8Ryj7HkFr6753qhnKCWuU/dfDp2mDxY5G4P5Q0FWDhSMJ1iXgze0BO2W9iBL9EEmSKxbbjmD3KZDHj1e7tTF1FlU='

fail 7385810 2023-08-31 16:25:25 2023-08-31 19:02:31 2023-08-31 19:28:06 0:25:35 0:15:20 0:10:15 smithi main ubuntu 22.04 fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} 1
Failure Reason:

"2023-08-31T19:21:53.856416+0000 mon.a (mon.0) 108 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7385811 2023-08-31 16:25:26 2023-08-31 19:02:42 2023-08-31 21:15:31 2:12:49 2:03:56 0:08:53 smithi main centos 9.stream fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} 1
Failure Reason:

"2023-08-31T19:24:13.947263+0000 mon.a (mon.0) 87 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7385812 2023-08-31 16:25:26 2023-08-31 19:02:42 2023-08-31 19:55:56 0:53:14 0:34:36 0:18:38 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

Command failed on smithi027 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

pass 7385813 2023-08-31 16:25:27 2023-08-31 19:03:33 2023-08-31 19:46:44 0:43:11 0:36:36 0:06:35 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} 3
fail 7385814 2023-08-31 16:25:28 2023-08-31 19:03:33 2023-08-31 19:10:23 0:06:50 smithi main centos 9.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} 2
fail 7385815 2023-08-31 16:25:29 2023-08-31 19:03:34 2023-08-31 19:11:49 0:08:15 smithi main centos 9.stream fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}} 3
fail 7385816 2023-08-31 16:25:30 2023-08-31 19:04:44 2023-08-31 19:11:53 0:07:09 smithi main centos 9.stream fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} 2
fail 7385817 2023-08-31 16:25:30 2023-08-31 19:04:54 2023-08-31 19:53:10 0:48:16 0:27:41 0:20:35 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

Command failed on smithi002 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385818 2023-08-31 16:25:31 2023-08-31 19:05:45 2023-08-31 19:12:54 0:07:09 smithi main centos 9.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} 2
fail 7385819 2023-08-31 16:25:32 2023-08-31 19:05:55 2023-08-31 19:44:06 0:38:11 0:26:17 0:11:54 smithi main ubuntu 20.04 fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} 2
Failure Reason:

saw valgrind issues

fail 7385820 2023-08-31 16:25:33 2023-08-31 19:06:06 2023-08-31 19:15:36 0:09:30 smithi main ubuntu 22.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} 4
Failure Reason:

Host key for server 'smithi094.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIItTZaeKCT147PXQsv0OtDBS+QrdaADSBftGNjK/D4Cz', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCTlsUhPnyqH3OaeM8NNllq+isIifotBi0dk8TkoSxoerOhhfIYbLbrNc5fvcRGhPjRplSGow++pe/EiWOBmG8ovYKtYQQWCf6CZdCLH0q00aPxi/T2zdtnzeD6HrVmT9MI1cXJ3SwKvnZ9IgSH1meJRqJsBMybcjHeAoXMKQtLfT7yAc//VOTtXPgQM/ecaPxBr9whfwZxmgta6aiRN4W57XMRJv/+tT/8MGz2SnZxo2nRax6kx8bJYeVajjnlJ7xVrmKYhfJFbz0qfs95SCuXeyWPyhVrtjRxd4pYbOY6Mrz4q1Knb8ZKSkysWYAQxTqJEJZAlMTxr/VehUr0J5JJgzQwWJyWX6G+211xHTkglbc5ZVjAXK1g+n81hSn85VSuHbHiTy7cEInkeR/hXWVjWbe8Vow2S+6HIu+JHxxzdGKV6bgQsm2nr3wj7SAzN0HOYivZo/kPTQuiVVIJEawnhy/8xEzz4BjzbC+gsjKEJiHyprmzs48uf56ryg8JO1E='

pass 7385821 2023-08-31 16:25:33 2023-08-31 19:08:06 2023-08-31 20:30:04 1:21:58 1:15:40 0:06:18 smithi main rhel 8.6 fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} 2
fail 7385822 2023-08-31 16:25:34 2023-08-31 19:08:47 2023-08-31 20:06:30 0:57:43 0:34:30 0:23:13 smithi main centos 8.stream fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} 2
Failure Reason:

Command failed on smithi082 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385823 2023-08-31 16:25:35 2023-08-31 19:09:27 2023-08-31 20:00:31 0:51:04 0:45:19 0:05:45 smithi main rhel 8.6 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} 2
Failure Reason:

Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin)

fail 7385824 2023-08-31 16:25:36 2023-08-31 19:09:38 2023-08-31 19:17:11 0:07:33 smithi main centos 9.stream fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} 2
fail 7385825 2023-08-31 16:25:37 2023-08-31 19:10:08 2023-08-31 20:03:29 0:53:21 0:35:47 0:17:34 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

Command failed on smithi029 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385826 2023-08-31 16:25:37 2023-08-31 19:10:09 2023-08-31 19:57:55 0:47:46 0:37:42 0:10:04 smithi main ubuntu 22.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} 2
Failure Reason:

"2023-08-31T19:30:21.561588+0000 mon.a (mon.0) 157 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7385827 2023-08-31 16:25:38 2023-08-31 19:10:19 2023-08-31 19:17:16 0:06:57 smithi main centos 9.stream fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
fail 7385828 2023-08-31 16:25:39 2023-08-31 19:10:29 2023-08-31 19:18:31 0:08:02 smithi main centos 9.stream fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
fail 7385829 2023-08-31 16:25:40 2023-08-31 19:11:30 2023-08-31 19:18:46 0:07:16 smithi main centos 9.stream fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
pass 7385830 2023-08-31 16:25:41 2023-08-31 19:11:30 2023-08-31 20:56:57 1:45:27 1:39:02 0:06:25 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} 3
fail 7385831 2023-08-31 16:25:41 2023-08-31 19:11:51 2023-08-31 19:59:55 0:48:04 0:24:59 0:23:05 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

Command failed on smithi003 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385832 2023-08-31 16:25:42 2023-08-31 19:12:31 2023-08-31 19:20:05 0:07:34 smithi main ubuntu 22.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
Failure Reason:

Host key for server 'smithi055.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIPPxG4D4Td7ufK5hc1yERQ4pLHi5nns67Lr9jb1cJ+SS', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCwabvKPS7tAGrd5ZkD33G59UHJ+1Is67W6Lm8hv98d0W553BEnICD+wnyalvU51Kzr19pmRFuRPmOpocK3vTOpI81P92tOIKRRlfkHJ29d1uNTAmItS2X1ke1P9SkJ6/wPh/JSX1lCwZt7dWIWABzfloctqq23OVs7BBEQWza1sPmDSItbu2CsrZXyxFJBTL0apNwu8dEj6yPscMTKwWbUbJvE6PZyeUpW9YDu9jZunAGr30SU+h5+2znx2VmzgRCYH+K7b/PiqgH2qlEulj2wADWMt5RSol7+Lhtzh8c6vQn8ROE72MEW8e93Mhfcg22t98/37RGwTicMa9b0YOR85sUs9pRAZAeLGqpmuAUkXsnDgTH6XkiMAXWfTonfUmLI0R/W33hG6pKSGNcrOxLaE4aSVzPdz3r0QvFbpYF/lQgIwZjjYF/G09htPu7rvu2XdLJ4m62fnWwQVBbtiAzcyoBRRhmifOKTibVE4DbmvP8KRrq1ykt1zIa/pFfoOk8='

fail 7385833 2023-08-31 16:25:43 2023-08-31 19:12:32 2023-08-31 19:20:39 0:08:07 smithi main ubuntu 22.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} 2
Failure Reason:

Host key for server 'smithi139.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAILkAfjP3osPysiTPOFivMHFW/pdr5hVioGAFm8skYUi+', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC+cmVs+nH/xEr1czf38ZwS6SbnKZunSfx0TBF2rH+h5xNxpjhgHbPkx7lAD5E2VNErSUPlYs8tHXxTBd/YbX6jAwrXFGdlp/cMojzAiO3PQYDijFOTf3FCLRHqe+R/GNPgufuwOcZFln1NbQMM/nLvdXc+7b/1GrblkLxD9drG7Dfue9U/nBe/7g79eY8i1QI7HXZqTTQhDNzmm4ZjPinWDB2DQOL1t0FfrDkIVgzM7wYN2lNyKowbeMl1EEBFmbPKt6HxpB0Ig0fEh/2NtqFfrB7bGc2vHoE0F/84Hj62VScTTBoBN2XHF6FE8gnSMoP8Rd3ywLJZNLY+mtY1bY+R5WbLGWP2J3/l0XX5A4F4N3GbJiphgasQ78dP4strWRVnFMUnVgGR7kWT+SX8g99iTFr9Bp5N93HRUbas8tMCCM89L1sNDIO7G7rv4+TZn5JV/5Z6YtJAqS35vJYGmHQ3VFY0NnzVgclXabeMiUB+ML24jtbjAhcWYitZO/MNy6U='

pass 7385834 2023-08-31 16:25:44 2023-08-31 19:13:02 2023-08-31 19:58:26 0:45:24 0:36:15 0:09:09 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} 3
pass 7385835 2023-08-31 16:25:45 2023-08-31 19:14:53 2023-08-31 19:38:43 0:23:50 0:17:14 0:06:36 smithi main rhel 8.6 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} 2
fail 7385836 2023-08-31 16:25:45 2023-08-31 19:15:43 2023-08-31 19:24:39 0:08:56 smithi main ubuntu 22.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} 5
Failure Reason:

Host key for server 'smithi007.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAICWWRtGBG7Nlk1RUzeDaF1O/gZmxBQQG5EtmgFn2lak3', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCvOmhgI1Yrw62u7NN15KtYynlvNUr0jebqCTwM6cOLMwufMYEeVxIKzhwjA4t6j/p4qwXtz5B9Ytq7nxNhdmD1xIgUvyPmC8u2AlqQdgb1yex/zaZmMUAYy6hjetRvdfvVUtwxWRCJfjAhvueXdWY3L4NsFMTcG3eRhmxi1xPQsjx27tMX/u8xeCTH9Vd9g5F+8Ysjf+bVKNS2NYa+ZP+pg1VCmvYmmvobrFtahW0xAE0d4pO1sBAHR8t5a957iPRSvghbjNV3gr+s+C5k/ky/sQMfuVmaJah1cAninOFTz+Y0NfQg9eBCYzNcuFliwOPJaC+S+jSen1VFT1paI9bHTWWomFBcOm9j4vcRfYmBqMeNw23c2cVIvKUX7V9D4TUhwLERjIZhnO201lJLcSaxrkYiqlDO6ZDAnJzk49CRfcFWnUu3UuARk/wcsRr6HWS9ItcJ6qxqN9lMwbZhBpdH+BZ/aTjERFbsji6lBxemt3heVT74GLSfbOLWU34LcBU='

fail 7385837 2023-08-31 16:25:46 2023-08-31 19:16:04 2023-08-31 19:24:37 0:08:33 smithi main centos 9.stream fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} 2
fail 7385838 2023-08-31 16:25:47 2023-08-31 19:16:34 2023-08-31 20:20:49 1:04:15 0:54:15 0:10:00 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} 3
Failure Reason:

Command failed (workunit test suites/dbench.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c085c84adafd662335e941b0b7b9c406c85078a9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

fail 7385839 2023-08-31 16:25:48 2023-08-31 19:17:05 2023-08-31 20:01:02 0:43:57 0:32:25 0:11:32 smithi main ubuntu 22.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} 2
Failure Reason:

"2023-08-31T19:39:10.483793+0000 mon.a (mon.0) 157 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

fail 7385840 2023-08-31 16:25:49 2023-08-31 19:17:15 2023-08-31 20:02:28 0:45:13 0:29:37 0:15:36 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

Command failed on smithi019 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385841 2023-08-31 16:25:50 2023-08-31 19:17:26 2023-08-31 19:25:28 0:08:02 smithi main centos 9.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} 2
pass 7385842 2023-08-31 16:25:50 2023-08-31 19:18:36 2023-08-31 20:12:17 0:53:41 0:47:11 0:06:30 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} 3
fail 7385843 2023-08-31 16:25:51 2023-08-31 19:18:57 2023-08-31 19:27:01 0:08:04 smithi main centos 9.stream fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 7385844 2023-08-31 16:25:52 2023-08-31 19:20:07 2023-08-31 19:45:55 0:25:48 0:16:29 0:09:19 smithi main rhel 8.6 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} 2
pass 7385845 2023-08-31 16:25:53 2023-08-31 19:20:28 2023-08-31 20:12:00 0:51:32 0:41:05 0:10:27 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} 3
fail 7385846 2023-08-31 16:25:53 2023-08-31 19:20:48 2023-08-31 19:29:10 0:08:22 smithi main centos 9.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} 2
fail 7385847 2023-08-31 16:25:54 2023-08-31 19:23:04 2023-08-31 19:30:34 0:07:30 smithi main ubuntu 22.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

Host key for server 'smithi129.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIAlcPvZVC9BTx38UBguSjbQvZRmQw3oA6UDG7JNK8+BQ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCkaJSQR9w9I08MpmGHzY4R2qyskhB9DDKHasc0xKP15DZGHES94wwMehgeD/VtZu2i9CXyi934zk39q3q1+YKGzhUKC7EXB0XDB+5xpY5DUjviaT60sn0TMF9BpmJZZl2dILfr3KDd6mHY1BQ0NrIzn9OZeVuTr+I3E0uGy/G085YT+kRGhzmEjEnn183Sth7kQ7QCVmdqRJMFqxWHzDjk4PvAo5+qdmLOYXthLZ71JWtjN1UqJjsJp8hk29hLENfuwQTMh0hNMTu9/y/iMmfViFWb4TWYpEDRj/4yMvTTX2ZoulObTuFf4PaISL7iIb3btlYOlOqA45bDjxtxWfPqxwSjTRAVQOpD5JcqYQJuQACUPB2LmdFPf5KD0cLayfFC/92a6VqXUMA4zSWu02seuOLxpqMtoZHc0SAiQYWWs3dNzTrjHdZrGSyf2/tDVBT1qya5Fnci2RWmGadVP//yIaQXZ2rp7xCbFxsK9tUcRDvFD94TCnfjI7gso1V/qJs='

fail 7385848 2023-08-31 16:25:55 2023-08-31 19:23:04 2023-08-31 19:44:43 0:21:39 0:14:52 0:06:47 smithi main rhel 8.6 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} 2
Failure Reason:

Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)

fail 7385849 2023-08-31 16:25:56 2023-08-31 19:23:35 2023-08-31 20:14:56 0:51:21 0:28:22 0:22:59 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

Command failed on smithi057 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385850 2023-08-31 16:25:57 2023-08-31 19:23:45 2023-08-31 20:16:33 0:52:48 0:41:28 0:11:20 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} 3
Failure Reason:

error during scrub thrashing: rank damage found: {'backtrace'}

fail 7385851 2023-08-31 16:25:58 2023-08-31 19:24:46 2023-08-31 20:17:44 0:52:58 0:28:59 0:23:59 smithi main centos 8.stream fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} 2
Failure Reason:

Command failed on smithi094 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"

fail 7385852 2023-08-31 16:25:58 2023-08-31 19:24:46 2023-08-31 19:33:19 0:08:33 smithi main ubuntu 22.04 fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} 3
Failure Reason:

Host key for server 'smithi194.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIKO7LOOAiVzwo50p7dqfOINcxrheQoDMiSvOQlyom0uD', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDg3T7S2XzDKWywt2nCebKeUVip1hiXN0AHVgonBSW/BG372QR5GAzvDa8PA2GVRp1TXTihaeuQscoYJUiAxoG3wXoaBlBFnVtvaA73R6glvRVGmC/3CreI2JCwitG8UkndurqOAd2owk8ex4ro3vH37Q0xhg8eYuvQmTgfhsGW0pjc7MOaYqbo/XgWBikEaQ0xSJ7365TqcIDihSz9b8/w2lWgLNHDOZVUISsYfNA2J65kYKNZEiNVLRUK8sPBWXM/cpXmfCXZhK3jRo9tHk2g8tguA9J5alvOfvJqXixxToGDu3UZOBF7C/mRmoY1AMM4Tl8h/VQiCabaNjgzeqbif0GCyTC3XvJjpG/qCtFMp4HF/5JZ56C+6+VcFYO0aMnmyXyC2ekD1J1iNrf+IVQQHL6Owe1xqPm3JXNTezn+Z4Ok3JLKjtAL19qN3i01rAE33CkMEm41Uj9Z9jwXS2eKg7aiwxMifieEgt2OuQH+5cnmYbyY2bV2Ez5IKQlCNOc='

fail 7385853 2023-08-31 16:25:59 2023-08-31 19:25:37 2023-08-31 19:33:27 0:07:50 smithi main centos 9.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} 2
pass 7385854 2023-08-31 16:26:00 2023-08-31 19:27:07 2023-08-31 20:20:20 0:53:13 0:41:14 0:11:59 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} 3
fail 7385855 2023-08-31 16:26:01 2023-08-31 19:27:08 2023-08-31 19:36:19 0:09:11 smithi main ubuntu 22.04 fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{test}} 2
Failure Reason:

Host key for server 'smithi039.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIMSR7pht4nCxqzO1T3+ecxhkWphfi6mHKmJIYOZqPz1b', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC5br7Zd4l6Y9BKt6q68NkvwSQcnvFHNFSBpRs3gvpSm0xvYHfRgLxYZpT0oYJmD1cepeywsCRv8YCAIQHAo0WBFWkXJdeKdL0LEdkNlCdnKza8io3OI8IrvNLDsOnFb7WsH+DXq0RQOjdiCFfw4LGKJhwQq06b6GVWGiGUVNMk2nflKrM+rAv0kSvs58s37TrpU7BQOih4iJMYrbD72BUVM/nURQa9Q+Ugzzp7dvgYP5Meq1UjFhvCAai0+tZlZCb8mvug1CM64GDMl4uL7quO+NRwEfdo/88++Z2AqEOsDwGgKB/hDBCPAQUJ4b/kMUzBYYcLccjDMBm4Bh7kZyFprX1Upq2DJ36DZ/d+AHwNrcxEQnOn0OCIQsEBiZFlFLGDYN1kl1+k2U8DSLhmtoTtjigiEF+gEvZTqfSZMcWeJ6jlNo4Ll3/gkT8dvgQLv9YJTDUuv1VEl8u7Nh9me28Z4iqjGQVG8XldpYAMQQ2e8E2tJhZf/ofIkJfEl3D/Nrk='

fail 7385856 2023-08-31 16:26:02 2023-08-31 19:27:38 2023-08-31 19:35:54 0:08:16 smithi main ubuntu 22.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} 4
Failure Reason:

Host key for server 'smithi117.front.sepia.ceph.com' does not match: got 'AAAAC3NzaC1lZDI1NTE5AAAAIIulQwZrBxlnlBQVcFuZLDV6bHdXaSx5Hptl+5wWVe4q', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDIxt5aAXYEPe0M+BayWVuVusnQAmevxw8p1bSsjnmkprRi1w+Q0n/6wn2vvOr6ndirvHB0jQlFcEwitsDp5TKxBGplWUPOcuQFJK5ZfehoJIby+IYgufX4n70Wd9pTP7JOT9eWYXyGLtQfk0+cgMXpEVvWKih3/e4k0TfsqDcVPb0vaa+W1Y8C5CpRfl1flqqt/gohUtY3l01KesKPlbbeLtBBFN5H6lvQFKncwE9CQ2qRO9PZBySD6nnjAlUjNsfj9uJH9MZ5OZThM9dA4E97fo46bWWZntQ6NJ7fpHdQa9Q9FjS+oQtwQ5SMUlXrxOuMI5OGJic1CrhY4157qGFeO5al1WiXE9LutjbV2dShQKsMc9mBtQ1wW40wkvfdQN2j+Sc5OqstbsIRC7lpmAWwUEbvoJ7JL8vhBqmlcoRlKbVhQMmhcxZu6PerLiNuoBPkF9IkQPBxGaHflXg33TrLu23Iaym+E5e2d0jQYv5sD+J1sFQ/YaZ17lOLCDH6BLM='

fail 7385857 2023-08-31 16:26:02 2023-08-31 19:28:09 2023-08-31 19:36:03 0:07:54 smithi main centos 9.stream fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} 2
fail 7385858 2023-08-31 16:26:03 2023-08-31 19:28:49 2023-08-31 19:36:03 0:07:14 smithi main centos 9.stream fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
fail 7385859 2023-08-31 16:26:04 2023-08-31 19:29:20 2023-08-31 19:37:19 0:07:59 smithi main centos 9.stream fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
fail 7385860 2023-08-31 16:26:05 2023-08-31 19:30:40 2023-08-31 20:38:29 1:07:49 0:40:43 0:27:06 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

Command failed on smithi162 with status 22: "sudo ceph --cluster ceph config set client 'client mount timeout' 600"