Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 5915793 2021-02-26 05:55:48 2021-02-26 09:50:38 2021-02-26 10:18:44 0:28:06 0:15:35 0:12:31 smithi master ubuntu 18.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
pass 5915794 2021-02-26 05:55:49 2021-02-26 09:50:38 2021-02-26 10:08:28 0:17:50 0:08:34 0:09:16 smithi master fs/bugs/client_trim_caps/{begin clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i22073} 1
pass 5915795 2021-02-26 05:55:49 2021-02-26 09:50:39 2021-02-26 10:53:35 1:02:56 0:51:10 0:11:46 smithi master centos 8.2 fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest} 2
pass 5915796 2021-02-26 05:55:50 2021-02-26 09:50:39 2021-02-26 10:11:51 0:21:12 0:12:00 0:09:12 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} 2
pass 5915797 2021-02-26 05:55:51 2021-02-26 09:50:40 2021-02-26 10:09:48 0:19:08 0:11:55 0:07:13 smithi master rhel 8.3 fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client} 2
pass 5915798 2021-02-26 05:55:52 2021-02-26 09:50:40 2021-02-26 10:51:32 1:00:52 0:53:05 0:07:47 smithi master ubuntu 18.04 fs/mirror/{begin cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} 1
pass 5915799 2021-02-26 05:55:53 2021-02-26 09:50:41 2021-02-26 11:33:51 1:43:10 1:35:19 0:07:51 smithi master rhel 8.3 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} 2
fail 5915800 2021-02-26 05:55:53 2021-02-26 09:50:41 2021-02-26 10:18:32 0:27:51 0:13:34 0:14:17 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} 4
Failure Reason:

Test failure: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc)

pass 5915801 2021-02-26 05:55:54 2021-02-26 09:50:42 2021-02-26 10:37:38 0:46:56 0:39:50 0:07:06 smithi master rhel 8.3 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} 2
pass 5915802 2021-02-26 05:55:55 2021-02-26 09:50:42 2021-02-26 10:13:59 0:23:17 0:13:43 0:09:34 smithi master centos 8.2 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
pass 5915803 2021-02-26 05:55:56 2021-02-26 09:50:43 2021-02-26 10:34:10 0:43:27 0:33:16 0:10:11 smithi master ubuntu 18.04 fs/shell/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell} 2
pass 5915804 2021-02-26 05:55:57 2021-02-26 09:50:43 2021-02-26 11:17:11 1:26:28 1:17:43 0:08:45 smithi master centos 8.2 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
pass 5915805 2021-02-26 05:55:57 2021-02-26 09:50:45 2021-02-26 12:39:52 2:49:07 2:42:57 0:06:10 smithi master rhel 8.3 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} 2
pass 5915806 2021-02-26 05:55:58 2021-02-26 09:50:45 2021-02-26 10:08:37 0:17:52 0:09:14 0:08:38 smithi master ubuntu 18.04 fs/top/{begin cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} 1
pass 5915807 2021-02-26 05:55:59 2021-02-26 09:50:46 2021-02-26 10:22:35 0:31:49 0:22:16 0:09:33 smithi master centos 8.2 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} 2
fail 5915808 2021-02-26 05:56:00 2021-02-26 09:50:47 2021-02-26 10:31:18 0:40:31 0:27:41 0:12:50 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

fail 5915809 2021-02-26 05:56:00 2021-02-26 09:52:17 2021-02-26 10:23:08 0:30:51 0:21:13 0:09:38 smithi master centos 8.2 fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} 1
Failure Reason:

Test failure: test_add_directory_path_normalization (tasks.cephfs.test_mirroring.TestMirroring)

pass 5915810 2021-02-26 05:56:01 2021-02-26 09:52:18 2021-02-26 11:02:55 1:10:37 0:58:26 0:12:11 smithi master centos 8.2 fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/lockdep} 2
pass 5915811 2021-02-26 05:56:02 2021-02-26 09:52:58 2021-02-26 10:53:32 1:00:34 0:48:46 0:11:48 smithi master ubuntu 18.04 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} 2
pass 5915812 2021-02-26 05:56:03 2021-02-26 09:53:39 2021-02-26 10:15:58 0:22:19 0:15:08 0:07:11 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
fail 5915813 2021-02-26 05:56:04 2021-02-26 09:54:50 2021-02-26 10:37:40 0:42:50 0:35:08 0:07:42 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
Failure Reason:

SSH connection to smithi068 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

pass 5915814 2021-02-26 05:56:04 2021-02-26 09:56:31 2021-02-26 11:31:50 1:35:19 1:27:39 0:07:40 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
pass 5915815 2021-02-26 05:56:05 2021-02-26 09:56:41 2021-02-26 10:45:18 0:48:37 0:36:38 0:11:59 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
pass 5915816 2021-02-26 05:56:06 2021-02-26 09:58:02 2021-02-26 10:29:11 0:31:09 0:24:26 0:06:43 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} 2
pass 5915817 2021-02-26 05:56:07 2021-02-26 09:58:03 2021-02-26 10:48:08 0:50:05 0:42:29 0:07:36 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} 2
pass 5915818 2021-02-26 05:56:08 2021-02-26 09:58:25 2021-02-26 10:42:43 0:44:18 0:33:54 0:10:24 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
pass 5915819 2021-02-26 05:56:09 2021-02-26 09:59:05 2021-02-26 10:33:19 0:34:14 0:22:03 0:12:11 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
pass 5915820 2021-02-26 05:56:09 2021-02-26 10:00:16 2021-02-26 11:03:34 1:03:18 0:49:14 0:14:04 smithi master ubuntu 18.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 5915821 2021-02-26 05:56:10 2021-02-26 10:01:47 2021-02-26 10:30:00 0:28:13 0:14:46 0:13:27 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} 2
fail 5915822 2021-02-26 05:56:11 2021-02-26 10:02:58 2021-02-26 10:47:57 0:44:59 0:36:05 0:08:54 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
Failure Reason:

SSH connection to smithi007 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

pass 5915823 2021-02-26 05:56:12 2021-02-26 10:04:39 2021-02-26 10:27:04 0:22:25 0:11:02 0:11:23 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} 2
pass 5915824 2021-02-26 05:56:12 2021-02-26 10:05:30 2021-02-26 10:35:04 0:29:34 0:17:00 0:12:34 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 5915825 2021-02-26 05:56:13 2021-02-26 10:07:41 2021-02-26 10:42:41 0:35:00 0:23:01 0:11:59 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} 2
pass 5915826 2021-02-26 05:56:14 2021-02-26 10:08:12 2021-02-26 10:32:30 0:24:18 0:12:27 0:11:51 smithi master ubuntu 18.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} 2
fail 5915827 2021-02-26 05:56:15 2021-02-26 10:08:42 2021-02-26 10:37:24 0:28:42 0:16:23 0:12:19 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
Failure Reason:

"2021-02-26T10:31:19.168257+0000 mds.e (mds.2) 2 : cluster [WRN] Scrub error on inode 0x10000001e80 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_posix_testsuite/stress) see mds.e log and `damage ls` output for details" in cluster log

pass 5915828 2021-02-26 05:56:16 2021-02-26 10:11:03 2021-02-26 10:33:20 0:22:17 0:13:56 0:08:21 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} 2
pass 5915829 2021-02-26 05:56:16 2021-02-26 10:11:54 2021-02-26 10:57:38 0:45:44 0:32:05 0:13:39 smithi master ubuntu 18.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
pass 5915830 2021-02-26 05:56:17 2021-02-26 10:14:05 2021-02-26 10:31:52 0:17:47 0:11:25 0:06:22 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} 2
fail 5915831 2021-02-26 05:56:18 2021-02-26 10:14:35 2021-02-26 10:55:51 0:41:16 0:35:15 0:06:01 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
Failure Reason:

SSH connection to smithi038 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

pass 5915832 2021-02-26 05:56:19 2021-02-26 10:14:56 2021-02-26 10:47:16 0:32:20 0:24:10 0:08:10 smithi master rhel 8.3 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} 2
pass 5915833 2021-02-26 05:56:19 2021-02-26 10:16:07 2021-02-26 10:51:45 0:35:38 0:24:26 0:11:12 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
pass 5915834 2021-02-26 05:56:20 2021-02-26 10:16:28 2021-02-26 10:39:41 0:23:13 0:11:39 0:11:34 smithi master centos 8.2 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 5915835 2021-02-26 05:56:21 2021-02-26 10:18:28 2021-02-26 10:35:58 0:17:30 0:10:18 0:07:12 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} 2
pass 5915836 2021-02-26 05:56:22 2021-02-26 10:18:29 2021-02-26 10:41:58 0:23:29 0:13:28 0:10:01 smithi master ubuntu 18.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 5915837 2021-02-26 05:56:23 2021-02-26 10:18:39 2021-02-26 12:03:50 1:45:11 1:36:11 0:09:00 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
fail 5915838 2021-02-26 05:56:23 2021-02-26 10:18:40 2021-02-26 11:33:33 1:14:53 1:03:05 0:11:48 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
Failure Reason:

The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'}

pass 5915839 2021-02-26 05:56:24 2021-02-26 10:19:51 2021-02-26 11:11:48 0:51:57 0:41:31 0:10:26 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} 2
pass 5915840 2021-02-26 05:56:25 2021-02-26 10:19:51 2021-02-26 10:40:48 0:20:57 0:13:29 0:07:28 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
pass 5915841 2021-02-26 05:56:26 2021-02-26 10:21:32 2021-02-26 10:47:43 0:26:11 0:13:14 0:12:57 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush} 2
fail 5915842 2021-02-26 05:56:27 2021-02-26 10:22:42 2021-02-26 11:09:27 0:46:45 0:36:48 0:09:57 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
Failure Reason:

SSH connection to smithi067 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

pass 5915843 2021-02-26 05:56:27 2021-02-26 10:26:23 2021-02-26 10:54:14 0:27:51 0:17:44 0:10:07 smithi master ubuntu 18.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
pass 5915844 2021-02-26 05:56:28 2021-02-26 10:26:24 2021-02-26 12:04:24 1:38:00 1:24:29 0:13:31 smithi master centos 8.2 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
pass 5915845 2021-02-26 05:56:29 2021-02-26 10:27:14 2021-02-26 10:52:53 0:25:39 0:09:26 0:16:13 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} 5
pass 5915846 2021-02-26 05:56:30 2021-02-26 10:29:16 2021-02-26 10:55:13 0:25:57 0:19:37 0:06:20 smithi master rhel 8.3 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} 2
fail 5915847 2021-02-26 05:56:30 2021-02-26 10:30:07 2021-02-26 10:56:00 0:25:53 0:14:57 0:10:56 smithi master fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

Command failed on smithi069 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

fail 5915848 2021-02-26 05:56:31 2021-02-26 10:30:07 2021-02-26 11:39:23 1:09:16 0:59:50 0:09:26 smithi master centos 8.2 fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/valgrind} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi176 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3f5559f52738d7ae55c88f318d5acc55fd54d9dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

pass 5915849 2021-02-26 05:56:32 2021-02-26 10:30:28 2021-02-26 10:55:59 0:25:31 0:15:39 0:09:52 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
pass 5915850 2021-02-26 05:56:33 2021-02-26 10:31:29 2021-02-26 10:51:07 0:19:38 0:12:13 0:07:25 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} 2
pass 5915851 2021-02-26 05:56:34 2021-02-26 10:31:59 2021-02-26 11:02:29 0:30:30 0:19:16 0:11:14 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} 2
fail 5915852 2021-02-26 05:56:35 2021-02-26 10:32:40 2021-02-26 12:12:44 1:40:04 1:29:53 0:10:11 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

"2021-02-26T11:00:54.276144+0000 mds.f (mds.0) 89 : cluster [WRN] Scrub error on inode 0x1000000ca33 (/client.0/tmp/payload.1/multiple_rsync_payload.126674/firmware/intel/sof-tplg) see mds.f log and `damage ls` output for details" in cluster log

pass 5915853 2021-02-26 05:56:35 2021-02-26 10:33:10 2021-02-26 11:54:30 1:21:20 1:15:00 0:06:20 smithi master rhel 8.3 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
pass 5915854 2021-02-26 05:56:36 2021-02-26 10:33:21 2021-02-26 11:54:50 1:21:29 1:14:49 0:06:40 smithi master rhel 8.3 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} 2
pass 5915855 2021-02-26 05:56:37 2021-02-26 10:33:31 2021-02-26 11:14:41 0:41:10 0:34:51 0:06:19 smithi master rhel 8.3 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} 2
pass 5915856 2021-02-26 05:56:38 2021-02-26 10:33:42 2021-02-26 11:07:27 0:33:45 0:21:46 0:11:59 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} 2
pass 5915857 2021-02-26 05:56:38 2021-02-26 10:34:03 2021-02-26 12:15:36 1:41:33 1:29:51 0:11:42 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
pass 5915858 2021-02-26 05:56:39 2021-02-26 10:34:23 2021-02-26 10:58:39 0:24:16 0:12:10 0:12:06 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} 2
fail 5915859 2021-02-26 05:56:40 2021-02-26 10:35:14 2021-02-26 11:19:39 0:44:25 0:36:31 0:07:54 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
Failure Reason:

SSH connection to smithi027 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

pass 5915860 2021-02-26 05:56:41 2021-02-26 10:36:05 2021-02-26 11:03:42 0:27:37 0:15:32 0:12:05 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
pass 5915861 2021-02-26 05:56:42 2021-02-26 10:36:06 2021-02-26 11:45:23 1:09:17 0:57:00 0:12:17 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} 3
pass 5915862 2021-02-26 05:56:42 2021-02-26 10:37:37 2021-02-26 11:08:42 0:31:05 0:17:58 0:13:07 smithi master centos 8.2 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} 2
pass 5915863 2021-02-26 05:56:43 2021-02-26 10:37:47 2021-02-26 11:12:36 0:34:49 0:27:16 0:07:33 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} 2
fail 5915864 2021-02-26 05:56:44 2021-02-26 10:37:47 2021-02-26 11:39:08 1:01:21 0:50:06 0:11:15 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} 3
Failure Reason:

"2021-02-26T10:59:59.325156+0000 mds.e (mds.0) 29 : cluster [WRN] Scrub error on inode 0x10000000348 (/client.0/tmp/tmp) see mds.e log and `damage ls` output for details" in cluster log

pass 5915865 2021-02-26 05:56:45 2021-02-26 10:39:38 2021-02-26 12:09:01 1:29:23 1:18:23 0:11:00 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
pass 5915866 2021-02-26 05:56:46 2021-02-26 10:39:49 2021-02-26 11:10:03 0:30:14 0:19:05 0:11:09 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 5915867 2021-02-26 05:56:46 2021-02-26 10:40:50 2021-02-26 11:09:04 0:28:14 0:16:03 0:12:11 smithi master centos 8.2 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
fail 5915868 2021-02-26 05:56:47 2021-02-26 10:42:11 2021-02-26 11:20:34 0:38:23 0:19:51 0:18:32 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} 2
Failure Reason:

Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage)

pass 5915869 2021-02-26 05:56:48 2021-02-26 10:42:42 2021-02-26 11:04:21 0:21:39 0:10:57 0:10:42 smithi master ubuntu 18.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
fail 5915870 2021-02-26 05:56:49 2021-02-26 10:42:53 2021-02-26 11:27:33 0:44:40 0:36:49 0:07:51 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
Failure Reason:

SSH connection to smithi060 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

pass 5915871 2021-02-26 05:56:49 2021-02-26 10:44:14 2021-02-26 11:32:52 0:48:38 0:40:29 0:08:09 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} 2
pass 5915872 2021-02-26 05:56:50 2021-02-26 10:45:25 2021-02-26 11:26:51 0:41:26 0:29:43 0:11:43 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} 3
pass 5915873 2021-02-26 05:56:51 2021-02-26 10:47:26 2021-02-26 11:16:54 0:29:28 0:19:21 0:10:07 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
pass 5915874 2021-02-26 05:56:52 2021-02-26 10:47:46 2021-02-26 11:11:51 0:24:05 0:13:13 0:10:52 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 5915875 2021-02-26 05:56:53 2021-02-26 10:48:07 2021-02-26 11:26:55 0:38:48 0:25:58 0:12:50 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} 2
pass 5915876 2021-02-26 05:56:53 2021-02-26 10:48:28 2021-02-26 11:11:52 0:23:24 0:12:35 0:10:49 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
pass 5915877 2021-02-26 05:56:54 2021-02-26 10:50:29 2021-02-26 11:20:48 0:30:19 0:21:44 0:08:35 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
pass 5915878 2021-02-26 05:56:55 2021-02-26 10:51:39 2021-02-26 12:05:09 1:13:30 1:07:20 0:06:10 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
pass 5915879 2021-02-26 05:56:56 2021-02-26 10:51:50 2021-02-26 11:51:18 0:59:28 0:52:57 0:06:31 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} 2
pass 5915880 2021-02-26 05:56:57 2021-02-26 10:52:11 2021-02-26 11:18:34 0:26:23 0:13:57 0:12:26 smithi master ubuntu 18.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} 2
pass 5915881 2021-02-26 05:56:57 2021-02-26 10:53:01 2021-02-26 11:28:13 0:35:12 0:24:29 0:10:43 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
pass 5915882 2021-02-26 05:56:58 2021-02-26 10:53:02 2021-02-26 11:14:36 0:21:34 0:14:30 0:07:04 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
pass 5915883 2021-02-26 05:56:59 2021-02-26 10:53:43 2021-02-26 11:17:16 0:23:33 0:14:16 0:09:17 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
pass 5915884 2021-02-26 05:57:00 2021-02-26 10:53:43 2021-02-26 11:15:50 0:22:07 0:13:43 0:08:24 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} 2
pass 5915885 2021-02-26 05:57:01 2021-02-26 10:54:24 2021-02-26 11:18:38 0:24:14 0:12:41 0:11:33 smithi master ubuntu 18.04 fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs} 2
pass 5915886 2021-02-26 05:57:02 2021-02-26 10:55:15 2021-02-26 12:27:02 1:31:47 1:21:39 0:10:08 smithi master ubuntu 20.04 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} 2
pass 5915887 2021-02-26 05:57:02 2021-02-26 10:56:06 2021-02-26 11:19:41 0:23:35 0:11:06 0:12:29 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} 4
pass 5915888 2021-02-26 05:57:03 2021-02-26 10:56:16 2021-02-26 11:59:49 1:03:33 0:50:03 0:13:30 smithi master centos 8.2 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} 2
pass 5915889 2021-02-26 05:57:04 2021-02-26 10:56:17 2021-02-26 11:21:33 0:25:16 0:12:34 0:12:42 smithi master ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} 2
fail 5915890 2021-02-26 05:57:05 2021-02-26 10:57:48 2021-02-26 11:37:00 0:39:12 0:25:49 0:13:23 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}} 3
Failure Reason:

Command failed on smithi057 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

pass 5915891 2021-02-26 05:57:06 2021-02-26 10:58:49 2021-02-26 12:19:53 1:21:04 1:06:11 0:14:53 smithi master centos 8.2 fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/lockdep} 2
pass 5915892 2021-02-26 05:57:07 2021-02-26 11:02:40 2021-02-26 11:35:14 0:32:34 0:19:37 0:12:57 smithi master ubuntu 18.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
fail 5915893 2021-02-26 05:57:07 2021-02-26 11:03:01 2021-02-26 11:49:00 0:45:59 0:38:23 0:07:36 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
Failure Reason:

SSH connection to smithi107 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

pass 5915894 2021-02-26 05:57:08 2021-02-26 11:03:42 2021-02-26 12:35:14 1:31:32 1:21:01 0:10:31 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
pass 5915895 2021-02-26 05:57:09 2021-02-26 11:03:52 2021-02-26 11:37:26 0:33:34 0:21:32 0:12:02 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment} 2
pass 5915896 2021-02-26 05:57:10 2021-02-26 11:04:23 2021-02-26 12:11:54 1:07:31 0:51:28 0:16:03 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
pass 5915897 2021-02-26 05:57:10 2021-02-26 11:08:46 2021-02-26 12:24:01 1:15:15 1:04:19 0:10:56 smithi master centos 8.2 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
pass 5915898 2021-02-26 05:57:11 2021-02-26 11:09:06 2021-02-26 12:00:27 0:51:21 0:45:16 0:06:05 smithi master rhel 8.3 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} 2
pass 5915899 2021-02-26 05:57:12 2021-02-26 11:09:37 2021-02-26 11:43:06 0:33:29 0:20:14 0:13:15 smithi master centos 8.2 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} 2
pass 5915900 2021-02-26 05:57:13 2021-02-26 11:09:38 2021-02-26 12:03:57 0:54:19 0:43:07 0:11:12 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} 2
fail 5915901 2021-02-26 05:57:13 2021-02-26 11:10:09 2021-02-26 11:45:52 0:35:43 0:24:05 0:11:38 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
Failure Reason:

"2021-02-26T11:32:33.885583+0000 mds.f (mds.0) 29 : cluster [WRN] Scrub error on inode 0x100000001fa (/client.0/tmp/blogbench-1.0/man) see mds.f log and `damage ls` output for details" in cluster log

pass 5915902 2021-02-26 05:57:14 2021-02-26 11:11:49 2021-02-26 11:34:39 0:22:50 0:13:18 0:09:32 smithi master centos 8.2 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 5915903 2021-02-26 05:57:15 2021-02-26 11:12:00 2021-02-26 11:43:48 0:31:48 0:20:23 0:11:25 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair} 2
pass 5915904 2021-02-26 05:57:16 2021-02-26 11:12:00 2021-02-26 11:35:46 0:23:46 0:12:33 0:11:13 smithi master centos 8.2 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 5915905 2021-02-26 05:57:17 2021-02-26 11:12:31 2021-02-26 11:41:35 0:29:04 0:18:38 0:10:26 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
pass 5915906 2021-02-26 05:57:17 2021-02-26 11:12:42 2021-02-26 12:31:29 1:18:47 1:09:11 0:09:36 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
fail 5915907 2021-02-26 05:57:18 2021-02-26 11:12:42 2021-02-26 11:57:39 0:44:57 0:35:54 0:09:03 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
Failure Reason:

SSH connection to smithi079 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

pass 5915908 2021-02-26 05:57:19 2021-02-26 11:14:43 2021-02-26 11:33:28 0:18:45 0:12:59 0:05:46 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} 2
pass 5915909 2021-02-26 05:57:20 2021-02-26 11:14:44 2021-02-26 11:35:58 0:21:14 0:10:30 0:10:44 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} 2
pass 5915910 2021-02-26 05:57:21 2021-02-26 11:15:04 2021-02-26 11:46:32 0:31:28 0:17:53 0:13:35 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 5915911 2021-02-26 05:57:21 2021-02-26 11:17:05 2021-02-26 11:47:45 0:30:40 0:21:43 0:08:57 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} 2
fail 5915912 2021-02-26 05:57:22 2021-02-26 11:17:16 2021-02-26 11:46:55 0:29:39 0:18:55 0:10:44 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
Failure Reason:

"2021-02-26T11:37:43.309220+0000 mds.c (mds.0) 24 : cluster [WRN] Scrub error on inode 0x10000003020 (/client.0/tmp/tmp.JTUeTlTTuy) see mds.c log and `damage ls` output for details" in cluster log

fail 5915913 2021-02-26 05:57:23 2021-02-26 11:17:26 2021-02-26 11:41:44 0:24:18 0:16:11 0:08:07 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} 2
Failure Reason:

Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)

pass 5915914 2021-02-26 05:57:24 2021-02-26 11:18:37 2021-02-26 11:42:27 0:23:50 0:14:13 0:09:37 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} 2
pass 5915915 2021-02-26 05:57:24 2021-02-26 11:18:48 2021-02-26 12:11:19 0:52:31 0:37:15 0:15:16 smithi master ubuntu 18.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} 3
pass 5915916 2021-02-26 05:57:25 2021-02-26 11:19:49 2021-02-26 11:38:37 0:18:48 0:13:30 0:05:18 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} 3
pass 5915917 2021-02-26 05:57:26 2021-02-26 11:19:50 2021-02-26 12:25:47 1:05:57 0:55:57 0:10:00 smithi master ubuntu 18.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}} 2
pass 5915918 2021-02-26 05:57:27 2021-02-26 11:19:51 2021-02-26 11:46:35 0:26:44 0:12:47 0:13:57 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry} 2
pass 5915919 2021-02-26 05:57:28 2021-02-26 11:20:41 2021-02-26 11:57:02 0:36:21 0:25:25 0:10:56 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
pass 5915920 2021-02-26 05:57:28 2021-02-26 11:20:52 2021-02-26 14:24:58 3:04:06 2:54:34 0:09:32 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
pass 5915921 2021-02-26 05:57:29 2021-02-26 11:21:43 2021-02-26 12:14:30 0:52:47 0:40:37 0:12:10 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
pass 5915922 2021-02-26 05:57:30 2021-02-26 11:22:54 2021-02-26 12:03:24 0:40:30 0:19:16 0:21:14 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} 2
pass 5915923 2021-02-26 05:57:31 2021-02-26 11:26:55 2021-02-26 12:16:49 0:49:54 0:43:37 0:06:17 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} 2
pass 5915924 2021-02-26 05:57:32 2021-02-26 11:27:06 2021-02-26 11:53:23 0:26:17 0:14:24 0:11:53 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
fail 5915925 2021-02-26 05:57:33 2021-02-26 11:27:37 2021-02-26 12:12:58 0:45:21 0:39:23 0:05:58 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} 3
Failure Reason:

"2021-02-26T11:46:37.992460+0000 mds.f (mds.0) 34 : cluster [WRN] Scrub error on inode 0x10000000000 (/client.0) see mds.f log and `damage ls` output for details" in cluster log

pass 5915926 2021-02-26 05:57:33 2021-02-26 11:27:48 2021-02-26 11:57:07 0:29:19 0:18:39 0:10:40 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
pass 5915927 2021-02-26 05:57:34 2021-02-26 11:28:19 2021-02-26 11:52:10 0:23:51 0:12:27 0:11:24 smithi master centos 8.2 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 5915928 2021-02-26 05:57:35 2021-02-26 11:28:19 2021-02-26 12:01:28 0:33:09 0:17:35 0:15:34 smithi master ubuntu 18.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} 2
pass 5915929 2021-02-26 05:57:36 2021-02-26 11:31:51 2021-02-26 12:54:26 1:22:35 1:14:14 0:08:21 smithi master rhel 8.3 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
fail 5915930 2021-02-26 05:57:37 2021-02-26 11:33:02 2021-02-26 12:03:19 0:30:17 0:14:17 0:16:00 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} 5
Failure Reason:

Test failure: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc)

fail 5915931 2021-02-26 05:57:37 2021-02-26 11:33:43 2021-02-26 12:05:29 0:31:46 0:18:05 0:13:41 smithi master ubuntu 20.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} 2
Failure Reason:

Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)

pass 5915932 2021-02-26 05:57:38 2021-02-26 11:33:53 2021-02-26 11:56:02 0:22:09 0:10:48 0:11:21 smithi master ubuntu 18.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
fail 5915933 2021-02-26 05:57:39 2021-02-26 11:34:44 2021-02-26 12:19:45 0:45:01 0:29:59 0:15:02 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

pass 5915934 2021-02-26 05:57:40 2021-02-26 11:35:25 2021-02-26 13:17:19 1:41:54 1:31:13 0:10:41 smithi master centos 8.2 fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/valgrind} 2
pass 5915935 2021-02-26 05:57:41 2021-02-26 11:35:56 2021-02-26 12:04:34 0:28:38 0:16:59 0:11:39 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
pass 5915936 2021-02-26 05:57:41 2021-02-26 11:37:06 2021-02-26 11:58:39 0:21:33 0:11:28 0:10:05 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} 2
fail 5915937 2021-02-26 05:57:42 2021-02-26 11:37:07 2021-02-26 12:58:16 1:21:09 1:08:54 0:12:15 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
Failure Reason:

"2021-02-26T12:07:37.690915+0000 mds.c (mds.2) 55 : cluster [WRN] Scrub error on inode 0x10000000263 (/client.0/tmp/testdir/dir1/dir2) see mds.c log and `damage ls` output for details" in cluster log

pass 5915938 2021-02-26 05:57:43 2021-02-26 11:38:48 2021-02-26 11:59:52 0:21:04 0:13:49 0:07:15 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} 2
pass 5915939 2021-02-26 05:57:44 2021-02-26 11:38:48 2021-02-26 14:02:16 2:23:28 2:15:52 0:07:36 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
pass 5915940 2021-02-26 05:57:45 2021-02-26 11:39:19 2021-02-26 11:58:15 0:18:56 0:12:56 0:06:00 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} 2
pass 5915941 2021-02-26 05:57:45 2021-02-26 11:39:30 2021-02-26 12:12:44 0:33:14 0:24:26 0:08:48 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
pass 5915942 2021-02-26 05:57:46 2021-02-26 11:41:51 2021-02-26 12:02:44 0:20:53 0:14:19 0:06:34 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
pass 5915943 2021-02-26 05:57:47 2021-02-26 11:41:51 2021-02-26 13:16:09 1:34:18 1:23:08 0:11:10 smithi master ubuntu 20.04 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
pass 5915944 2021-02-26 05:57:48 2021-02-26 11:42:32 2021-02-26 12:10:58 0:28:26 0:16:40 0:11:46 smithi master centos 8.2 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} 2
pass 5915945 2021-02-26 05:57:48 2021-02-26 11:43:12 2021-02-26 12:15:37 0:32:25 0:20:23 0:12:02 smithi master ubuntu 20.04 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} 2
pass 5915946 2021-02-26 05:57:49 2021-02-26 11:43:53 2021-02-26 13:41:24 1:57:31 1:43:59 0:13:32 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} 3
pass 5915947 2021-02-26 05:57:50 2021-02-26 11:45:54 2021-02-26 12:12:48 0:26:54 0:14:55 0:11:59 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} 2
pass 5915948 2021-02-26 05:57:51 2021-02-26 11:45:55 2021-02-26 12:09:43 0:23:48 0:16:54 0:06:54 smithi master rhel 8.3 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/iozone}} 2
fail 5915949 2021-02-26 05:57:52 2021-02-26 11:46:35 2021-02-26 12:37:02 0:50:27 0:40:03 0:10:24 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} 3
Failure Reason:

"2021-02-26T12:06:54.671918+0000 mds.c (mds.0) 24 : cluster [WRN] Scrub error on inode 0x10000000348 (/client.0/tmp/tmp) see mds.c log and `damage ls` output for details" in cluster log

pass 5915950 2021-02-26 05:57:52 2021-02-26 11:46:46 2021-02-26 13:04:25 1:17:39 1:05:55 0:11:44 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
pass 5915951 2021-02-26 05:57:53 2021-02-26 11:47:27 2021-02-26 12:12:53 0:25:26 0:15:14 0:10:12 smithi master ubuntu 18.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 5915952 2021-02-26 05:57:54 2021-02-26 11:47:48 2021-02-26 12:10:37 0:22:49 0:11:16 0:11:33 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota} 2
fail 5915953 2021-02-26 05:57:55 2021-02-26 11:49:08 2021-02-26 12:12:36 0:23:28 0:14:50 0:08:38 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
Failure Reason:

"2021-02-26T12:08:07.737971+0000 mds.f (mds.0) 24 : cluster [WRN] Scrub error on inode 0x100000004d9 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/network) see mds.f log and `damage ls` output for details" in cluster log

pass 5915954 2021-02-26 05:57:56 2021-02-26 11:51:29 2021-02-26 12:47:54 0:56:25 0:46:07 0:10:18 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} 2
pass 5915955 2021-02-26 05:57:56 2021-02-26 11:52:20 2021-02-26 12:42:59 0:50:39 0:37:38 0:13:01 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
pass 5915956 2021-02-26 05:57:57 2021-02-26 11:53:31 2021-02-26 12:21:19 0:27:48 0:16:42 0:11:06 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} 2
pass 5915957 2021-02-26 05:57:58 2021-02-26 11:54:01 2021-02-26 12:24:07 0:30:06 0:17:33 0:12:33 smithi master ubuntu 18.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
pass 5915958 2021-02-26 05:57:59 2021-02-26 11:54:02 2021-02-26 12:16:41 0:22:39 0:12:13 0:10:26 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 5915959 2021-02-26 05:58:00 2021-02-26 11:54:53 2021-02-26 12:14:31 0:19:38 0:11:43 0:07:55 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
pass 5915960 2021-02-26 05:58:00 2021-02-26 11:56:13 2021-02-26 12:28:44 0:32:31 0:20:23 0:12:08 smithi master ubuntu 18.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} 3
pass 5915961 2021-02-26 05:58:01 2021-02-26 11:57:04 2021-02-26 12:24:39 0:27:35 0:16:32 0:11:03 smithi master ubuntu 18.04 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
pass 5915962 2021-02-26 05:58:02 2021-02-26 11:57:15 2021-02-26 12:16:35 0:19:20 0:12:42 0:06:38 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap} 2
pass 5915963 2021-02-26 05:58:03 2021-02-26 11:57:46 2021-02-26 12:20:32 0:22:46 0:13:20 0:09:26 smithi master centos 8.2 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
fail 5915964 2021-02-26 05:58:04 2021-02-26 11:57:47 2021-02-26 12:31:46 0:33:59 0:26:04 0:07:55 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
Failure Reason:

SSH connection to smithi082 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

pass 5915965 2021-02-26 05:58:04 2021-02-26 11:58:48 2021-02-26 12:35:03 0:36:15 0:23:53 0:12:22 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} 2
pass 5915966 2021-02-26 05:58:05 2021-02-26 11:59:59 2021-02-26 12:31:43 0:31:44 0:20:23 0:11:21 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
pass 5915967 2021-02-26 05:58:06 2021-02-26 12:00:00 2021-02-26 12:34:57 0:34:57 0:22:34 0:12:23 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} 2
pass 5915968 2021-02-26 05:58:07 2021-02-26 12:00:01 2021-02-26 12:28:10 0:28:09 0:15:58 0:12:11 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
pass 5915969 2021-02-26 05:58:08 2021-02-26 12:00:31 2021-02-26 12:25:14 0:24:43 0:13:35 0:11:08 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} 3
pass 5915970 2021-02-26 05:58:08 2021-02-26 12:01:33 2021-02-26 12:20:13 0:18:40 0:10:52 0:07:48 smithi master rhel 8.3 fs/libcephfs/{begin clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python} 2
pass 5915971 2021-02-26 05:58:09 2021-02-26 12:03:04 2021-02-26 13:36:33 1:33:29 1:19:21 0:14:08 smithi master centos 8.2 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} 2
pass 5915972 2021-02-26 05:58:10 2021-02-26 12:03:25 2021-02-26 12:24:51 0:21:26 0:08:36 0:12:50 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} 4
pass 5915973 2021-02-26 05:58:11 2021-02-26 12:03:25 2021-02-26 13:00:03 0:56:38 0:44:47 0:11:51 smithi master ubuntu 20.04 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} 2
pass 5915974 2021-02-26 05:58:12 2021-02-26 12:03:56 2021-02-26 14:16:55 2:12:59 2:01:10 0:11:49 smithi master centos 8.2 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} 2
fail 5915975 2021-02-26 05:58:12 2021-02-26 12:03:57 2021-02-26 12:29:57 0:26:00 0:15:24 0:10:36 smithi master fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

Command failed on smithi017 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

pass 5915976 2021-02-26 05:58:13 2021-02-26 12:04:07 2021-02-26 13:23:40 1:19:33 1:09:34 0:09:59 smithi master centos 8.2 fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/lockdep} 2
pass 5915977 2021-02-26 05:58:14 2021-02-26 12:04:28 2021-02-26 12:28:52 0:24:24 0:12:37 0:11:47 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
pass 5915978 2021-02-26 05:58:15 2021-02-26 12:04:39 2021-02-26 12:38:32 0:33:53 0:22:53 0:11:00 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} 2
fail 5915979 2021-02-26 05:58:15 2021-02-26 12:04:59 2021-02-26 13:37:04 1:32:05 1:19:25 0:12:40 smithi master ubuntu 18.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
Failure Reason:

"2021-02-26T12:34:06.624079+0000 mds.a (mds.0) 84 : cluster [WRN] Scrub error on inode 0x10000000201 (/client.0/tmp/testdir/dir1/dir2/dir3) see mds.a log and `damage ls` output for details" in cluster log

pass 5915980 2021-02-26 05:58:16 2021-02-26 12:05:20 2021-02-26 13:56:40 1:51:20 1:44:34 0:06:46 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
pass 5915981 2021-02-26 05:58:17 2021-02-26 12:05:40 2021-02-26 12:46:31 0:40:51 0:29:55 0:10:56 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
pass 5915982 2021-02-26 05:58:18 2021-02-26 12:09:11 2021-02-26 12:43:01 0:33:50 0:26:42 0:07:08 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} 2
pass 5915983 2021-02-26 05:58:19 2021-02-26 12:09:52 2021-02-26 12:41:52 0:32:00 0:24:04 0:07:56 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} 2
dead 5915984 2021-02-26 05:58:19 2021-02-26 12:10:43 2021-02-27 00:20:05 12:09:22 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
Failure Reason:

hit max job timeout

pass 5915985 2021-02-26 05:58:20 2021-02-26 12:11:24 2021-02-26 12:46:58 0:35:34 0:23:42 0:11:52 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
pass 5915986 2021-02-26 05:58:21 2021-02-26 12:11:24 2021-02-26 13:19:44 1:08:20 0:57:59 0:10:21 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 5915987 2021-02-26 05:58:22 2021-02-26 12:12:05 2021-02-26 12:37:46 0:25:41 0:19:23 0:06:18 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration} 2
fail 5915988 2021-02-26 05:58:23 2021-02-26 12:12:46 2021-02-26 13:36:23 1:23:37 1:12:21 0:11:16 smithi master ubuntu 18.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
Failure Reason:

"2021-02-26T12:32:54.906830+0000 mds.f (mds.0) 24 : cluster [WRN] Scrub error on inode 0x10000000348 (/client.0/tmp/tmp) see mds.f log and `damage ls` output for details" in cluster log

pass 5915989 2021-02-26 05:58:23 2021-02-26 12:12:57 2021-02-26 13:32:21 1:19:24 1:10:32 0:08:52 smithi master ubuntu 20.04 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
pass 5915990 2021-02-26 05:58:24 2021-02-26 12:12:57 2021-02-26 12:42:32 0:29:35 0:19:18 0:10:17 smithi master ubuntu 18.04 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} 2
pass 5915991 2021-02-26 05:58:25 2021-02-26 12:12:58 2021-02-26 13:10:14 0:57:16 0:46:51 0:10:25 smithi master ubuntu 20.04 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} 2
pass 5915992 2021-02-26 05:58:26 2021-02-26 12:12:58 2021-02-26 12:34:15 0:21:17 0:11:30 0:09:47 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} 2
pass 5915993 2021-02-26 05:58:26 2021-02-26 12:12:59 2021-02-26 12:32:16 0:19:17 0:13:40 0:05:37 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 5915994 2021-02-26 05:58:27 2021-02-26 12:13:09 2021-02-26 12:39:33 0:26:24 0:14:26 0:11:58 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} 2
pass 5915995 2021-02-26 05:58:28 2021-02-26 12:14:40 2021-02-26 12:33:43 0:19:03 0:12:33 0:06:30 smithi master rhel 8.3 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 5915996 2021-02-26 05:58:29 2021-02-26 12:14:41 2021-02-26 12:39:34 0:24:53 0:13:27 0:11:26 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay} 2
pass 5915997 2021-02-26 05:58:30 2021-02-26 12:14:41 2021-02-26 12:37:23 0:22:42 0:11:02 0:11:40 smithi master ubuntu 18.04 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
pass 5915998 2021-02-26 05:58:30 2021-02-26 12:15:42 2021-02-26 12:45:04 0:29:22 0:19:06 0:10:16 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
pass 5915999 2021-02-26 05:58:31 2021-02-26 12:15:42 2021-02-26 12:37:51 0:22:09 0:13:33 0:08:36 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} 2
pass 5916000 2021-02-26 05:58:32 2021-02-26 12:16:43 2021-02-26 12:50:21 0:33:38 0:23:26 0:10:12 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} 3
pass 5916001 2021-02-26 05:58:33 2021-02-26 12:16:44 2021-02-26 13:01:35 0:44:51 0:33:48 0:11:03 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 5916002 2021-02-26 05:58:33 2021-02-26 12:19:55 2021-02-26 12:43:18 0:23:23 0:12:15 0:11:08 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota} 2
pass 5916003 2021-02-26 05:58:34 2021-02-26 12:19:55 2021-02-26 12:55:59 0:36:04 0:23:54 0:12:10 smithi master ubuntu 20.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} 2
fail 5916004 2021-02-26 05:58:35 2021-02-26 12:19:56 2021-02-26 13:01:41 0:41:45 0:35:07 0:06:38 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
Failure Reason:

SSH connection to smithi044 was lost: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,name=0,conf=/etc/ceph/ceph.conf,norbytes,nowsync'

fail 5916005 2021-02-26 05:58:36 2021-02-26 12:20:37 2021-02-26 14:20:32 1:59:55 1:47:35 0:12:20 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

"2021-02-26T13:16:19.192527+0000 mds.e (mds.0) 29 : cluster [WRN] client.4584 isn't responding to mclientcaps(revoke), ino 0x1000000f059 pending pAsxLsXsxFscr issued pAsxLsXsxFsxcrwb, sent 304.975494 seconds ago" in cluster log

pass 5916006 2021-02-26 05:58:36 2021-02-26 12:21:27 2021-02-26 13:47:13 1:25:46 1:12:46 0:13:00 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
pass 5916007 2021-02-26 05:58:37 2021-02-26 12:24:08 2021-02-26 12:48:28 0:24:20 0:12:50 0:11:30 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} 2
pass 5916008 2021-02-26 05:58:38 2021-02-26 12:24:19 2021-02-26 13:04:29 0:40:10 0:30:35 0:09:35 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} 2
pass 5916009 2021-02-26 05:58:39 2021-02-26 12:24:50 2021-02-26 12:46:28 0:21:38 0:12:59 0:08:39 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
pass 5916010 2021-02-26 05:58:40 2021-02-26 12:25:00 2021-02-26 12:49:02 0:24:02 0:12:17 0:11:45 smithi master ubuntu 18.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
pass 5916011 2021-02-26 05:58:40 2021-02-26 12:25:21 2021-02-26 13:03:25 0:38:04 0:30:21 0:07:43 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} 2
pass 5916012 2021-02-26 05:58:41 2021-02-26 12:25:52 2021-02-26 12:55:41 0:29:49 0:18:36 0:11:13 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
pass 5916013 2021-02-26 05:58:42 2021-02-26 12:27:03 2021-02-26 14:03:48 1:36:45 1:23:46 0:12:59 smithi master ubuntu 20.04 fs/mixed-clients/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} 2
pass 5916014 2021-02-26 05:58:43 2021-02-26 12:28:14 2021-02-26 13:03:17 0:35:03 0:19:32 0:15:31 smithi master ubuntu 20.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} 5
pass 5916015 2021-02-26 05:58:43 2021-02-26 12:29:04 2021-02-26 13:06:26 0:37:22 0:30:16 0:07:06 smithi master rhel 8.3 fs/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} 2
fail 5916016 2021-02-26 05:58:44 2021-02-26 12:29:05 2021-02-26 13:09:14 0:40:09 0:26:34 0:13:35 smithi master fs/upgrade/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}} 3
Failure Reason:

Command failed on smithi087 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

fail 5916017 2021-02-26 05:58:45 2021-02-26 12:31:36 2021-02-26 14:38:57 2:07:21 1:55:56 0:11:25 smithi master centos 8.2 fs/verify/{begin centos_latest clusters/1a5s-mds-1c-client conf/{client mds mon osd} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/valgrind} 2
Failure Reason:

Command failed on smithi105 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'

pass 5916018 2021-02-26 05:58:46 2021-02-26 12:31:47 2021-02-26 12:50:48 0:19:01 0:11:56 0:07:05 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
pass 5916019 2021-02-26 05:58:46 2021-02-26 12:31:57 2021-02-26 12:51:34 0:19:37 0:11:54 0:07:43 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} 2
fail 5916020 2021-02-26 05:58:47 2021-02-26 12:32:28 2021-02-26 13:56:43 1:24:15 1:13:45 0:10:30 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

"2021-02-26T12:58:34.146066+0000 mds.a (mds.0) 54 : cluster [WRN] Scrub error on inode 0x10000000263 (/client.0/tmp/testdir/dir1) see mds.a log and `damage ls` output for details" in cluster log

pass 5916021 2021-02-26 05:58:48 2021-02-26 12:33:49 2021-02-26 12:51:21 0:17:32 0:10:48 0:06:44 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} 2
pass 5916022 2021-02-26 05:58:49 2021-02-26 12:34:19 2021-02-26 13:08:00 0:33:41 0:24:00 0:09:41 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} 2
pass 5916023 2021-02-26 05:58:50 2021-02-26 12:35:00 2021-02-26 13:17:20 0:42:20 0:31:09 0:11:11 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
fail 5916024 2021-02-26 05:58:50 2021-02-26 12:35:11 2021-02-26 13:14:49 0:39:38 0:25:06 0:14:32 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
Failure Reason:

"2021-02-26T12:59:41.226921+0000 mds.f (mds.0) 24 : cluster [WRN] Scrub error on inode 0x100000010da (/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-22) see mds.f log and `damage ls` output for details" in cluster log

pass 5916025 2021-02-26 05:58:51 2021-02-26 12:37:12 2021-02-26 13:02:47 0:25:35 0:15:38 0:09:57 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
pass 5916026 2021-02-26 05:58:52 2021-02-26 12:37:12 2021-02-26 13:02:54 0:25:42 0:15:38 0:10:04 smithi master centos 8.2 fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
fail 5916027 2021-02-26 05:58:53 2021-02-26 12:37:32 2021-02-26 13:21:10 0:43:38 0:31:01 0:12:37 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} 2
Failure Reason:

"2021-02-26T13:19:18.871401+0000 mon.a (mon.0) 498 : cluster [WRN] Replacing daemon mds.c as rank 0 with standby daemon mds.d" in cluster log

pass 5916028 2021-02-26 05:58:54 2021-02-26 12:37:53 2021-02-26 13:01:19 0:23:26 0:12:46 0:10:40 smithi master centos 8.2 fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
pass 5916029 2021-02-26 05:58:54 2021-02-26 12:37:53 2021-02-26 14:00:28 1:22:35 1:13:56 0:08:39 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 5916030 2021-02-26 05:58:55 2021-02-26 12:39:34 2021-02-26 13:03:11 0:23:37 0:13:00 0:10:37 smithi master ubuntu 18.04 fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag_enable multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} 2
fail 5916031 2021-02-26 05:58:56 2021-02-26 12:39:45 2021-02-26 13:43:12 1:03:27 0:53:53 0:09:34 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} 3
Failure Reason:

"2021-02-26T13:08:27.452566+0000 mon.a (mon.0) 247 : cluster [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster log

pass 5916032 2021-02-26 05:58:57 2021-02-26 12:40:05 2021-02-26 13:05:32 0:25:27 0:12:07 0:13:20 smithi master ubuntu 20.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} 2
pass 5916033 2021-02-26 05:58:58 2021-02-26 12:41:56 2021-02-26 14:17:45 1:35:49 1:27:58 0:07:51 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
pass 5916034 2021-02-26 05:58:58 2021-02-26 12:42:37 2021-02-26 13:06:50 0:24:13 0:13:20 0:10:53 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 5916035 2021-02-26 05:58:59 2021-02-26 12:43:08 2021-02-26 14:30:18 1:47:10 1:36:16 0:10:54 smithi master centos 8.2 fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
pass 5916036 2021-02-26 05:59:00 2021-02-26 12:43:08 2021-02-26 13:40:58 0:57:50 0:47:14 0:10:36 smithi master ubuntu 18.04 fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} 2
pass 5916037 2021-02-26 05:59:01 2021-02-26 12:43:29 2021-02-26 13:34:27 0:50:58 0:42:08 0:08:50 smithi master rhel 8.3 fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} 2
fail 5916038 2021-02-26 05:59:02 2021-02-26 12:45:10 2021-02-26 13:11:33 0:26:23 0:19:17 0:07:06 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
Failure Reason:

"2021-02-26T13:03:57.414858+0000 mds.a (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000c24 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_hpi_testsuite) see mds.a log and `damage ls` output for details" in cluster log

pass 5916039 2021-02-26 05:59:02 2021-02-26 12:46:31 2021-02-26 13:17:50 0:31:19 0:25:23 0:05:56 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} 2
pass 5916040 2021-02-26 05:59:03 2021-02-26 12:46:41 2021-02-26 13:06:29 0:19:48 0:09:12 0:10:36 smithi master ubuntu 18.04 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} 2
pass 5916041 2021-02-26 05:59:04 2021-02-26 12:46:42 2021-02-26 13:13:16 0:26:34 0:17:51 0:08:43 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} 3
pass 5916042 2021-02-26 05:59:05 2021-02-26 12:48:03 2021-02-26 13:22:18 0:34:15 0:23:16 0:10:59 smithi master centos 8.2 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
pass 5916043 2021-02-26 05:59:06 2021-02-26 12:48:33 2021-02-26 13:14:22 0:25:49 0:15:20 0:10:29 smithi master centos 8.2 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 5916044 2021-02-26 05:59:06 2021-02-26 12:49:04 2021-02-26 13:07:31 0:18:27 0:10:43 0:07:44 smithi master rhel 8.3 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush} 2
pass 5916045 2021-02-26 05:59:07 2021-02-26 12:50:24 2021-02-26 13:12:36 0:22:12 0:10:47 0:11:25 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
pass 5916046 2021-02-26 05:59:08 2021-02-26 12:50:25 2021-02-26 13:22:19 0:31:54 0:21:55 0:09:59 smithi master ubuntu 20.04 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} 3
pass 5916047 2021-02-26 05:59:09 2021-02-26 12:50:55 2021-02-26 14:00:56 1:10:01 1:02:16 0:07:45 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
pass 5916048 2021-02-26 05:59:10 2021-02-26 12:51:46 2021-02-26 13:18:03 0:26:17 0:15:52 0:10:25 smithi master rhel 8.3 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} 2
pass 5916049 2021-02-26 05:59:10 2021-02-26 12:54:37 2021-02-26 13:30:08 0:35:31 0:22:23 0:13:08 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} 2
pass 5916050 2021-02-26 05:59:11 2021-02-26 12:55:48 2021-02-26 13:17:59 0:22:11 0:15:10 0:07:01 smithi master rhel 8.3 fs/workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3