Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 5735619 2020-12-24 22:49:08 2020-12-24 22:50:40 2020-12-24 23:14:39 0:23:59 0:12:17 0:11:42 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5735620 2020-12-24 22:49:08 2020-12-24 22:50:55 2020-12-24 23:50:55 1:00:00 0:50:47 0:09:13 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi139 with status 23: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=33b60211b54317df021e1b30c3d0b85dc41a20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh'

pass 5735621 2020-12-24 22:49:09 2020-12-24 22:52:07 2020-12-25 01:52:10 3:00:03 0:41:58 2:18:05 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
pass 5735622 2020-12-24 22:49:10 2020-12-24 22:52:34 2020-12-24 23:34:33 0:41:59 0:22:45 0:19:14 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
pass 5735623 2020-12-24 22:49:11 2020-12-24 22:52:37 2020-12-25 01:46:40 2:54:03 1:10:11 1:43:52 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5735624 2020-12-24 22:49:11 2020-12-24 22:52:40 2020-12-24 23:42:40 0:50:00 0:42:53 0:07:07 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

"2020-12-24T23:06:59.751017+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000348 (/client.0/tmp/tmp) see mds.f log and `damage ls` output for details" in cluster log

pass 5735625 2020-12-24 22:49:12 2020-12-24 22:54:28 2020-12-24 23:36:28 0:42:00 0:12:39 0:29:21 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5735626 2020-12-24 22:49:13 2020-12-24 22:54:28 2020-12-24 23:36:28 0:42:00 0:17:36 0:24:24 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2020-12-24T23:30:15.761129+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000c23 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_hpi_testsuite) see mds.f log and `damage ls` output for details" in cluster log

pass 5735627 2020-12-24 22:49:14 2020-12-24 22:54:32 2020-12-24 23:32:32 0:38:00 0:28:20 0:09:40 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5735628 2020-12-24 22:49:15 2020-12-24 22:54:37 2020-12-25 00:00:38 1:06:01 0:12:11 0:53:50 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5735629 2020-12-24 22:49:15 2020-12-24 22:54:38 2020-12-25 00:36:39 1:42:01 0:20:35 1:21:26 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5735630 2020-12-24 22:49:16 2020-12-24 22:54:55 2020-12-25 00:02:55 1:08:00 0:50:42 0:17:18 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5735631 2020-12-24 22:49:17 2020-12-24 22:54:56 2020-12-24 23:30:56 0:36:00 0:13:42 0:22:18 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
fail 5735632 2020-12-24 22:49:18 2020-12-24 22:55:57 2020-12-24 23:13:56 0:17:59 0:12:21 0:05:38 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
Failure Reason:

"2020-12-24T23:11:08.671676+0000 mds.b (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000002cd (/client.0/tmp/tmp) see mds.b log and `damage ls` output for details" in cluster log

pass 5735633 2020-12-24 22:49:18 2020-12-24 22:56:54 2020-12-24 23:18:54 0:22:00 0:10:38 0:11:22 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5735634 2020-12-24 22:49:19 2020-12-24 22:57:25 2020-12-24 23:59:25 1:02:00 0:46:26 0:15:34 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

"2020-12-24T23:38:41.141331+0000 mds.a (mds.0) 149 : cluster [WRN] Scrub error on inode 0x1000001545d (/client.0/tmp/payload.2/multiple_rsync_payload.143906/modules/4.18.0-193.28.1.el8_2.x86_64/kernel/drivers/hwmon) see mds.a log and `damage ls` output for details" in cluster log

fail 5735635 2020-12-24 22:49:20 2020-12-24 22:57:25 2020-12-24 23:55:25 0:58:00 0:47:33 0:10:27 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
Failure Reason:

"2020-12-24T23:35:21.371835+0000 mon.b (mon.0) 280 : cluster [WRN] Replacing daemon mds.d as rank 0 with standby daemon mds.f" in cluster log

fail 5735636 2020-12-24 22:49:21 2020-12-24 22:57:37 2020-12-25 00:07:38 1:10:01 0:43:30 0:26:31 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

Command failed on smithi033 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'

pass 5735637 2020-12-24 22:49:22 2020-12-24 22:58:04 2020-12-25 00:06:04 1:08:00 1:00:15 0:07:45 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5735638 2020-12-24 22:49:22 2020-12-24 22:59:21 2020-12-25 00:37:22 1:38:01 0:41:22 0:56:39 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

"2020-12-25T00:08:24.452663+0000 mds.d (mds.0) 24 : cluster [WRN] Scrub error on inode 0x10000000347 (/client.0/tmp/tmp) see mds.d log and `damage ls` output for details" in cluster log

pass 5735639 2020-12-24 22:49:23 2020-12-24 23:01:03 2020-12-25 00:35:04 1:34:01 0:14:14 1:19:47 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
pass 5735640 2020-12-24 22:49:24 2020-12-24 23:01:03 2020-12-25 00:11:04 1:10:01 0:14:14 0:55:47 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
pass 5735641 2020-12-24 22:49:25 2020-12-24 23:01:03 2020-12-24 23:33:03 0:32:00 0:24:54 0:07:06 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5735642 2020-12-24 22:49:26 2020-12-24 23:01:03 2020-12-25 00:09:04 1:08:01 0:14:35 0:53:26 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5735643 2020-12-24 22:49:26 2020-12-24 23:01:25 2020-12-25 00:45:27 1:44:02 0:21:37 1:22:25 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
fail 5735644 2020-12-24 22:49:27 2020-12-24 23:02:28 2020-12-25 00:40:29 1:38:01 0:32:57 1:05:04 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
Failure Reason:

The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}

pass 5735645 2020-12-24 22:49:28 2020-12-24 23:02:31 2020-12-24 23:24:31 0:22:00 0:15:45 0:06:15 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5735646 2020-12-24 22:49:29 2020-12-24 23:02:31 2020-12-24 23:36:31 0:34:00 0:15:15 0:18:45 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5735647 2020-12-24 22:49:29 2020-12-24 23:02:33 2020-12-25 01:00:35 1:58:02 0:15:05 1:42:57 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5735648 2020-12-24 22:49:30 2020-12-24 23:02:34 2020-12-25 00:08:34 1:06:00 0:11:56 0:54:04 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

Command failed (workunit test fs/misc/direct_io.py) on smithi144 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=33b60211b54317df021e1b30c3d0b85dc41a20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py'

pass 5735649 2020-12-24 22:49:31 2020-12-24 23:02:35 2020-12-25 01:16:37 2:14:02 1:41:09 0:32:53 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5735650 2020-12-24 22:49:32 2020-12-24 23:02:40 2020-12-25 00:26:41 1:24:01 0:52:08 0:31:53 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

"2020-12-24T23:44:06.213890+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x1000000061e (/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-5) see mds.f log and `damage ls` output for details" in cluster log

pass 5735651 2020-12-24 22:49:33 2020-12-24 23:03:17 2020-12-25 00:33:18 1:30:01 1:15:38 0:14:23 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5735652 2020-12-24 22:49:33 2020-12-24 23:04:54 2020-12-25 00:08:55 1:04:01 0:47:18 0:16:43 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

Command failed (workunit test suites/ffsb.sh) on smithi164 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=33b60211b54317df021e1b30c3d0b85dc41a20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

pass 5735653 2020-12-24 22:49:34 2020-12-24 23:04:54 2020-12-24 23:52:54 0:48:00 0:14:55 0:33:05 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5735654 2020-12-24 22:49:35 2020-12-24 23:04:54 2020-12-25 00:24:55 1:20:01 0:14:14 1:05:47 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2020-12-25T00:19:17.379657+0000 mds.a (mds.3) 2 : cluster [WRN] Scrub error on inode 0x100000011d2 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_hpi_testsuite/marshal) see mds.a log and `damage ls` output for details" in cluster log

pass 5735655 2020-12-24 22:49:36 2020-12-24 23:04:58 2020-12-24 23:48:58 0:44:00 0:25:38 0:18:22 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5735656 2020-12-24 22:49:36 2020-12-24 23:07:23 2020-12-24 23:47:23 0:40:00 0:12:36 0:27:24 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5735657 2020-12-24 22:49:37 2020-12-24 23:08:13 2020-12-25 00:02:13 0:54:00 0:19:14 0:34:46 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5735658 2020-12-24 22:49:38 2020-12-24 23:08:56 2020-12-25 00:26:57 1:18:01 1:05:56 0:12:05 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5735659 2020-12-24 22:49:39 2020-12-24 23:10:51 2020-12-24 23:48:51 0:38:00 0:15:55 0:22:05 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5735660 2020-12-24 22:49:40 2020-12-24 23:10:51 2020-12-24 23:44:51 0:34:00 0:13:08 0:20:52 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5735661 2020-12-24 22:49:40 2020-12-24 23:13:03 2020-12-24 23:39:02 0:25:59 0:12:41 0:13:18 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5735662 2020-12-24 22:49:41 2020-12-24 23:13:57 2020-12-24 23:43:57 0:30:00 0:11:03 0:18:57 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

Command failed (workunit test fs/misc/direct_io.py) on smithi146 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=33b60211b54317df021e1b30c3d0b85dc41a20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py'

pass 5735663 2020-12-24 22:49:42 2020-12-24 23:15:05 2020-12-25 00:41:06 1:26:01 0:35:55 0:50:06 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
pass 5735664 2020-12-24 22:49:43 2020-12-24 23:15:05 2020-12-25 00:37:06 1:22:01 0:28:00 0:54:01 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
pass 5735665 2020-12-24 22:49:43 2020-12-24 23:17:03 2020-12-25 00:15:03 0:58:00 0:49:14 0:08:46 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5735666 2020-12-24 22:49:44 2020-12-24 23:17:03 2020-12-25 01:09:04 1:52:01 0:49:30 1:02:31 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

Command failed (workunit test suites/ffsb.sh) on smithi103 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=33b60211b54317df021e1b30c3d0b85dc41a20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

pass 5735667 2020-12-24 22:49:45 2020-12-24 23:17:39 2020-12-25 00:01:39 0:44:00 0:15:48 0:28:12 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5735668 2020-12-24 22:49:46 2020-12-24 23:19:09 2020-12-24 23:47:09 0:28:00 0:16:42 0:11:18 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2020-12-24T23:39:47.861081+0000 mds.d (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000c24 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_hpi_testsuite) see mds.d log and `damage ls` output for details" in cluster log

pass 5735669 2020-12-24 22:49:47 2020-12-24 23:19:09 2020-12-25 00:23:10 1:04:01 0:37:57 0:26:04 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5735670 2020-12-24 22:49:48 2020-12-24 23:20:38 2020-12-25 00:02:38 0:42:00 0:10:55 0:31:05 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5735671 2020-12-24 22:49:48 2020-12-24 23:21:26 2020-12-25 00:13:27 0:52:01 0:23:49 0:28:12 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5735672 2020-12-24 22:49:49 2020-12-24 23:23:01 2020-12-25 00:35:02 1:12:01 0:53:34 0:18:27 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5735673 2020-12-24 22:49:50 2020-12-24 23:24:43 2020-12-24 23:52:43 0:28:00 0:14:09 0:13:51 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
fail 5735674 2020-12-24 22:49:51 2020-12-24 23:24:43 2020-12-25 00:20:44 0:56:01 0:11:45 0:44:16 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
Failure Reason:

"2020-12-25T00:18:05.708512+0000 mds.f (mds.0) 24 : cluster [WRN] Scrub error on inode 0x10000000488 (/client.0/tmp/tmp/fstest_8315c891fb2a368236ae9924df66ce69) see mds.f log and `damage ls` output for details" in cluster log

pass 5735675 2020-12-24 22:49:52 2020-12-24 23:24:43 2020-12-24 23:56:43 0:32:00 0:14:41 0:17:19 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
pass 5735676 2020-12-24 22:49:52 2020-12-24 23:26:56 2020-12-25 00:56:57 1:30:01 0:34:20 0:55:41 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
pass 5735677 2020-12-24 22:49:53 2020-12-24 23:29:11 2020-12-25 03:25:15 3:56:04 2:32:51 1:23:13 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5735678 2020-12-24 22:49:54 2020-12-24 23:31:20 2020-12-25 00:27:21 0:56:01 0:45:19 0:10:42 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

Command failed (workunit test suites/blogbench.sh) on smithi039 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=33b60211b54317df021e1b30c3d0b85dc41a20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh'

pass 5735679 2020-12-24 22:49:55 2020-12-24 23:31:20 2020-12-25 01:19:22 1:48:02 1:03:13 0:44:49 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5735680 2020-12-24 22:49:56 2020-12-24 23:32:58 2020-12-25 00:14:58 0:42:00 0:26:37 0:15:23 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

"2020-12-24T23:57:06.026112+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000348 (/client.0/tmp/tmp) see mds.f log and `damage ls` output for details" in cluster log

pass 5735681 2020-12-24 22:49:56 2020-12-24 23:32:58 2020-12-25 00:10:58 0:38:00 0:18:45 0:19:15 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
pass 5735682 2020-12-24 22:49:57 2020-12-24 23:33:04 2020-12-24 23:53:03 0:19:59 0:13:03 0:06:56 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
pass 5735683 2020-12-24 22:49:58 2020-12-24 23:35:00 2020-12-25 00:35:00 1:00:00 0:34:28 0:25:32 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5735684 2020-12-24 22:49:59 2020-12-24 23:35:00 2020-12-25 00:15:00 0:40:00 0:12:04 0:27:56 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5735685 2020-12-24 22:49:59 2020-12-24 23:36:55 2020-12-25 00:34:55 0:58:00 0:20:11 0:37:49 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5735686 2020-12-24 22:50:00 2020-12-24 23:36:55 2020-12-25 00:50:56 1:14:01 1:01:46 0:12:15 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5735687 2020-12-24 22:50:01 2020-12-24 23:36:55 2020-12-25 00:04:55 0:28:00 0:15:48 0:12:12 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5735688 2020-12-24 22:50:02 2020-12-24 23:36:55 2020-12-25 00:08:55 0:32:00 0:13:06 0:18:54 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5735689 2020-12-24 22:50:02 2020-12-24 23:39:24 2020-12-25 00:53:25 1:14:01 0:14:51 0:59:10 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5735690 2020-12-24 22:50:03 2020-12-24 23:39:25 2020-12-25 00:17:24 0:37:59 0:27:53 0:10:06 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

"2020-12-25T00:02:40.597303+0000 mds.c (mds.0) 39 : cluster [WRN] Scrub error on inode 0x1000000879d (/client.0/tmp/payload.1/multiple_rsync_payload.142449/locale) see mds.c log and `damage ls` output for details" in cluster log

pass 5735691 2020-12-24 22:50:04 2020-12-24 23:39:27 2020-12-25 00:47:27 1:08:00 0:55:35 0:12:25 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5735692 2020-12-24 22:50:05 2020-12-24 23:42:58 2020-12-25 00:38:58 0:56:00 0:38:06 0:17:54 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

Command failed (workunit test suites/blogbench.sh) on smithi139 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=33b60211b54317df021e1b30c3d0b85dc41a20ab TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh'

pass 5735693 2020-12-24 22:50:05 2020-12-24 23:42:58 2020-12-25 01:20:59 1:38:01 1:01:46 0:36:15 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
pass 5735694 2020-12-24 22:50:06 2020-12-24 23:42:58 2020-12-25 00:32:58 0:50:00 0:39:43 0:10:17 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
pass 5735695 2020-12-24 22:50:07 2020-12-24 23:44:23 2020-12-25 01:02:23 1:18:00 0:17:28 1:00:32 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5735696 2020-12-24 22:50:08 2020-12-24 23:44:26 2020-12-25 00:40:26 0:56:00 0:15:46 0:40:14 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2020-12-25T00:33:18.746524+0000 mds.d (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000001fb (/client.0/tmp/fsstress/ltp-full-20091231/testcases/ballista) see mds.d log and `damage ls` output for details" in cluster log

pass 5735697 2020-12-24 22:50:09 2020-12-24 23:44:52 2020-12-25 00:34:52 0:50:00 0:28:28 0:21:32 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5735698 2020-12-24 22:50:09 2020-12-24 23:46:54 2020-12-25 00:08:54 0:22:00 0:10:45 0:11:15 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5735699 2020-12-24 22:50:10 2020-12-24 23:46:56 2020-12-25 00:40:56 0:54:00 0:23:36 0:30:24 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5735700 2020-12-24 22:50:11 2020-12-24 23:47:07 2020-12-25 00:41:07 0:54:00 0:39:41 0:14:19 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5735701 2020-12-24 22:50:11 2020-12-24 23:47:10 2020-12-25 00:11:10 0:24:00 0:16:56 0:07:04 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5735702 2020-12-24 22:50:12 2020-12-24 23:47:24 2020-12-25 00:13:24 0:26:00 0:11:08 0:14:52 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3