Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 5757960 2021-01-06 00:07:56 2021-01-06 00:08:49 2021-01-06 00:26:49 0:18:00 0:10:56 0:07:04 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5757961 2021-01-06 00:07:57 2021-01-06 00:08:49 2021-01-06 01:14:50 1:06:01 0:55:11 0:10:50 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

"2021-01-06T00:36:46.853709+0000 mds.a (mds.0) 39 : cluster [WRN] Scrub error on inode 0x10000000264 (/client.0/tmp/testdir/dir1/dir2) see mds.a log and `damage ls` output for details" in cluster log

pass 5757962 2021-01-06 00:07:58 2021-01-06 00:08:50 2021-01-06 00:50:50 0:42:00 0:32:11 0:09:49 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
pass 5757963 2021-01-06 00:07:59 2021-01-06 00:10:17 2021-01-06 00:40:17 0:30:00 0:21:43 0:08:17 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
pass 5757964 2021-01-06 00:08:00 2021-01-06 00:10:17 2021-01-06 01:42:18 1:32:01 1:22:18 0:09:43 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5757965 2021-01-06 00:08:00 2021-01-06 00:10:17 2021-01-06 01:14:18 1:04:01 0:54:08 0:09:53 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

"2021-01-06T00:32:57.101825+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000347 (/client.0/tmp/tmp) see mds.f log and `damage ls` output for details" in cluster log

pass 5757966 2021-01-06 00:08:01 2021-01-06 00:12:28 2021-01-06 00:38:27 0:25:59 0:13:30 0:12:29 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5757967 2021-01-06 00:08:02 2021-01-06 00:12:28 2021-01-06 00:40:28 0:28:00 0:15:51 0:12:09 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2021-01-06T00:34:20.931220+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000012e0 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_posix_testsuite) see mds.f log and `damage ls` output for details" in cluster log

pass 5757968 2021-01-06 00:08:03 2021-01-06 00:12:28 2021-01-06 00:38:27 0:25:59 0:16:22 0:09:37 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5757969 2021-01-06 00:08:04 2021-01-06 00:14:26 2021-01-06 00:38:26 0:24:00 0:14:43 0:09:17 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5757970 2021-01-06 00:08:04 2021-01-06 00:14:26 2021-01-06 00:44:26 0:30:00 0:21:42 0:08:18 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5757971 2021-01-06 00:08:05 2021-01-06 00:14:30 2021-01-06 01:22:30 1:08:00 0:52:48 0:15:12 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5757972 2021-01-06 00:08:06 2021-01-06 00:16:41 2021-01-06 00:40:41 0:24:00 0:11:56 0:12:04 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5757973 2021-01-06 00:08:07 2021-01-06 00:16:41 2021-01-06 00:58:41 0:42:00 0:16:25 0:25:35 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5757974 2021-01-06 00:08:08 2021-01-06 00:16:41 2021-01-06 00:42:41 0:26:00 0:12:08 0:13:52 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5757975 2021-01-06 00:08:08 2021-01-06 00:16:42 2021-01-06 01:44:43 1:28:01 1:14:27 0:13:34 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

"2021-01-06T00:54:15.278068+0000 mds.f (mds.0) 144 : cluster [WRN] Scrub error on inode 0x10000017cf0 (/client.0/tmp/payload.2/multiple_rsync_payload.149884/python3.6/site-packages) see mds.f log and `damage ls` output for details" in cluster log

pass 5757976 2021-01-06 00:08:09 2021-01-06 00:18:31 2021-01-06 00:54:30 0:35:59 0:28:26 0:07:33 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5757977 2021-01-06 00:08:10 2021-01-06 00:18:31 2021-01-06 01:12:31 0:54:00 0:41:07 0:12:53 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

"2021-01-06T00:44:06.109225+0000 mds.d (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000001fe (/client.0/tmp/blogbench-1.0/src) see mds.d log and `damage ls` output for details" in cluster log

pass 5757978 2021-01-06 00:08:11 2021-01-06 00:20:04 2021-01-06 01:24:05 1:04:01 0:50:19 0:13:42 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5757979 2021-01-06 00:08:11 2021-01-06 00:20:15 2021-01-06 01:50:16 1:30:01 1:13:36 0:16:25 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

"2021-01-06T00:46:37.412281+0000 mds.d (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000347 (/client.0/tmp/tmp) see mds.d log and `damage ls` output for details" in cluster log

pass 5757980 2021-01-06 00:08:12 2021-01-06 00:20:15 2021-01-06 01:14:15 0:54:00 0:11:36 0:42:24 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
pass 5757981 2021-01-06 00:08:13 2021-01-06 00:20:25 2021-01-06 01:18:25 0:58:00 0:20:38 0:37:22 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
pass 5757982 2021-01-06 00:08:14 2021-01-06 00:20:25 2021-01-06 01:02:25 0:42:00 0:22:45 0:19:15 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5757983 2021-01-06 00:08:14 2021-01-06 00:22:40 2021-01-06 00:56:40 0:34:00 0:23:21 0:10:39 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5757984 2021-01-06 00:08:15 2021-01-06 00:22:40 2021-01-06 00:50:40 0:28:00 0:20:22 0:07:38 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5757985 2021-01-06 00:08:16 2021-01-06 00:24:25 2021-01-06 02:30:27 2:06:02 1:00:16 1:05:46 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5757986 2021-01-06 00:08:17 2021-01-06 00:24:26 2021-01-06 01:02:25 0:37:59 0:16:09 0:21:50 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5757987 2021-01-06 00:08:17 2021-01-06 00:24:26 2021-01-06 01:06:25 0:41:59 0:12:31 0:29:28 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5757988 2021-01-06 00:08:18 2021-01-06 00:24:26 2021-01-06 00:48:25 0:23:59 0:12:14 0:11:45 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5757989 2021-01-06 00:08:19 2021-01-06 00:24:54 2021-01-06 01:18:55 0:54:01 0:42:27 0:11:34 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

"2021-01-06T00:49:40.624164+0000 mds.f (mds.0) 39 : cluster [WRN] Scrub error on inode 0x10000000263 (/client.0/tmp/testdir/dir1) see mds.f log and `damage ls` output for details" in cluster log

pass 5757990 2021-01-06 00:08:20 2021-01-06 00:26:25 2021-01-06 01:24:25 0:58:00 0:46:47 0:11:13 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5757991 2021-01-06 00:08:21 2021-01-06 00:26:25 2021-01-06 01:18:25 0:52:00 0:24:12 0:27:48 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

"2021-01-06T01:03:14.286827+0000 mds.b (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000008da (/client.0/tmp/blogbench-1.0/src/blogtest_in/blog-8) see mds.b log and `damage ls` output for details" in cluster log

pass 5757992 2021-01-06 00:08:21 2021-01-06 00:26:25 2021-01-06 01:58:26 1:32:01 1:23:40 0:08:21 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
pass 5757993 2021-01-06 00:08:22 2021-01-06 00:26:27 2021-01-06 01:12:27 0:46:00 0:29:29 0:16:31 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
pass 5757994 2021-01-06 00:08:23 2021-01-06 00:26:50 2021-01-06 01:04:50 0:38:00 0:14:49 0:23:11 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5757995 2021-01-06 00:08:24 2021-01-06 00:28:49 2021-01-06 00:52:49 0:24:00 0:14:21 0:09:39 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2021-01-06T00:46:47.053468+0000 mds.b (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000004d9 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/network) see mds.b log and `damage ls` output for details" in cluster log

pass 5757996 2021-01-06 00:08:24 2021-01-06 00:29:20 2021-01-06 01:07:19 0:37:59 0:21:54 0:16:05 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5757997 2021-01-06 00:08:25 2021-01-06 00:30:36 2021-01-06 01:14:34 0:43:58 0:15:13 0:28:45 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5757998 2021-01-06 00:08:26 2021-01-06 00:30:35 2021-01-06 01:02:33 0:31:58 0:22:00 0:09:58 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5757999 2021-01-06 00:08:27 2021-01-06 00:30:35 2021-01-06 02:00:35 1:30:00 0:47:45 0:42:15 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5758000 2021-01-06 00:08:27 2021-01-06 00:30:36 2021-01-06 00:52:33 0:21:57 0:12:22 0:09:35 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
fail 5758001 2021-01-06 00:08:28 2021-01-06 00:32:46 2021-01-06 01:36:46 1:04:00 0:16:23 0:47:37 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
Failure Reason:

Command failed (workunit test suites/pjd.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0ed162b51928c50f20cee111f8292828eda755e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

pass 5758002 2021-01-06 00:08:29 2021-01-06 00:32:46 2021-01-06 01:22:46 0:50:00 0:13:11 0:36:49 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5758003 2021-01-06 00:08:30 2021-01-06 00:32:46 2021-01-06 01:48:47 1:16:01 1:01:51 0:14:10 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi204 with status 23: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0ed162b51928c50f20cee111f8292828eda755e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh'

pass 5758004 2021-01-06 00:08:31 2021-01-06 00:32:46 2021-01-06 01:20:46 0:48:00 0:40:02 0:07:58 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
pass 5758005 2021-01-06 00:08:31 2021-01-06 00:32:46 2021-01-06 01:16:46 0:44:00 0:23:21 0:20:39 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
pass 5758006 2021-01-06 00:08:32 2021-01-06 00:33:07 2021-01-06 02:03:08 1:30:01 1:01:51 0:28:10 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5758007 2021-01-06 00:08:33 2021-01-06 00:34:41 2021-01-06 01:40:42 1:06:01 0:50:04 0:15:57 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

"2021-01-06T01:00:34.786495+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000347 (/client.0/tmp/tmp) see mds.f log and `damage ls` output for details" in cluster log

pass 5758008 2021-01-06 00:08:34 2021-01-06 00:34:41 2021-01-06 00:56:41 0:22:00 0:12:25 0:09:35 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5758009 2021-01-06 00:08:35 2021-01-06 00:34:55 2021-01-06 01:06:55 0:32:00 0:17:45 0:14:15 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2021-01-06T01:00:38.384223+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000030ba (/client.0/tmp/tmp.4FavzLeTAO/p7) see mds.f log and `damage ls` output for details" in cluster log

pass 5758010 2021-01-06 00:08:35 2021-01-06 00:38:52 2021-01-06 01:20:52 0:42:00 0:31:21 0:10:39 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5758011 2021-01-06 00:08:36 2021-01-06 00:38:52 2021-01-06 01:24:52 0:46:00 0:12:39 0:33:21 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5758012 2021-01-06 00:08:37 2021-01-06 00:38:52 2021-01-06 01:20:52 0:42:00 0:20:26 0:21:34 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5758013 2021-01-06 00:08:38 2021-01-06 00:40:43 2021-01-06 01:40:43 1:00:00 0:44:48 0:15:12 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5758014 2021-01-06 00:08:39 2021-01-06 00:40:43 2021-01-06 01:12:42 0:31:59 0:17:00 0:14:59 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5758015 2021-01-06 00:08:39 2021-01-06 00:40:43 2021-01-06 01:22:43 0:42:00 0:12:00 0:30:00 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5758016 2021-01-06 00:08:40 2021-01-06 00:40:43 2021-01-06 01:22:43 0:42:00 0:12:56 0:29:04 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5758017 2021-01-06 00:08:41 2021-01-06 00:42:27 2021-01-06 02:08:28 1:26:01 0:48:14 0:37:47 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

"2021-01-06T01:38:46.654407+0000 mds.c (mds.0) 109 : cluster [WRN] Scrub error on inode 0x100000166c1 (/client.0/tmp/payload.2/multiple_rsync_payload.146944/python3.6/site-packages) see mds.c log and `damage ls` output for details" in cluster log

pass 5758018 2021-01-06 00:08:42 2021-01-06 00:42:28 2021-01-06 02:30:29 1:48:01 0:36:58 1:11:03 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5758019 2021-01-06 00:08:43 2021-01-06 00:42:42 2021-01-06 01:56:43 1:14:01 0:31:28 0:42:33 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

"2021-01-06T01:34:19.107259+0000 mds.a (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000001fa (/client.0/tmp/blogbench-1.0/man) see mds.a log and `damage ls` output for details" in cluster log

pass 5758020 2021-01-06 00:08:43 2021-01-06 00:44:55 2021-01-06 01:48:55 1:04:00 0:52:32 0:11:28 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
pass 5758021 2021-01-06 00:08:44 2021-01-06 00:47:03 2021-01-06 02:07:03 1:20:00 0:55:43 0:24:17 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
pass 5758022 2021-01-06 00:08:45 2021-01-06 00:47:29 2021-01-06 01:15:28 0:27:59 0:16:40 0:11:19 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
pass 5758023 2021-01-06 00:08:46 2021-01-06 00:48:41 2021-01-06 01:56:41 1:08:00 0:15:54 0:52:06 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
pass 5758024 2021-01-06 00:08:47 2021-01-06 00:48:41 2021-01-06 01:58:41 1:10:00 0:46:57 0:23:03 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5758025 2021-01-06 00:08:47 2021-01-06 00:48:47 2021-01-06 01:10:47 0:22:00 0:11:42 0:10:18 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5758026 2021-01-06 00:08:48 2021-01-06 00:50:52 2021-01-06 01:34:52 0:44:00 0:23:36 0:20:24 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5758027 2021-01-06 00:08:49 2021-01-06 00:50:52 2021-01-06 02:06:53 1:16:01 1:02:59 0:13:02 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5758028 2021-01-06 00:08:50 2021-01-06 00:50:52 2021-01-06 01:44:53 0:54:01 0:11:30 0:42:31 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5758029 2021-01-06 00:08:51 2021-01-06 00:53:00 2021-01-06 01:33:00 0:40:00 0:12:05 0:27:55 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5758030 2021-01-06 00:08:52 2021-01-06 00:53:00 2021-01-06 01:37:00 0:44:00 0:14:11 0:29:49 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5758031 2021-01-06 00:08:52 2021-01-06 00:53:50 2021-01-06 05:29:55 4:36:05 3:26:03 1:10:02 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi198 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0ed162b51928c50f20cee111f8292828eda755e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh'

pass 5758032 2021-01-06 00:08:53 2021-01-06 00:54:58 2021-01-06 03:05:00 2:10:02 1:56:02 0:14:00 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5758033 2021-01-06 00:08:54 2021-01-06 00:57:02 2021-01-06 02:53:04 1:56:02 0:39:06 1:16:56 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

"2021-01-06T02:22:15.034612+0000 mds.e (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000001fe (/client.0/tmp/blogbench-1.0/src) see mds.e log and `damage ls` output for details" in cluster log

pass 5758034 2021-01-06 00:08:55 2021-01-06 00:57:02 2021-01-06 02:37:04 1:40:02 1:03:52 0:36:10 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5758035 2021-01-06 00:08:55 2021-01-06 00:59:08 2021-01-06 02:03:09 1:04:01 0:48:55 0:15:06 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

"2021-01-06T01:28:19.943061+0000 mon.c (mon.0) 314 : cluster [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster log

pass 5758036 2021-01-06 00:08:56 2021-01-06 00:59:53 2021-01-06 01:37:54 0:38:01 0:17:27 0:20:34 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5758037 2021-01-06 00:08:57 2021-01-06 01:00:44 2021-01-06 01:30:44 0:30:00 0:14:55 0:15:05 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2021-01-06T01:24:49.976026+0000 mds.c (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000030a5 (/client.0/tmp/tmp.sUMR2sJDr7) see mds.c log and `damage ls` output for details" in cluster log

pass 5758038 2021-01-06 00:08:58 2021-01-06 01:02:51 2021-01-06 02:00:52 0:58:01 0:28:34 0:29:27 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5758039 2021-01-06 00:08:58 2021-01-06 01:02:51 2021-01-06 01:34:51 0:32:00 0:13:21 0:18:39 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5758040 2021-01-06 00:08:59 2021-01-06 01:02:51 2021-01-06 02:08:52 1:06:01 0:20:27 0:45:34 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5758041 2021-01-06 00:09:00 2021-01-06 01:03:03 2021-01-06 02:25:03 1:22:00 1:01:50 0:20:10 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5758042 2021-01-06 00:09:01 2021-01-06 01:05:10 2021-01-06 01:33:10 0:28:00 0:18:24 0:09:36 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
fail 5758043 2021-01-06 00:09:01 2021-01-06 01:06:51 2021-01-06 02:24:52 1:18:01 0:13:58 1:04:03 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
Failure Reason:

"2021-01-06T02:20:50.195229+0000 mds.f (mds.0) 34 : cluster [WRN] Scrub error on inode 0x10000000503 (/client.0/tmp/tmp/fstest_faf672f27bd90e88f53ce1207f3f9d6c) see mds.f log and `damage ls` output for details" in cluster log

pass 5758044 2021-01-06 00:09:02 2021-01-06 01:06:56 2021-01-06 01:36:56 0:30:00 0:14:20 0:15:40 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5758045 2021-01-06 00:09:03 2021-01-06 01:07:07 2021-01-06 01:51:07 0:44:00 0:28:08 0:15:52 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

"2021-01-06T01:33:32.070596+0000 mds.f (mds.0) 34 : cluster [WRN] Scrub error on inode 0x10000000263 (/client.0/tmp/testdir/dir1) see mds.f log and `damage ls` output for details" in cluster log

pass 5758046 2021-01-06 00:09:04 2021-01-06 01:07:20 2021-01-06 02:21:21 1:14:01 0:49:12 0:24:49 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
pass 5758047 2021-01-06 00:09:05 2021-01-06 01:10:29 2021-01-06 01:38:29 0:28:00 0:19:37 0:08:23 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
pass 5758048 2021-01-06 00:09:06 2021-01-06 01:10:48 2021-01-06 02:26:49 1:16:01 1:08:17 0:07:44 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
pass 5758049 2021-01-06 00:09:06 2021-01-06 01:12:54 2021-01-06 03:28:56 2:16:02 0:30:16 1:45:46 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
pass 5758050 2021-01-06 00:09:07 2021-01-06 01:12:54 2021-01-06 02:18:55 1:06:01 0:16:58 0:49:03 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5758051 2021-01-06 00:09:08 2021-01-06 01:12:54 2021-01-06 02:16:55 1:04:01 0:16:24 0:47:37 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2021-01-06T02:10:58.190500+0000 mds.d (mds.0) 24 : cluster [WRN] Scrub error on inode 0x100000012e0 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/open_posix_testsuite) see mds.d log and `damage ls` output for details" in cluster log

pass 5758052 2021-01-06 00:09:09 2021-01-06 01:13:49 2021-01-06 02:11:49 0:58:00 0:45:29 0:12:31 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5758053 2021-01-06 00:09:10 2021-01-06 01:14:44 2021-01-06 01:34:44 0:20:00 0:11:11 0:08:49 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5758054 2021-01-06 00:09:11 2021-01-06 01:14:42 2021-01-06 02:08:43 0:54:01 0:24:20 0:29:41 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5758055 2021-01-06 00:09:11 2021-01-06 01:14:43 2021-01-06 02:52:44 1:38:01 0:37:31 1:00:30 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5758056 2021-01-06 00:09:12 2021-01-06 01:14:51 2021-01-06 02:20:52 1:06:01 0:13:09 0:52:52 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
fail 5758057 2021-01-06 00:09:13 2021-01-06 01:15:30 2021-01-06 01:37:29 0:21:59 0:12:00 0:09:59 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
Failure Reason:

"2021-01-06T01:33:27.478525+0000 mds.c (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000002cd (/client.0/tmp/tmp) see mds.c log and `damage ls` output for details" in cluster log

pass 5758058 2021-01-06 00:09:14 2021-01-06 01:17:15 2021-01-06 02:29:16 1:12:01 0:14:53 0:57:08 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
dead 5758059 2021-01-06 00:09:14 2021-01-06 01:18:53 2021-01-06 01:46:52 0:27:59 0:03:44 0:24:15 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

{'Failure object was': {'smithi120.front.sepia.ceph.com': {'ansible_facts': {}, 'msg': 'The following modules failed to execute: setup\\n', 'failed_modules': {'setup': {'failed': True, 'module_stdout': '', 'module_stderr': '', 'msg': 'MODULE FAILURE\\nSee stdout/stderr for the exact error', 'rc': -13}}, '_ansible_verbose_override': True, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '')"}

pass 5758060 2021-01-06 00:09:15 2021-01-06 01:18:53 2021-01-06 04:46:56 3:28:03 2:40:29 0:47:34 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5758061 2021-01-06 00:09:16 2021-01-06 01:18:56 2021-01-06 02:12:56 0:54:00 0:45:38 0:08:22 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

Command failed on smithi071 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'

pass 5758062 2021-01-06 00:09:17 2021-01-06 01:21:13 2021-01-06 02:39:14 1:18:01 1:06:50 0:11:11 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
pass 5758063 2021-01-06 00:09:18 2021-01-06 01:21:13 2021-01-06 02:21:14 1:00:01 0:38:22 0:21:39 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
pass 5758064 2021-01-06 00:09:18 2021-01-06 01:21:13 2021-01-06 02:33:14 1:12:01 0:17:03 0:54:58 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
pass 5758065 2021-01-06 00:09:19 2021-01-06 01:22:59 2021-01-06 01:50:58 0:27:59 0:14:43 0:13:16 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
pass 5758066 2021-01-06 00:09:20 2021-01-06 01:22:59 2021-01-06 02:08:58 0:45:59 0:35:33 0:10:26 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5758067 2021-01-06 00:09:21 2021-01-06 01:22:59 2021-01-06 01:58:58 0:35:59 0:13:02 0:22:57 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5758068 2021-01-06 00:09:21 2021-01-06 01:22:59 2021-01-06 01:54:58 0:31:59 0:19:59 0:12:00 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5758069 2021-01-06 00:09:22 2021-01-06 01:24:33 2021-01-06 02:36:33 1:12:00 0:57:26 0:14:34 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5758070 2021-01-06 00:09:23 2021-01-06 01:24:33 2021-01-06 03:36:35 2:12:02 0:19:34 1:52:28 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5758071 2021-01-06 00:09:24 2021-01-06 01:24:53 2021-01-06 02:50:54 1:26:01 0:13:46 1:12:15 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5758072 2021-01-06 00:09:25 2021-01-06 01:27:04 2021-01-06 01:59:03 0:31:59 0:11:47 0:20:12 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5758073 2021-01-06 00:09:25 2021-01-06 01:31:14 2021-01-06 02:29:14 0:58:00 0:43:54 0:14:06 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi192 with status 23: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0ed162b51928c50f20cee111f8292828eda755e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh'

pass 5758074 2021-01-06 00:09:26 2021-01-06 01:33:21 2021-01-06 02:11:21 0:38:00 0:28:03 0:09:57 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5758075 2021-01-06 00:09:27 2021-01-06 01:33:21 2021-01-06 02:41:21 1:08:00 0:40:26 0:27:34 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

"2021-01-06T02:12:36.246646+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000001fe (/client.0/tmp/blogbench-1.0/src) see mds.f log and `damage ls` output for details" in cluster log

pass 5758076 2021-01-06 00:09:28 2021-01-06 01:33:55 2021-01-06 02:59:56 1:26:01 1:04:10 0:21:51 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5758077 2021-01-06 00:09:28 2021-01-06 01:35:12 2021-01-06 02:57:12 1:22:00 1:08:39 0:13:21 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

"2021-01-06T02:08:44.714241+0000 mon.a (mon.0) 292 : cluster [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO)" in cluster log

pass 5758078 2021-01-06 00:09:29 2021-01-06 01:35:12 2021-01-06 01:55:11 0:19:59 0:11:56 0:08:03 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5758079 2021-01-06 00:09:30 2021-01-06 01:35:12 2021-01-06 02:05:11 0:29:59 0:19:18 0:10:41 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2021-01-06T01:59:13.218617+0000 mds.a (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000b97 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/mce-test) see mds.a log and `damage ls` output for details" in cluster log

pass 5758080 2021-01-06 00:09:31 2021-01-06 01:37:15 2021-01-06 02:29:15 0:52:00 0:22:58 0:29:02 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5758081 2021-01-06 00:09:32 2021-01-06 01:37:15 2021-01-06 02:11:15 0:34:00 0:23:25 0:10:35 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5758082 2021-01-06 00:09:32 2021-01-06 01:37:15 2021-01-06 02:05:15 0:28:00 0:19:56 0:08:04 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5758083 2021-01-06 00:09:33 2021-01-06 01:37:30 2021-01-06 03:31:32 1:54:02 1:00:53 0:53:09 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5758084 2021-01-06 00:09:34 2021-01-06 01:37:56 2021-01-06 02:07:55 0:29:59 0:13:14 0:16:45 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5758085 2021-01-06 00:09:35 2021-01-06 01:38:56 2021-01-06 02:54:57 1:16:01 0:11:53 1:04:08 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5758086 2021-01-06 00:09:36 2021-01-06 01:41:09 2021-01-06 02:05:09 0:24:00 0:10:24 0:13:36 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5758087 2021-01-06 00:09:36 2021-01-06 01:41:09 2021-01-06 05:57:14 4:16:05 3:27:38 0:48:27 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi195 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0ed162b51928c50f20cee111f8292828eda755e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh'

pass 5758088 2021-01-06 00:09:37 2021-01-06 01:42:49 2021-01-06 02:34:49 0:52:00 0:32:02 0:19:58 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
pass 5758089 2021-01-06 00:09:38 2021-01-06 01:45:13 2021-01-06 02:27:12 0:41:59 0:18:12 0:23:47 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
pass 5758090 2021-01-06 00:09:39 2021-01-06 01:45:12 2021-01-06 02:55:13 1:10:01 1:00:59 0:09:02 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
pass 5758091 2021-01-06 00:09:40 2021-01-06 01:47:25 2021-01-06 03:09:26 1:22:01 1:00:25 0:21:36 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
pass 5758092 2021-01-06 00:09:40 2021-01-06 01:49:16 2021-01-06 02:29:16 0:40:00 0:13:42 0:26:18 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5758093 2021-01-06 00:09:41 2021-01-06 01:49:16 2021-01-06 02:23:15 0:33:59 0:17:12 0:16:47 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2021-01-06T02:14:49.526043+0000 mds.b (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000001f22 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel) see mds.b log and `damage ls` output for details" in cluster log

pass 5758094 2021-01-06 00:09:42 2021-01-06 01:50:44 2021-01-06 02:18:44 0:28:00 0:16:32 0:11:28 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5758095 2021-01-06 00:09:43 2021-01-06 01:50:59 2021-01-06 02:20:59 0:30:00 0:15:05 0:14:55 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5758096 2021-01-06 00:09:43 2021-01-06 01:51:09 2021-01-06 02:39:11 0:48:02 0:21:51 0:26:11 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
fail 5758097 2021-01-06 00:09:44 2021-01-06 01:55:29 2021-01-06 03:07:30 1:12:01 0:50:52 0:21:09 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
Failure Reason:

The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}

pass 5758098 2021-01-06 00:09:45 2021-01-06 01:55:29 2021-01-06 03:21:30 1:26:01 0:13:57 1:12:04 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
fail 5758099 2021-01-06 00:09:46 2021-01-06 01:56:59 2021-01-06 02:30:59 0:34:00 0:17:41 0:16:19 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
Failure Reason:

Command failed (workunit test suites/pjd.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d0ed162b51928c50f20cee111f8292828eda755e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

pass 5758100 2021-01-06 00:09:47 2021-01-06 01:56:59 2021-01-06 02:30:59 0:34:00 0:12:34 0:21:26 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5758101 2021-01-06 00:09:47 2021-01-06 01:56:59 2021-01-06 03:05:01 1:08:02 0:52:18 0:15:44 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

"2021-01-06T02:49:39.547892+0000 mds.f (mds.0) 284 : cluster [WRN] Scrub error on inode 0x10000013d68 (/client.0/tmp/payload.2/multiple_rsync_payload.126544/python3/dist-packages) see mds.f log and `damage ls` output for details" in cluster log

pass 5758102 2021-01-06 00:09:48 2021-01-06 01:58:54 2021-01-06 02:48:54 0:50:00 0:41:47 0:08:13 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5758103 2021-01-06 00:09:49 2021-01-06 01:58:54 2021-01-06 03:16:55 1:18:01 0:36:38 0:41:23 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

"2021-01-06T02:51:49.708813+0000 mds.a (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000001fa (/client.0/tmp/blogbench-1.0/man) see mds.a log and `damage ls` output for details" in cluster log

pass 5758104 2021-01-06 00:09:50 2021-01-06 01:59:00 2021-01-06 03:17:00 1:18:00 1:00:17 0:17:43 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
fail 5758105 2021-01-06 00:09:51 2021-01-06 01:59:05 2021-01-06 03:05:05 1:06:00 0:31:21 0:34:39 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
Failure Reason:

"2021-01-06T02:41:35.835879+0000 mds.f (mds.0) 19 : cluster [WRN] Scrub error on inode 0x10000000347 (/client.0/tmp/tmp) see mds.f log and `damage ls` output for details" in cluster log

pass 5758106 2021-01-06 00:09:52 2021-01-06 01:59:34 2021-01-06 03:09:35 1:10:01 0:13:53 0:56:08 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
pass 5758107 2021-01-06 00:09:52 2021-01-06 02:00:44 2021-01-06 02:48:45 0:48:01 0:19:09 0:28:52 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
pass 5758108 2021-01-06 00:09:53 2021-01-06 02:00:45 2021-01-06 03:00:45 1:00:00 0:29:33 0:30:27 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5758109 2021-01-06 00:09:54 2021-01-06 02:00:53 2021-01-06 02:30:52 0:29:59 0:12:10 0:17:49 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5758110 2021-01-06 00:09:55 2021-01-06 02:03:35 2021-01-06 02:55:35 0:52:00 0:20:25 0:31:35 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5758111 2021-01-06 00:09:55 2021-01-06 02:03:35 2021-01-06 03:17:36 1:14:01 0:35:19 0:38:42 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5758112 2021-01-06 00:09:56 2021-01-06 02:05:35 2021-01-06 02:33:35 0:28:00 0:13:53 0:14:07 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5758113 2021-01-06 00:09:57 2021-01-06 02:05:36 2021-01-06 02:27:35 0:21:59 0:11:18 0:10:41 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3
pass 5758114 2021-01-06 00:09:58 2021-01-06 02:05:35 2021-01-06 02:49:35 0:44:00 0:11:27 0:32:33 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/direct_io}} 3
fail 5758115 2021-01-06 00:09:59 2021-01-06 02:07:20 2021-01-06 03:09:21 1:02:01 0:40:35 0:21:26 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/fs/misc}} 3
Failure Reason:

"2021-01-06T02:41:27.866453+0000 mds.e (mds.3) 2 : cluster [WRN] Scrub error on inode 0x10000000266 (/client.0/tmp/testdir/dir1/dir2/dir3/dir4) see mds.e log and `damage ls` output for details" in cluster log

pass 5758116 2021-01-06 00:09:59 2021-01-06 02:07:20 2021-01-06 03:09:22 1:02:02 0:48:24 0:13:38 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/kernel_untar_build}} 3
fail 5758117 2021-01-06 00:10:00 2021-01-06 02:08:03 2021-01-06 02:48:06 0:40:03 0:20:00 0:20:03 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/blogbench}} 3
Failure Reason:

"2021-01-06T02:35:36.377225+0000 mds.c (mds.0) 19 : cluster [WRN] Scrub error on inode 0x100000001fa (/client.0/tmp/blogbench-1.0/man) see mds.c log and `damage ls` output for details" in cluster log

pass 5758118 2021-01-06 00:10:01 2021-01-06 02:08:57 2021-01-06 04:14:58 2:06:01 1:17:23 0:48:38 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/suites/dbench}} 3
pass 5758119 2021-01-06 00:10:02 2021-01-06 02:08:57 2021-01-06 03:14:57 1:06:00 0:41:25 0:24:35 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/ffsb}} 3
pass 5758120 2021-01-06 00:10:03 2021-01-06 02:08:57 2021-01-06 02:52:57 0:44:00 0:14:30 0:29:30 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/fs/norstats}} 3
fail 5758121 2021-01-06 00:10:03 2021-01-06 02:09:00 2021-01-06 02:30:59 0:21:59 0:14:22 0:07:37 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/fsstress}} 3
Failure Reason:

"2021-01-06T02:24:49.260374+0000 mds.a (mds.4) 2 : cluster [WRN] Scrub error on inode 0x10000002178 (/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/containers) see mds.a log and `damage ls` output for details" in cluster log

pass 5758122 2021-01-06 00:10:04 2021-01-06 02:11:42 2021-01-06 02:59:42 0:48:00 0:24:05 0:23:55 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no tasks/{0-check-counter workunit/suites/fsx}} 3
pass 5758123 2021-01-06 00:10:05 2021-01-06 02:11:42 2021-01-06 02:51:42 0:40:00 0:15:31 0:24:29 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes tasks/{0-check-counter workunit/suites/fsync-tester}} 3
pass 5758124 2021-01-06 00:10:06 2021-01-06 02:11:50 2021-01-06 03:01:51 0:50:01 0:21:42 0:28:19 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no tasks/{0-check-counter workunit/fs/test_o_trunc}} 3
pass 5758125 2021-01-06 00:10:07 2021-01-06 02:13:22 2021-01-06 02:59:22 0:46:00 0:29:20 0:16:40 smithi master ubuntu 18.04 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes tasks/{0-check-counter workunit/suites/iogen}} 3
pass 5758126 2021-01-06 00:10:08 2021-01-06 02:16:56 2021-01-06 02:40:55 0:23:59 0:15:38 0:08:21 smithi master rhel 8.3 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}} 3
pass 5758127 2021-01-06 00:10:09 2021-01-06 02:19:12 2021-01-06 03:05:12 0:46:00 0:15:49 0:30:11 smithi master centos 8.2 fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes tasks/{0-check-counter workunit/suites/pjd}} 3