Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 6288729 2021-07-23 15:30:23 2021-07-23 15:31:08 2021-07-23 16:18:43 0:47:35 0:36:34 0:11:01 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
pass 6288730 2021-07-23 15:30:24 2021-07-23 15:31:08 2021-07-23 16:50:15 1:19:07 1:07:01 0:12:06 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
pass 6288731 2021-07-23 15:30:25 2021-07-23 15:31:09 2021-07-23 18:35:10 3:04:01 2:56:34 0:07:27 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
pass 6288732 2021-07-23 15:30:26 2021-07-23 15:31:09 2021-07-23 16:31:02 0:59:53 0:49:07 0:10:46 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
pass 6288733 2021-07-23 15:30:27 2021-07-23 15:31:09 2021-07-23 17:12:56 1:41:47 1:26:44 0:15:03 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} 3
pass 6288734 2021-07-23 15:30:28 2021-07-23 15:31:09 2021-07-23 17:06:36 1:35:27 1:24:41 0:10:46 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} 3
pass 6288735 2021-07-23 15:30:29 2021-07-23 15:31:10 2021-07-23 16:18:11 0:47:01 0:35:46 0:11:15 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 6288736 2021-07-23 15:30:30 2021-07-23 15:31:11 2021-07-23 16:23:52 0:52:41 0:36:27 0:16:14 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
pass 6288737 2021-07-23 15:30:31 2021-07-23 15:31:12 2021-07-23 16:37:54 1:06:42 0:56:30 0:10:12 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} 3
pass 6288738 2021-07-23 15:30:32 2021-07-23 15:31:12 2021-07-23 16:15:53 0:44:41 0:34:04 0:10:37 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 6288739 2021-07-23 15:30:33 2021-07-23 15:31:12 2021-07-23 16:31:12 1:00:00 0:44:14 0:15:46 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
fail 6288740 2021-07-23 15:30:34 2021-07-23 15:31:13 2021-07-23 16:20:07 0:48:54 0:40:12 0:08:42 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
Failure Reason:

The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}

pass 6288741 2021-07-23 15:30:35 2021-07-23 15:31:14 2021-07-23 16:17:46 0:46:32 0:35:51 0:10:41 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
pass 6288742 2021-07-23 15:30:36 2021-07-23 15:31:14 2021-07-23 16:24:56 0:53:42 0:36:26 0:17:16 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} 3
pass 6288743 2021-07-23 15:30:37 2021-07-23 15:31:15 2021-07-23 16:17:35 0:46:20 0:36:07 0:10:13 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
pass 6288744 2021-07-23 15:30:38 2021-07-23 15:31:15 2021-07-23 16:48:13 1:16:58 1:06:46 0:10:12 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
pass 6288745 2021-07-23 15:30:39 2021-07-23 15:31:15 2021-07-23 16:43:55 1:12:40 0:57:25 0:15:15 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
fail 6288746 2021-07-23 15:30:40 2021-07-23 15:31:16 2021-07-23 21:10:03 5:38:47 5:28:01 0:10:46 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
Failure Reason:

Command failed (workunit test suites/blogbench.sh) on smithi039 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=555dae6fd9f5e3e09be0e33786ecd00be413585b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh'

pass 6288747 2021-07-23 15:30:41 2021-07-23 15:31:17 2021-07-23 16:41:08 1:09:51 1:02:47 0:07:04 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 6288748 2021-07-23 15:30:42 2021-07-23 15:31:17 2021-07-23 17:04:47 1:33:30 1:17:12 0:16:18 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
pass 6288749 2021-07-23 15:30:43 2021-07-23 15:31:18 2021-07-23 16:19:53 0:48:35 0:39:11 0:09:24 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 6288750 2021-07-23 15:30:44 2021-07-23 15:31:18 2021-07-23 16:20:55 0:49:37 0:39:31 0:10:06 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
pass 6288751 2021-07-23 15:30:45 2021-07-23 15:31:18 2021-07-23 16:25:46 0:54:28 0:39:27 0:15:01 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} 3
pass 6288752 2021-07-23 15:30:45 2021-07-23 15:31:19 2021-07-23 16:22:41 0:51:22 0:40:44 0:10:38 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 6288753 2021-07-23 15:30:46 2021-07-23 15:31:19 2021-07-23 16:21:51 0:50:32 0:41:49 0:08:43 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
pass 6288754 2021-07-23 15:30:47 2021-07-23 15:31:19 2021-07-23 17:06:36 1:35:17 1:18:51 0:16:26 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
fail 6288755 2021-07-23 15:30:48 2021-07-23 15:31:19 2021-07-23 16:01:01 0:29:42 0:19:53 0:09:49 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
Failure Reason:

Command failed on smithi155 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:dc6e6aee83d05e539da09c20751974c378bcb8fa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid dd3680ba-ebcd-11eb-8c23-001a4aab830c -- ceph orch daemon add osd smithi155:vg_nvme/lv_3'

pass 6288756 2021-07-23 15:30:49 2021-07-23 15:31:20 2021-07-23 16:20:33 0:49:13 0:38:02 0:11:11 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} 3
pass 6288757 2021-07-23 15:30:50 2021-07-23 15:31:20 2021-07-23 16:22:36 0:51:16 0:34:09 0:17:07 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
fail 6288758 2021-07-23 15:30:51 2021-07-23 15:31:20 2021-07-23 17:43:01 2:11:41 2:02:49 0:08:52 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds

pass 6288759 2021-07-23 15:30:52 2021-07-23 15:31:21 2021-07-23 16:25:33 0:54:12 0:45:47 0:08:25 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
dead 6288760 2021-07-23 15:30:53 2021-07-23 15:31:21 2021-07-24 03:42:31 12:11:10 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
Failure Reason:

hit max job timeout

pass 6288761 2021-07-23 15:30:54 2021-07-23 15:31:21 2021-07-23 17:17:57 1:46:36 1:36:28 0:10:08 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 6288762 2021-07-23 15:30:55 2021-07-23 15:31:23 2021-07-23 16:53:16 1:21:53 1:14:06 0:07:47 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
pass 6288763 2021-07-23 15:30:56 2021-07-23 15:31:23 2021-07-23 16:22:33 0:51:10 0:34:59 0:16:11 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} 3
pass 6288764 2021-07-23 15:30:57 2021-07-23 15:31:23 2021-07-23 16:21:49 0:50:26 0:40:11 0:10:15 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
pass 6288765 2021-07-23 15:30:58 2021-07-23 15:31:26 2021-07-23 16:09:47 0:38:21 0:31:45 0:06:36 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} 3
pass 6288766 2021-07-23 15:30:59 2021-07-23 15:31:26 2021-07-23 16:23:35 0:52:09 0:35:34 0:16:35 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 6288767 2021-07-23 15:31:00 2021-07-23 15:31:27 2021-07-23 16:28:54 0:57:27 0:46:50 0:10:37 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} 3
pass 6288768 2021-07-23 15:31:01 2021-07-23 15:31:27 2021-07-23 17:14:22 1:42:55 1:31:54 0:11:01 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
pass 6288769 2021-07-23 15:31:02 2021-07-23 15:31:28 2021-07-23 16:24:54 0:53:26 0:40:17 0:13:09 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
pass 6288770 2021-07-23 15:31:03 2021-07-23 15:31:29 2021-07-23 16:09:24 0:37:55 0:30:11 0:07:44 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} 3
pass 6288771 2021-07-23 15:31:04 2021-07-23 15:31:29 2021-07-23 16:05:39 0:34:10 0:27:14 0:06:56 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
fail 6288772 2021-07-23 15:31:05 2021-07-23 15:31:29 2021-07-23 16:12:40 0:41:11 0:23:49 0:17:22 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

Command failed on smithi057 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:dc6e6aee83d05e539da09c20751974c378bcb8fa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2ffcf788-ebcf-11eb-8c23-001a4aab830c -- ceph mon dump -f json'

pass 6288773 2021-07-23 15:31:06 2021-07-23 15:31:30 2021-07-23 18:44:35 3:13:05 3:03:01 0:10:04 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
pass 6288774 2021-07-23 15:31:07 2021-07-23 15:31:30 2021-07-23 16:32:29 1:00:59 0:50:42 0:10:17 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
pass 6288775 2021-07-23 15:31:08 2021-07-23 15:31:31 2021-07-23 17:19:40 1:48:09 1:33:48 0:14:21 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 6288776 2021-07-23 15:31:10 2021-07-23 15:31:31 2021-07-23 16:33:14 1:01:43 0:52:50 0:08:53 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
pass 6288777 2021-07-23 15:31:11 2021-07-23 15:31:32 2021-07-23 16:06:56 0:35:24 0:26:38 0:08:46 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
fail 6288778 2021-07-23 15:31:12 2021-07-23 15:31:32 2021-07-23 16:53:02 1:21:30 1:01:52 0:19:38 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
Failure Reason:

error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds

pass 6288779 2021-07-23 15:31:13 2021-07-23 16:05:50 2021-07-23 16:54:00 0:48:10 0:39:43 0:08:27 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
pass 6288780 2021-07-23 15:31:14 2021-07-23 16:07:00 2021-07-23 16:40:24 0:33:24 0:24:23 0:09:01 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 6288781 2021-07-23 15:31:15 2021-07-23 16:09:31 2021-07-23 16:58:35 0:49:04 0:37:11 0:11:53 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
pass 6288782 2021-07-23 15:31:16 2021-07-23 16:09:51 2021-07-23 17:22:39 1:12:48 1:05:11 0:07:37 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
pass 6288783 2021-07-23 15:31:17 2021-07-23 16:11:33 2021-07-23 16:43:15 0:31:42 0:25:06 0:06:36 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
pass 6288784 2021-07-23 15:31:18 2021-07-23 16:12:43 2021-07-23 16:52:55 0:40:12 0:28:00 0:12:12 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
pass 6288785 2021-07-23 15:31:20 2021-07-23 16:13:44 2021-07-23 16:48:48 0:35:04 0:27:12 0:07:52 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
pass 6288786 2021-07-23 15:31:22 2021-07-23 16:15:54 2021-07-23 17:13:00 0:57:06 0:49:09 0:07:57 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
pass 6288787 2021-07-23 15:31:23 2021-07-23 16:17:45 2021-07-23 17:23:04 1:05:19 0:55:44 0:09:35 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
pass 6288788 2021-07-23 15:31:24 2021-07-23 16:17:55 2021-07-23 17:01:34 0:43:39 0:35:47 0:07:52 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
pass 6288789 2021-07-23 15:31:25 2021-07-23 16:18:15 2021-07-23 17:42:08 1:23:53 1:17:03 0:06:50 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 6288790 2021-07-23 15:31:26 2021-07-23 16:18:46 2021-07-23 17:55:05 1:36:19 1:26:35 0:09:44 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} 3
pass 6288791 2021-07-23 15:31:27 2021-07-23 16:19:56 2021-07-23 16:54:58 0:35:02 0:28:19 0:06:43 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 6288792 2021-07-23 15:31:28 2021-07-23 16:20:16 2021-07-23 16:54:30 0:34:14 0:26:02 0:08:12 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
pass 6288793 2021-07-23 15:31:30 2021-07-23 16:20:37 2021-07-23 17:10:34 0:49:57 0:39:57 0:10:00 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} 3
pass 6288794 2021-07-23 15:31:31 2021-07-23 16:20:57 2021-07-23 16:54:53 0:33:56 0:26:54 0:07:02 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} 3
pass 6288795 2021-07-23 15:31:32 2021-07-23 16:21:58 2021-07-23 17:01:17 0:39:19 0:33:11 0:06:08 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} 3
fail 6288796 2021-07-23 15:31:33 2021-07-23 16:21:58 2021-07-23 17:11:48 0:49:50 0:38:35 0:11:15 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
Failure Reason:

The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}

pass 6288797 2021-07-23 15:31:34 2021-07-23 16:22:38 2021-07-23 17:00:08 0:37:30 0:30:03 0:07:27 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
fail 6288798 2021-07-23 15:31:35 2021-07-23 16:22:39 2021-07-23 16:44:10 0:21:31 0:14:58 0:06:33 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
Failure Reason:

Command failed on smithi111 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:dc6e6aee83d05e539da09c20751974c378bcb8fa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 7f7a43c4-ebd4-11eb-8c23-001a4aab830c -- ceph mon dump -f json'

pass 6288799 2021-07-23 15:31:36 2021-07-23 16:22:49 2021-07-23 17:01:26 0:38:37 0:26:54 0:11:43 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
pass 6288800 2021-07-23 15:31:37 2021-07-23 16:23:39 2021-07-23 18:17:09 1:53:30 1:46:01 0:07:29 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
pass 6288801 2021-07-23 15:31:38 2021-07-23 16:24:00 2021-07-23 17:15:52 0:51:52 0:45:23 0:06:29 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
dead 6288802 2021-07-23 15:31:39 2021-07-23 16:25:00 2021-07-24 04:33:27 12:08:27 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
Failure Reason:

hit max job timeout

pass 6288803 2021-07-23 15:31:40 2021-07-23 16:25:01 2021-07-23 17:31:21 1:06:20 0:58:40 0:07:40 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} 3
pass 6288804 2021-07-23 15:31:41 2021-07-23 16:25:41 2021-07-23 17:40:46 1:15:05 1:09:17 0:05:48 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
pass 6288805 2021-07-23 15:31:42 2021-07-23 16:25:51 2021-07-23 17:08:19 0:42:28 0:28:15 0:14:13 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 6288806 2021-07-23 15:31:43 2021-07-23 16:29:02 2021-07-23 17:08:52 0:39:50 0:32:02 0:07:48 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
pass 6288807 2021-07-23 15:31:44 2021-07-23 16:29:12 2021-07-23 17:04:45 0:35:33 0:28:07 0:07:26 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} 3
pass 6288808 2021-07-23 15:31:45 2021-07-23 16:30:03 2021-07-23 17:10:46 0:40:43 0:27:54 0:12:49 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} 3
pass 6288809 2021-07-23 15:31:46 2021-07-23 16:31:04 2021-07-23 17:12:53 0:41:49 0:33:51 0:07:58 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
pass 6288810 2021-07-23 15:31:47 2021-07-23 16:31:14 2021-07-23 17:57:36 1:26:22 1:19:49 0:06:33 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
pass 6288811 2021-07-23 15:31:48 2021-07-23 16:32:35 2021-07-23 17:10:08 0:37:33 0:27:44 0:09:49 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
pass 6288812 2021-07-23 15:31:49 2021-07-23 16:33:15 2021-07-23 17:08:04 0:34:49 0:25:37 0:09:12 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} 3
pass 6288813 2021-07-23 15:31:50 2021-07-23 16:34:56 2021-07-23 17:07:02 0:32:06 0:23:07 0:08:59 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} 3
pass 6288814 2021-07-23 15:31:51 2021-07-23 16:37:56 2021-07-23 17:49:28 1:11:32 1:00:10 0:11:22 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
pass 6288815 2021-07-23 15:31:52 2021-07-23 16:40:27 2021-07-23 18:30:15 1:49:48 1:43:06 0:06:42 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
pass 6288816 2021-07-23 15:31:53 2021-07-23 16:41:18 2021-07-23 17:32:12 0:50:54 0:43:24 0:07:30 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
pass 6288817 2021-07-23 15:31:54 2021-07-23 16:43:18 2021-07-23 18:29:09 1:45:51 1:36:27 0:09:24 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 6288818 2021-07-23 15:31:55 2021-07-23 16:43:59 2021-07-23 18:07:39 1:23:40 1:16:05 0:07:35 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} 3
pass 6288819 2021-07-23 15:31:56 2021-07-23 16:44:19 2021-07-23 17:20:01 0:35:42 0:24:25 0:11:17 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 6288820 2021-07-23 15:31:57 2021-07-23 16:48:20 2021-07-23 17:33:59 0:45:39 0:36:07 0:09:32 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} 3
pass 6288821 2021-07-23 15:31:58 2021-07-23 16:48:50 2021-07-23 17:33:28 0:44:38 0:35:58 0:08:40 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
pass 6288822 2021-07-23 15:31:58 2021-07-23 16:50:21 2021-07-23 17:24:08 0:33:47 0:24:21 0:09:26 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} 3
pass 6288823 2021-07-23 15:31:59 2021-07-23 16:53:01 2021-07-23 17:39:37 0:46:36 0:36:40 0:09:56 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
fail 6288824 2021-07-23 15:32:00 2021-07-23 16:53:02 2021-07-23 18:18:09 1:25:07 1:17:30 0:07:37 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
Failure Reason:

The following counters failed to be set on mds daemons: {'mds.exported', 'mds.imported'}

pass 6288825 2021-07-23 15:32:01 2021-07-23 16:53:12 2021-07-23 17:28:17 0:35:05 0:29:02 0:06:03 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
pass 6288826 2021-07-23 15:32:02 2021-07-23 16:53:22 2021-07-23 17:31:07 0:37:45 0:28:12 0:09:33 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
pass 6288827 2021-07-23 15:32:03 2021-07-23 16:54:03 2021-07-23 17:28:11 0:34:08 0:27:20 0:06:48 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
pass 6288828 2021-07-23 15:32:04 2021-07-23 16:54:33 2021-07-23 17:47:50 0:53:17 0:46:41 0:06:36 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
dead 6288829 2021-07-23 15:32:05 2021-07-23 16:54:53 2021-07-24 05:03:02 12:08:09 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
Failure Reason:

hit max job timeout

pass 6288830 2021-07-23 15:32:06 2021-07-23 16:55:04 2021-07-23 17:41:23 0:46:19 0:36:14 0:10:05 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
pass 6288831 2021-07-23 15:32:07 2021-07-23 16:58:44 2021-07-23 18:59:48 2:01:04 1:53:21 0:07:43 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 6288832 2021-07-23 15:32:08 2021-07-23 17:00:15 2021-07-23 18:32:46 1:32:31 1:21:11 0:11:20 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
pass 6288833 2021-07-23 15:32:09 2021-07-23 17:01:26 2021-07-23 17:37:04 0:35:38 0:28:04 0:07:34 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 6288834 2021-07-23 15:32:10 2021-07-23 17:01:36 2021-07-23 17:35:23 0:33:47 0:27:10 0:06:37 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
pass 6288835 2021-07-23 15:32:11 2021-07-23 17:01:37 2021-07-23 17:53:48 0:52:11 0:39:34 0:12:37 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
pass 6288836 2021-07-23 15:32:12 2021-07-23 17:04:48 2021-07-23 17:38:07 0:33:19 0:25:32 0:07:47 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} 3
pass 6288837 2021-07-23 15:32:13 2021-07-23 17:04:58 2021-07-23 17:45:40 0:40:42 0:33:35 0:07:07 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
fail 6288838 2021-07-23 15:32:14 2021-07-23 17:06:39 2021-07-23 18:24:53 1:18:14 1:06:52 0:11:22 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} 3
Failure Reason:

The following counters failed to be set on mds daemons: {'mds.imported', 'mds.exported'}

pass 6288839 2021-07-23 15:32:15 2021-07-23 17:06:39 2021-07-23 17:40:18 0:33:39 0:26:37 0:07:02 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} 3
fail 6288840 2021-07-23 15:32:15 2021-07-23 17:07:09 2021-07-23 17:31:14 0:24:05 0:16:17 0:07:48 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} 3
Failure Reason:

Command failed on smithi139 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:dc6e6aee83d05e539da09c20751974c378bcb8fa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc73e7fc-ebda-11eb-8c23-001a4aab830c -- ceph orch daemon add osd smithi139:vg_nvme/lv_3'

pass 6288841 2021-07-23 15:32:16 2021-07-23 17:08:10 2021-07-23 17:47:29 0:39:19 0:28:49 0:10:30 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} 3
fail 6288842 2021-07-23 15:32:17 2021-07-23 17:08:20 2021-07-23 18:48:09 1:39:49 1:32:25 0:07:24 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds

pass 6288843 2021-07-23 15:32:18 2021-07-23 17:09:00 2021-07-23 18:11:18 1:02:18 0:54:41 0:07:37 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} 3
pass 6288844 2021-07-23 15:32:19 2021-07-23 17:10:11 2021-07-23 18:06:26 0:56:15 0:44:17 0:11:58 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} 3
pass 6288845 2021-07-23 15:32:20 2021-07-23 17:10:41 2021-07-23 18:48:23 1:37:42 1:31:23 0:06:19 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} 3
pass 6288846 2021-07-23 15:32:21 2021-07-23 17:10:52 2021-07-23 18:34:36 1:23:44 1:17:00 0:06:44 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} 3
pass 6288847 2021-07-23 15:32:22 2021-07-23 17:11:52 2021-07-23 17:52:14 0:40:22 0:29:10 0:11:12 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} 3
pass 6288848 2021-07-23 15:32:23 2021-07-23 17:13:03 2021-07-23 17:47:57 0:34:54 0:28:22 0:06:32 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} 3
pass 6288849 2021-07-23 15:32:24 2021-07-23 17:13:03 2021-07-23 17:58:50 0:45:47 0:37:50 0:07:57 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} 3
pass 6288850 2021-07-23 15:32:25 2021-07-23 17:13:04 2021-07-23 17:51:28 0:38:24 0:27:49 0:10:35 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} 3
pass 6288851 2021-07-23 15:32:26 2021-07-23 17:14:24 2021-07-23 17:56:48 0:42:24 0:34:21 0:08:03 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} 3
fail 6288852 2021-07-23 15:32:27 2021-07-23 17:15:55 2021-07-23 17:43:09 0:27:14 0:17:56 0:09:18 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} 3
Failure Reason:

Command failed on smithi196 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:dc6e6aee83d05e539da09c20751974c378bcb8fa shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2780b808-ebdc-11eb-8c23-001a4aab830c -- ceph orch daemon add osd smithi196:vg_nvme/lv_2'

pass 6288853 2021-07-23 15:32:28 2021-07-23 17:18:05 2021-07-23 18:02:54 0:44:49 0:32:14 0:12:35 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 rhel_8 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} 3
pass 6288854 2021-07-23 15:32:29 2021-07-23 17:19:46 2021-07-23 17:53:47 0:34:01 0:25:46 0:08:15 smithi master rhel 8.4 fs:workload/{begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 rhel_8 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} 3