Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 6847389 2022-05-24 14:47:48 2022-05-25 00:54:15 2022-05-25 01:12:48 0:18:33 0:13:03 0:05:30 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} 2
Failure Reason:

Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead)

fail 6847390 2022-05-24 14:47:48 2022-05-25 00:54:16 2022-05-25 01:25:56 0:31:40 0:22:02 0:09:38 smithi master ubuntu 20.04 fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full} 1
Failure Reason:

"2022-05-25T01:13:25.059423+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847391 2022-05-24 14:47:49 2022-05-25 00:54:16 2022-05-25 01:23:46 0:29:30 0:18:55 0:10:35 smithi master ubuntu 20.04 fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} 1
Failure Reason:

"2022-05-25T01:13:50.295831+0000 mon.a (mon.0) 83 : cluster [WRN] Health check failed: 3 mgr modules have failed dependencies (MGR_MODULE_DEPENDENCY)" in cluster log

fail 6847392 2022-05-24 14:47:50 2022-05-25 00:54:17 2022-05-25 01:33:08 0:38:51 0:21:38 0:17:13 smithi master ubuntu 20.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} 4
Failure Reason:

"2022-05-25T01:24:42.916580+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847393 2022-05-24 14:47:51 2022-05-25 00:54:17 2022-05-25 01:39:10 0:44:53 0:23:34 0:21:19 smithi master ubuntu 20.04 fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell} 2
Failure Reason:

"2022-05-25T01:29:45.823795+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847394 2022-05-24 14:47:52 2022-05-25 00:54:17 2022-05-25 01:38:02 0:43:45 0:24:01 0:19:44 smithi master ubuntu 20.04 fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

"2022-05-25T01:29:31.189250+0000 mgr.y (mgr.4113) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847395 2022-05-24 14:47:52 2022-05-25 00:54:18 2022-05-25 01:39:18 0:45:00 0:36:17 0:08:43 smithi master centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
pass 6847396 2022-05-24 14:47:53 2022-05-25 00:54:18 2022-05-25 03:27:33 2:33:15 2:23:43 0:09:32 smithi master centos 8.stream fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} 1
fail 6847397 2022-05-24 14:47:54 2022-05-25 00:54:18 2022-05-25 01:38:42 0:44:24 0:24:55 0:19:29 smithi master ubuntu 20.04 fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/lockdep} 2
Failure Reason:

"2022-05-25T01:29:00.157898+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847398 2022-05-24 14:47:55 2022-05-25 00:54:19 2022-05-25 02:29:37 1:35:18 1:25:08 0:10:10 smithi master rhel 8.5 fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} 2
dead 6847399 2022-05-24 14:47:55 2022-05-25 00:54:19 2022-05-25 07:35:53 6:41:34 smithi master centos 8.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} 2
Failure Reason:

hit max job timeout

pass 6847400 2022-05-24 14:47:56 2022-05-25 00:54:19 2022-05-25 01:25:38 0:31:19 0:22:06 0:09:13 smithi master rhel 8.5 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} 2
fail 6847401 2022-05-24 14:47:57 2022-05-25 00:54:20 2022-05-25 03:34:01 2:39:41 2:28:28 0:11:13 smithi master rhel 8.5 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds

fail 6847402 2022-05-24 14:47:58 2022-05-25 00:54:20 2022-05-25 01:05:04 0:10:44 smithi master centos 8.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} 2
Failure Reason:

Command failed on smithi141 with status 1: 'sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm'

fail 6847403 2022-05-24 14:47:58 2022-05-25 00:54:21 2022-05-25 01:30:37 0:36:16 0:20:44 0:15:32 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T01:21:26.416661+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847404 2022-05-24 14:47:59 2022-05-25 00:54:21 2022-05-25 01:15:23 0:21:02 0:12:03 0:08:59 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} 2
Failure Reason:

Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)

fail 6847405 2022-05-24 14:48:00 2022-05-25 00:54:21 2022-05-25 01:38:04 0:43:43 0:23:12 0:20:31 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T01:29:22.507001+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847406 2022-05-24 14:48:01 2022-05-25 00:54:22 2022-05-25 01:37:37 0:43:15 0:23:30 0:19:45 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T01:27:54.726197+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847407 2022-05-24 14:48:01 2022-05-25 00:54:22 2022-05-25 01:39:11 0:44:49 0:24:15 0:20:34 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

"2022-05-25T01:29:10.876601+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847408 2022-05-24 14:48:02 2022-05-25 00:54:22 2022-05-25 01:39:28 0:45:06 0:24:08 0:20:58 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} 2
Failure Reason:

"2022-05-25T01:29:51.149351+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847409 2022-05-24 14:48:03 2022-05-25 00:54:23 2022-05-25 01:36:44 0:42:21 0:23:01 0:19:20 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T01:28:58.258545+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847410 2022-05-24 14:48:04 2022-05-25 00:54:23 2022-05-25 02:49:47 1:55:24 1:41:14 0:14:10 smithi master centos 8.stream fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
dead 6847411 2022-05-24 14:48:05 2022-05-25 00:54:23 2022-05-25 07:37:55 6:43:32 smithi master centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

hit max job timeout

fail 6847412 2022-05-24 14:48:05 2022-05-25 00:54:24 2022-05-25 01:33:23 0:38:59 0:21:33 0:17:26 smithi master ubuntu 20.04 fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

"2022-05-25T01:25:00.195290+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847413 2022-05-24 14:48:06 2022-05-25 00:54:24 2022-05-25 01:38:57 0:44:33 0:23:48 0:20:45 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair} 2
Failure Reason:

"2022-05-25T01:28:52.793005+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847414 2022-05-24 14:48:07 2022-05-25 00:54:24 2022-05-25 01:30:34 0:36:10 0:21:05 0:15:05 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T01:21:41.244871+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847415 2022-05-24 14:48:07 2022-05-25 00:54:25 2022-05-25 04:45:47 3:51:22 3:40:40 0:10:42 smithi master centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi142 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e182ac491fa82dd054c4d834853a809d881824e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

pass 6847416 2022-05-24 14:48:08 2022-05-25 00:54:25 2022-05-25 01:31:12 0:36:47 0:24:05 0:12:42 smithi master centos 8.stream fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
fail 6847417 2022-05-24 14:48:09 2022-05-25 00:54:26 2022-05-25 01:39:40 0:45:14 0:24:15 0:20:59 smithi master ubuntu 20.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} 5
Failure Reason:

"2022-05-25T01:30:03.464807+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847418 2022-05-24 14:48:10 2022-05-25 00:54:26 2022-05-25 01:27:40 0:33:14 0:21:56 0:11:18 smithi master rhel 8.5 fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} 2
Failure Reason:

Test failure: test_moncap_with_blanket_allow (tasks.cephfs.test_multifs_auth.TestMONCaps)

fail 6847419 2022-05-24 14:48:10 2022-05-25 00:54:26 2022-05-25 01:48:32 0:54:06 0:40:02 0:14:04 smithi master ubuntu 20.04 fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/valgrind} 2
Failure Reason:

"2022-05-25T01:37:28.029936+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847420 2022-05-24 14:48:11 2022-05-25 00:54:27 2022-05-25 01:24:11 0:29:44 0:17:25 0:12:19 smithi master centos 8.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} 2
Failure Reason:

Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)

fail 6847421 2022-05-24 14:48:12 2022-05-25 00:54:27 2022-05-25 01:33:03 0:38:36 0:21:20 0:17:16 smithi master fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} 2
Failure Reason:

"2022-05-25T01:22:26.586521+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847422 2022-05-24 14:48:13 2022-05-25 00:54:27 2022-05-25 01:25:24 0:30:57 0:18:56 0:12:01 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T01:17:16.057871+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847423 2022-05-24 14:48:13 2022-05-25 00:54:28 2022-05-25 01:26:32 0:32:04 0:19:41 0:12:23 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

"2022-05-25T01:17:08.424756+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847424 2022-05-24 14:48:14 2022-05-25 00:54:28 2022-05-25 01:37:06 0:42:38 0:22:23 0:20:15 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T01:27:45.878936+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847425 2022-05-24 14:48:15 2022-05-25 00:54:28 2022-05-25 01:30:31 0:36:03 0:23:44 0:12:19 smithi master rhel 8.5 fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} 2
fail 6847426 2022-05-24 14:48:16 2022-05-25 00:54:29 2022-05-25 01:38:48 0:44:19 0:25:08 0:19:11 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T01:29:38.452256+0000 mgr.y (mgr.4116) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847427 2022-05-24 14:48:16 2022-05-25 00:54:29 2022-05-25 02:14:58 1:20:29 1:10:20 0:10:09 smithi master rhel 8.5 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} 3
pass 6847428 2022-05-24 14:48:17 2022-05-25 00:54:30 2022-05-25 03:59:29 3:04:59 2:51:59 0:13:00 smithi master rhel 8.5 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} 3
fail 6847429 2022-05-24 14:48:18 2022-05-25 00:54:30 2022-05-25 01:18:57 0:24:27 0:16:13 0:08:14 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} 2
Failure Reason:

Test failure: test_scrub_backtrace (tasks.cephfs.test_multimds_misc.TestScrub2)

fail 6847430 2022-05-24 14:48:19 2022-05-25 00:54:30 2022-05-25 04:19:51 3:25:21 3:11:37 0:13:44 smithi master centos 8.stream fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T02:11:15.394527+0000 mds.f (mds.2) 24 : cluster [WRN] client.4711 isn't responding to mclientcaps(revoke), ino 0x3000000305d pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.004525 seconds ago" in cluster log

pass 6847431 2022-05-24 14:48:20 2022-05-25 00:54:31 2022-05-25 01:46:05 0:51:34 0:44:01 0:07:33 smithi master rhel 8.5 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
fail 6847432 2022-05-24 14:48:20 2022-05-25 00:54:31 2022-05-25 01:29:03 0:34:32 0:19:13 0:15:19 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T01:21:00.391811+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847433 2022-05-24 14:48:21 2022-05-25 00:54:32 2022-05-25 01:38:54 0:44:22 0:23:44 0:20:38 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T01:29:35.573304+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847434 2022-05-24 14:48:22 2022-05-25 00:54:32 2022-05-25 01:39:18 0:44:46 0:23:54 0:20:52 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} 2
Failure Reason:

"2022-05-25T01:29:52.581748+0000 mgr.x (mgr.4106) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847435 2022-05-24 14:48:23 2022-05-25 00:54:32 2022-05-25 01:38:54 0:44:22 0:23:41 0:20:41 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

"2022-05-25T01:29:07.798027+0000 mgr.x (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847436 2022-05-24 14:48:23 2022-05-25 00:54:33 2022-05-25 01:29:54 0:35:21 0:25:58 0:09:23 smithi master rhel 8.5 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} 3
pass 6847437 2022-05-24 14:48:24 2022-05-25 00:54:33 2022-05-25 01:50:43 0:56:10 0:43:55 0:12:15 smithi master centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 6847438 2022-05-24 14:48:25 2022-05-25 00:54:33 2022-05-25 01:40:07 0:45:34 0:24:25 0:21:09 smithi master ubuntu 20.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} 4
Failure Reason:

"2022-05-25T01:30:06.185678+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847439 2022-05-24 14:48:26 2022-05-25 00:54:34 2022-05-25 01:35:05 0:40:31 0:20:28 0:20:03 smithi master ubuntu 20.04 fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

"2022-05-25T01:25:56.653450+0000 mgr.y (mgr.4098) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847440 2022-05-24 14:48:26 2022-05-25 00:54:34 2022-05-25 01:39:34 0:45:00 0:23:43 0:21:17 smithi master ubuntu 20.04 fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} 2
Failure Reason:

"2022-05-25T01:28:48.220637+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847441 2022-05-24 14:48:27 2022-05-25 00:54:35 2022-05-25 01:38:29 0:43:54 0:22:19 0:21:35 smithi master ubuntu 20.04 fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} 2
Failure Reason:

"2022-05-25T01:29:16.734478+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847442 2022-05-24 14:48:28 2022-05-25 01:05:06 2022-05-25 01:41:42 0:36:36 0:19:15 0:17:21 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T01:32:36.607480+0000 mgr.y (mgr.4098) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

dead 6847443 2022-05-24 14:48:29 2022-05-25 01:12:58 2022-05-25 07:56:24 6:43:26 smithi master centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

hit max job timeout

fail 6847444 2022-05-24 14:48:29 2022-05-25 01:15:30 2022-05-25 01:50:28 0:34:58 0:19:09 0:15:49 smithi master fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} 2
Failure Reason:

"2022-05-25T01:40:23.250776+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847445 2022-05-24 14:48:30 2022-05-25 01:19:02 2022-05-25 01:53:36 0:34:34 0:18:03 0:16:31 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T01:44:08.039515+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847446 2022-05-24 14:48:31 2022-05-25 01:24:13 2022-05-25 01:52:32 0:28:19 0:18:44 0:09:35 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} 2
Failure Reason:

Test failure: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks)

fail 6847447 2022-05-24 14:48:32 2022-05-25 01:25:34 2022-05-25 01:55:25 0:29:51 0:19:02 0:10:49 smithi master ubuntu 20.04 fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

"2022-05-25T01:45:51.028001+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847448 2022-05-24 14:48:33 2022-05-25 01:25:44 2022-05-25 01:45:11 0:19:27 0:12:42 0:06:45 smithi master rhel 8.5 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} 2
fail 6847449 2022-05-24 14:48:33 2022-05-25 01:26:05 2022-05-25 01:55:50 0:29:45 0:19:03 0:10:42 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T01:46:16.456685+0000 mgr.y (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847450 2022-05-24 14:48:34 2022-05-25 01:26:36 2022-05-25 02:18:47 0:52:11 0:41:07 0:11:04 smithi master centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
pass 6847451 2022-05-24 14:48:35 2022-05-25 01:27:47 2022-05-25 02:58:02 1:30:15 1:19:37 0:10:38 smithi master rhel 8.5 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
fail 6847452 2022-05-24 14:48:36 2022-05-25 01:29:08 2022-05-25 01:59:21 0:30:13 0:19:18 0:10:55 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} 2
Failure Reason:

"2022-05-25T01:50:39.016601+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847453 2022-05-24 14:48:37 2022-05-25 01:29:59 2022-05-25 02:04:02 0:34:03 0:22:10 0:11:53 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

"2022-05-25T01:51:14.516892+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847454 2022-05-24 14:48:37 2022-05-25 01:30:41 2022-05-25 02:01:56 0:31:15 0:19:42 0:11:33 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} 2
Failure Reason:

"2022-05-25T01:51:54.771172+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847455 2022-05-24 14:48:38 2022-05-25 01:30:41 2022-05-25 02:04:28 0:33:47 0:21:56 0:11:51 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T01:51:54.418635+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847456 2022-05-24 14:48:39 2022-05-25 01:30:42 2022-05-25 02:00:42 0:30:00 0:19:06 0:10:54 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T01:52:20.234288+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847457 2022-05-24 14:48:40 2022-05-25 01:31:22 2022-05-25 01:58:19 0:26:57 0:17:47 0:09:10 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} 2
Failure Reason:

Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays)

fail 6847458 2022-05-24 14:48:41 2022-05-25 01:33:13 2022-05-25 02:02:34 0:29:21 0:18:34 0:10:47 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T01:53:56.776160+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847459 2022-05-24 14:48:41 2022-05-25 01:33:13 2022-05-25 02:03:06 0:29:53 0:19:44 0:10:09 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} 2
Failure Reason:

"2022-05-25T01:54:12.991922+0000 mgr.y (mgr.4103) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847460 2022-05-24 14:48:42 2022-05-25 01:33:14 2022-05-25 02:04:29 0:31:15 0:19:36 0:11:39 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration} 2
Failure Reason:

"2022-05-25T01:54:01.520683+0000 mgr.x (mgr.4102) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847461 2022-05-24 14:48:43 2022-05-25 01:33:24 2022-05-25 02:10:45 0:37:21 0:28:28 0:08:53 smithi master rhel 8.5 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} 2
fail 6847462 2022-05-24 14:48:44 2022-05-25 01:35:15 2022-05-25 02:06:05 0:30:50 0:18:24 0:12:26 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T01:56:32.302219+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847463 2022-05-24 14:48:45 2022-05-25 01:36:55 2022-05-25 02:04:13 0:27:18 0:17:59 0:09:19 smithi master rhel 8.5 fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}} 2
Failure Reason:

Command failed (workunit test libcephfs/test.sh) on smithi070 with status 141: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e182ac491fa82dd054c4d834853a809d881824e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'

fail 6847464 2022-05-24 14:48:45 2022-05-25 01:37:16 2022-05-25 02:14:05 0:36:49 0:22:19 0:14:30 smithi master ubuntu 20.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} 5
Failure Reason:

"2022-05-25T02:03:23.709156+0000 mgr.x (mgr.4114) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847465 2022-05-24 14:48:46 2022-05-25 01:38:06 2022-05-25 02:22:46 0:44:40 0:36:08 0:08:32 smithi master rhel 8.5 fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} 2
Failure Reason:

Test failure: test_moncap_with_blanket_allow (tasks.cephfs.test_multifs_auth.TestMONCaps)

fail 6847466 2022-05-24 14:48:47 2022-05-25 01:38:07 2022-05-25 02:07:58 0:29:51 0:19:08 0:10:43 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T01:59:14.576216+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847467 2022-05-24 14:48:48 2022-05-25 01:38:37 2022-05-25 02:24:18 0:45:41 0:35:42 0:09:59 smithi master centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
fail 6847468 2022-05-24 14:48:48 2022-05-25 01:38:57 2022-05-25 02:10:07 0:31:10 0:19:27 0:11:43 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T02:01:12.373823+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847469 2022-05-24 14:48:49 2022-05-25 01:38:58 2022-05-25 02:10:08 0:31:10 0:20:15 0:10:55 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

"2022-05-25T02:01:25.163444+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847470 2022-05-24 14:48:50 2022-05-25 01:38:58 2022-05-25 02:12:19 0:33:21 0:20:35 0:12:46 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota} 2
Failure Reason:

"2022-05-25T02:03:16.631361+0000 mgr.y (mgr.4116) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847471 2022-05-24 14:48:51 2022-05-25 01:38:59 2022-05-25 02:10:41 0:31:42 0:19:58 0:11:44 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T02:00:58.101898+0000 mgr.y (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847472 2022-05-24 14:48:51 2022-05-25 01:39:19 2022-05-25 03:05:59 1:26:40 1:19:13 0:07:27 smithi master centos 8.stream fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
fail 6847473 2022-05-24 14:48:52 2022-05-25 01:39:19 2022-05-25 02:11:13 0:31:54 0:21:02 0:10:52 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} 2
Failure Reason:

"2022-05-25T02:03:04.804046+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847474 2022-05-24 14:48:53 2022-05-25 01:39:20 2022-05-25 02:08:34 0:29:14 0:18:50 0:10:24 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T02:00:28.518338+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847475 2022-05-24 14:48:54 2022-05-25 01:39:20 2022-05-25 02:11:08 0:31:48 0:19:31 0:12:17 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T02:01:47.935472+0000 mgr.y (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847476 2022-05-24 14:48:54 2022-05-25 01:39:20 2022-05-25 02:12:49 0:33:29 0:20:55 0:12:34 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T02:03:15.821058+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847477 2022-05-24 14:48:55 2022-05-25 01:39:31 2022-05-25 02:12:06 0:32:35 0:19:05 0:13:30 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T02:03:14.185283+0000 mgr.y (mgr.4118) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847478 2022-05-24 14:48:56 2022-05-25 01:39:41 2022-05-25 02:11:02 0:31:21 0:19:40 0:11:41 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} 2
Failure Reason:

"2022-05-25T02:02:01.162236+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847479 2022-05-24 14:48:57 2022-05-25 01:39:42 2022-05-25 02:14:09 0:34:27 0:20:58 0:13:29 smithi master ubuntu 20.04 fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} 2
Failure Reason:

"2022-05-25T02:03:28.068557+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847480 2022-05-24 14:48:58 2022-05-25 01:39:42 2022-05-25 02:11:28 0:31:46 0:20:38 0:11:08 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
Failure Reason:

"2022-05-25T02:03:09.026699+0000 mgr.y (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847481 2022-05-24 14:48:58 2022-05-25 01:40:12 2022-05-25 02:11:47 0:31:35 0:20:34 0:11:01 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T02:03:23.401689+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847482 2022-05-24 14:48:59 2022-05-25 01:40:13 2022-05-25 02:20:15 0:40:02 0:32:45 0:07:17 smithi master rhel 8.5 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} 2
fail 6847483 2022-05-24 14:49:00 2022-05-25 01:41:43 2022-05-25 02:14:35 0:32:52 0:19:08 0:13:44 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} 2
Failure Reason:

"2022-05-25T02:05:25.042407+0000 mgr.y (mgr.4107) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

dead 6847484 2022-05-24 14:49:01 2022-05-25 01:45:14 2022-05-25 08:26:54 6:41:40 smithi master centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

hit max job timeout

fail 6847485 2022-05-24 14:49:01 2022-05-25 01:46:15 2022-05-25 02:17:48 0:31:33 0:19:49 0:11:44 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

"2022-05-25T02:08:15.467253+0000 mgr.x (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847486 2022-05-24 14:49:02 2022-05-25 01:48:35 2022-05-25 02:21:00 0:32:25 0:19:12 0:13:13 smithi master ubuntu 20.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} 4
Failure Reason:

"2022-05-25T02:12:49.924361+0000 mgr.y (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847487 2022-05-24 14:49:03 2022-05-25 01:50:46 2022-05-25 05:08:01 3:17:15 3:05:00 0:12:15 smithi master rhel 8.5 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} 3
Failure Reason:

error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds

fail 6847488 2022-05-24 14:49:04 2022-05-25 01:52:37 2022-05-25 02:18:48 0:26:11 0:16:26 0:09:45 smithi master centos 8.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} 2
Failure Reason:

Test failure: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace)

fail 6847489 2022-05-24 14:49:04 2022-05-25 01:53:38 2022-05-25 02:22:58 0:29:20 0:17:58 0:11:22 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T02:15:22.594233+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847490 2022-05-24 14:49:05 2022-05-25 01:55:29 2022-05-25 02:18:54 0:23:25 0:15:52 0:07:33 smithi master rhel 8.5 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} 2
fail 6847491 2022-05-24 14:49:06 2022-05-25 01:55:59 2022-05-25 02:27:50 0:31:51 0:19:13 0:12:38 smithi master fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} 2
Failure Reason:

"2022-05-25T02:18:44.683816+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847492 2022-05-24 14:49:07 2022-05-25 01:58:20 2022-05-25 02:28:38 0:30:18 0:18:33 0:11:45 smithi master ubuntu 20.04 fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} 2
Failure Reason:

"2022-05-25T02:19:00.995442+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847493 2022-05-24 14:49:07 2022-05-25 01:59:30 2022-05-25 02:25:51 0:26:21 0:19:16 0:07:05 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} 2
fail 6847494 2022-05-24 14:49:08 2022-05-25 02:00:51 2022-05-25 02:31:19 0:30:28 0:19:15 0:11:13 smithi master ubuntu 20.04 fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python} 2
Failure Reason:

"2022-05-25T02:22:06.259597+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847495 2022-05-24 14:49:09 2022-05-25 02:02:01 2022-05-25 02:32:04 0:30:03 0:19:21 0:10:42 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} 2
Failure Reason:

"2022-05-25T02:22:33.710096+0000 mgr.x (mgr.4108) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847496 2022-05-24 14:49:10 2022-05-25 02:02:42 2022-05-25 02:33:05 0:30:23 0:18:45 0:11:38 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T02:24:03.876298+0000 mgr.y (mgr.4105) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847497 2022-05-24 14:49:11 2022-05-25 02:03:12 2022-05-25 05:53:22 3:50:10 3:40:29 0:09:41 smithi master centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi167 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e182ac491fa82dd054c4d834853a809d881824e5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

fail 6847498 2022-05-24 14:49:11 2022-05-25 02:04:13 2022-05-25 02:33:37 0:29:24 0:19:23 0:10:01 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

"2022-05-25T02:24:11.014635+0000 mgr.x (mgr.4111) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847499 2022-05-24 14:49:12 2022-05-25 02:04:23 2022-05-25 02:33:42 0:29:19 0:19:37 0:09:42 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} 2
Failure Reason:

"2022-05-25T02:24:28.673594+0000 mgr.y (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847500 2022-05-24 14:49:13 2022-05-25 02:04:33 2022-05-25 02:41:46 0:37:13 0:27:39 0:09:34 smithi master rhel 8.5 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} 2
fail 6847501 2022-05-24 14:49:14 2022-05-25 02:04:34 2022-05-25 02:25:05 0:20:31 0:12:38 0:07:53 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} 2
Failure Reason:

Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage)

fail 6847502 2022-05-24 14:49:14 2022-05-25 02:06:14 2022-05-25 02:35:11 0:28:57 0:18:00 0:10:57 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} 2
Failure Reason:

Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan)

fail 6847503 2022-05-24 14:49:15 2022-05-25 02:08:05 2022-05-25 02:37:51 0:29:46 0:19:07 0:10:39 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T02:28:36.732213+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847504 2022-05-24 14:49:16 2022-05-25 02:08:36 2022-05-25 02:42:48 0:34:12 0:20:13 0:13:59 smithi master ubuntu 20.04 fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} 5
Failure Reason:

"2022-05-25T02:33:42.804396+0000 mgr.y (mgr.4101) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847505 2022-05-24 14:49:17 2022-05-25 02:10:46 2022-05-25 02:47:34 0:36:48 0:27:38 0:09:10 smithi master centos 8.stream fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} 2
Failure Reason:

Test failure: test_moncap_with_blanket_allow (tasks.cephfs.test_multifs_auth.TestMONCaps)

fail 6847506 2022-05-24 14:49:18 2022-05-25 02:10:47 2022-05-25 02:40:22 0:29:35 0:19:19 0:10:16 smithi master ubuntu 20.04 fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} 2
Failure Reason:

"2022-05-25T02:32:22.872251+0000 mgr.y (mgr.4106) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847507 2022-05-24 14:49:18 2022-05-25 02:11:07 2022-05-25 02:41:03 0:29:56 0:18:54 0:11:02 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} 2
Failure Reason:

"2022-05-25T02:32:55.897881+0000 mgr.x (mgr.4104) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847508 2022-05-24 14:49:19 2022-05-25 02:11:17 2022-05-25 02:52:41 0:41:24 0:35:03 0:06:21 smithi master rhel 8.5 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} 2
fail 6847509 2022-05-24 14:49:20 2022-05-25 02:11:18 2022-05-25 02:30:33 0:19:15 0:13:00 0:06:15 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} 2
Failure Reason:

Test failure: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub)

fail 6847510 2022-05-24 14:49:21 2022-05-25 02:11:38 2022-05-25 02:41:21 0:29:43 0:18:35 0:11:08 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} 2
Failure Reason:

"2022-05-25T02:31:59.690029+0000 mgr.y (mgr.4110) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847511 2022-05-24 14:49:21 2022-05-25 02:11:49 2022-05-25 02:42:42 0:30:53 0:19:48 0:11:05 smithi master ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} 2
Failure Reason:

"2022-05-25T02:34:14.473408+0000 mgr.x (mgr.4100) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

fail 6847512 2022-05-24 14:49:22 2022-05-25 02:12:09 2022-05-25 02:43:44 0:31:35 0:19:43 0:11:52 smithi master ubuntu 20.04 fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} 2
Failure Reason:

"2022-05-25T02:33:14.280344+0000 mgr.y (mgr.4113) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

dead 6847513 2022-05-24 14:49:23 2022-05-25 02:12:30 2022-05-25 08:53:08 6:40:38 smithi master centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

hit max job timeout

fail 6847514 2022-05-24 14:49:24 2022-05-25 02:12:50 2022-05-25 02:44:40 0:31:50 0:19:23 0:12:27 smithi master ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair} 2
Failure Reason:

"2022-05-25T02:35:48.036929+0000 mgr.x (mgr.4109) 1 : cluster [ERR] Failed to load ceph-mgr modules: cephadm, nfs, orchestrator, dashboard, volumes, test_orchestrator" in cluster log

pass 6847515 2022-05-24 14:49:25 2022-05-25 02:14:11 2022-05-25 03:04:57 0:50:46 0:42:10 0:08:36 smithi master centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} 2
fail 6847516 2022-05-24 14:49:25 2022-05-25 02:14:11 2022-05-25 05:59:55 3:45:44 3:36:30 0:09:14 smithi master rhel 8.5 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} 2
Failure Reason:

"2022-05-25T03:31:35.464969+0000 mds.b (mds.1) 26 : cluster [WRN] client.4770 isn't responding to mclientcaps(revoke), ino 0x2000000b1a6 pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.004054 seconds ago" in cluster log

fail 6847517 2022-05-24 14:49:26 2022-05-25 02:14:11 2022-05-25 02:33:45 0:19:34 0:12:23 0:07:11 smithi master rhel 8.5 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} 2
Failure Reason:

Test failure: test_flush (tasks.cephfs.test_flush.TestFlush)

dead 6847518 2022-05-24 14:49:27 2022-05-25 02:14:42 2022-05-25 11:28:12 9:13:30 smithi master rhel 8.5 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} 3
Failure Reason:

hit max job timeout