Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 6541567 2021-12-03 03:21:09 2021-12-03 11:12:56 2021-12-03 11:37:24 0:24:28 0:10:49 0:13:39 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_deep_flatten} 1
pass 6541568 2021-12-03 03:21:10 2021-12-03 11:12:57 2021-12-03 11:45:58 0:33:01 0:20:03 0:12:58 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6541569 2021-12-03 03:21:11 2021-12-03 11:12:57 2021-12-03 11:55:20 0:42:23 0:29:12 0:13:11 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_fio} 3
pass 6541570 2021-12-03 03:21:12 2021-12-03 11:13:07 2021-12-03 11:45:36 0:32:29 0:19:58 0:12:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_data_pool} 3
fail 6541571 2021-12-03 03:21:13 2021-12-03 11:13:38 2021-12-03 16:08:02 4:54:24 4:40:45 0:13:39 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{crc} msgr-failures/few tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi110 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludekl5h0jqv -r -- '-g auto -g blockdev -x clone'"

pass 6541572 2021-12-03 03:21:13 2021-12-03 11:13:58 2021-12-03 12:09:54 0:55:56 0:42:53 0:13:03 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/backoff thrashosds-health workloads/rbd_fio} 3
fail 6541573 2021-12-03 03:21:14 2021-12-03 11:14:08 2021-12-03 11:22:25 0:08:17 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major

pass 6541574 2021-12-03 03:21:15 2021-12-03 11:14:29 2021-12-03 11:36:22 0:21:53 0:09:51 0:12:02 smithi master krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} 1
pass 6541575 2021-12-03 03:21:16 2021-12-03 11:14:40 2021-12-03 11:38:53 0:24:13 0:12:42 0:11:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_exclusive_option} 3
fail 6541576 2021-12-03 03:21:17 2021-12-03 11:14:40 2021-12-03 11:47:27 0:32:47 0:19:49 0:12:58 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2021-12-03T11:35:08.182072+0000 mon.a (mon.0) 182 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541577 2021-12-03 03:21:18 2021-12-03 11:15:30 2021-12-03 11:40:18 0:24:48 0:11:44 0:13:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} 3
pass 6541578 2021-12-03 03:21:18 2021-12-03 11:16:21 2021-12-03 11:42:14 0:25:53 0:12:28 0:13:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6541579 2021-12-03 03:21:19 2021-12-03 11:16:31 2021-12-03 12:05:29 0:48:58 0:37:24 0:11:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6541580 2021-12-03 03:21:20 2021-12-03 11:17:42 2021-12-03 11:47:08 0:29:26 0:13:43 0:15:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6541581 2021-12-03 03:21:21 2021-12-03 11:20:43 2021-12-03 11:42:54 0:22:11 0:10:18 0:11:53 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_discard} 1
pass 6541582 2021-12-03 03:21:22 2021-12-03 11:20:43 2021-12-03 11:53:21 0:32:38 0:17:21 0:15:17 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
fail 6541583 2021-12-03 03:21:23 2021-12-03 11:22:33 2021-12-03 11:54:31 0:31:58 0:19:37 0:12:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2021-12-03T11:51:50.085116+0000 mon.a (mon.0) 238 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6541584 2021-12-03 03:21:24 2021-12-03 11:22:34 2021-12-03 11:55:02 0:32:28 0:19:38 0:12:50 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2021-12-03T11:52:51.408611+0000 mon.a (mon.0) 284 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6541585 2021-12-03 03:21:24 2021-12-03 11:22:54 2021-12-03 11:55:52 0:32:58 0:19:33 0:13:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_enumerate} 3
Failure Reason:

"2021-12-03T11:54:51.950694+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 6541586 2021-12-03 03:21:25 2021-12-03 11:23:35 2021-12-03 11:56:19 0:32:44 0:19:36 0:13:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2021-12-03T11:54:15.251340+0000 mon.a (mon.0) 277 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541587 2021-12-03 03:21:26 2021-12-03 11:24:45 2021-12-03 11:48:57 0:24:12 0:12:20 0:11:52 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6541588 2021-12-03 03:21:27 2021-12-03 11:25:16 2021-12-03 11:49:38 0:24:22 0:11:08 0:13:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} 3
pass 6541589 2021-12-03 03:21:28 2021-12-03 11:25:56 2021-12-03 11:45:41 0:19:45 0:09:14 0:10:31 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} 1
pass 6541590 2021-12-03 03:21:28 2021-12-03 11:25:56 2021-12-03 12:01:24 0:35:28 0:22:49 0:12:39 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6541591 2021-12-03 03:21:29 2021-12-03 11:27:17 2021-12-03 11:51:44 0:24:27 0:12:04 0:12:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_symlinks} 3
fail 6541592 2021-12-03 03:21:30 2021-12-03 11:27:27 2021-12-03 12:00:44 0:33:17 0:19:40 0:13:37 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
Failure Reason:

"2021-12-03T11:47:14.861191+0000 mon.a (mon.0) 127 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541593 2021-12-03 03:21:31 2021-12-03 11:28:48 2021-12-03 11:58:36 0:29:48 0:16:17 0:13:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_concurrent} 3
pass 6541594 2021-12-03 03:21:32 2021-12-03 11:29:28 2021-12-03 11:51:40 0:22:12 0:11:02 0:11:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6541595 2021-12-03 03:21:33 2021-12-03 11:29:59 2021-12-03 12:00:22 0:30:23 0:19:21 0:11:02 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6541596 2021-12-03 03:21:33 2021-12-03 11:30:09 2021-12-03 11:52:30 0:22:21 0:11:12 0:11:09 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_image_read} 3
pass 6541597 2021-12-03 03:21:34 2021-12-03 11:30:49 2021-12-03 11:50:31 0:19:42 0:09:26 0:10:16 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_modprobe} 1
pass 6541598 2021-12-03 03:21:35 2021-12-03 11:30:50 2021-12-03 11:58:01 0:27:11 0:15:35 0:11:36 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6541599 2021-12-03 03:21:36 2021-12-03 11:31:50 2021-12-03 13:02:52 1:31:02 1:16:21 0:14:41 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6541600 2021-12-03 03:21:37 2021-12-03 11:34:51 2021-12-03 12:06:43 0:31:52 0:19:28 0:12:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2021-12-03T12:05:45.592869+0000 mon.a (mon.0) 245 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6541601 2021-12-03 03:21:37 2021-12-03 11:36:01 2021-12-03 12:09:12 0:33:11 0:19:54 0:13:17 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2021-12-03T12:06:32.763935+0000 mon.a (mon.0) 190 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541602 2021-12-03 03:21:38 2021-12-03 11:37:12 2021-12-03 11:59:31 0:22:19 0:10:45 0:11:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} 3
fail 6541603 2021-12-03 03:21:39 2021-12-03 11:37:22 2021-12-03 12:10:32 0:33:10 0:19:20 0:13:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_unmap} 3
Failure Reason:

"2021-12-03T12:07:42.819829+0000 mon.a (mon.0) 279 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541604 2021-12-03 03:21:40 2021-12-03 11:38:23 2021-12-03 12:03:00 0:24:37 0:11:36 0:13:01 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6541605 2021-12-03 03:21:41 2021-12-03 11:39:03 2021-12-03 12:18:36 0:39:33 0:26:04 0:13:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_simple_big} 3
pass 6541606 2021-12-03 03:21:42 2021-12-03 11:40:24 2021-12-03 11:59:49 0:19:25 0:09:24 0:10:01 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_msgr_segments} 1
pass 6541607 2021-12-03 03:21:43 2021-12-03 11:40:24 2021-12-03 12:16:48 0:36:24 0:24:57 0:11:27 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
fail 6541608 2021-12-03 03:21:43 2021-12-03 11:40:34 2021-12-03 12:12:08 0:31:34 0:19:35 0:11:59 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

"2021-12-03T12:10:47.423390+0000 mon.a (mon.0) 309 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541609 2021-12-03 03:21:44 2021-12-03 11:42:15 2021-12-03 12:21:43 0:39:28 0:26:34 0:12:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} 3
pass 6541610 2021-12-03 03:21:45 2021-12-03 11:43:25 2021-12-03 12:09:42 0:26:17 0:11:55 0:14:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6541611 2021-12-03 03:21:46 2021-12-03 11:45:26 2021-12-03 12:15:49 0:30:23 0:18:37 0:11:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6541612 2021-12-03 03:21:47 2021-12-03 11:45:46 2021-12-03 12:09:58 0:24:12 0:12:31 0:11:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_fallocate} 3
fail 6541613 2021-12-03 03:21:48 2021-12-03 11:45:47 2021-12-03 12:19:03 0:33:16 0:19:55 0:13:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} 3
Failure Reason:

"2021-12-03T12:17:12.123708+0000 mon.a (mon.0) 304 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6541614 2021-12-03 03:21:49 2021-12-03 11:46:07 2021-12-03 12:37:45 0:51:38 0:38:35 0:13:03 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
Failure Reason:

"2021-12-03T12:26:03.616126+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 31 sec, osd.1 has slow ops (SLOW_OPS)" in cluster log

fail 6541615 2021-12-03 03:21:50 2021-12-03 11:47:18 2021-12-03 12:19:41 0:32:23 0:20:08 0:12:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
Failure Reason:

"2021-12-03T12:17:48.196889+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541616 2021-12-03 03:21:51 2021-12-03 11:47:38 2021-12-03 12:10:04 0:22:26 0:09:57 0:12:29 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} 1
pass 6541617 2021-12-03 03:21:52 2021-12-03 11:47:38 2021-12-03 12:18:51 0:31:13 0:19:48 0:11:25 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6541618 2021-12-03 03:21:53 2021-12-03 11:48:59 2021-12-03 12:11:21 0:22:22 0:11:47 0:10:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_namespaces} 3
pass 6541619 2021-12-03 03:21:53 2021-12-03 11:49:39 2021-12-03 13:20:05 1:30:26 1:16:18 0:14:08 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6541620 2021-12-03 03:21:54 2021-12-03 11:51:50 2021-12-03 12:41:49 0:49:59 0:39:26 0:10:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6541621 2021-12-03 03:21:55 2021-12-03 11:51:50 2021-12-03 12:18:27 0:26:37 0:15:20 0:11:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
fail 6541622 2021-12-03 03:21:56 2021-12-03 11:52:41 2021-12-03 12:26:04 0:33:23 0:19:24 0:13:59 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
Failure Reason:

"2021-12-03T12:12:58.146539+0000 mon.a (mon.0) 113 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541623 2021-12-03 03:21:57 2021-12-03 11:53:21 2021-12-03 12:18:37 0:25:16 0:12:01 0:13:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netns} 3
pass 6541624 2021-12-03 03:21:58 2021-12-03 11:54:42 2021-12-03 12:16:24 0:21:42 0:10:02 0:11:40 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_read_only} 1
pass 6541625 2021-12-03 03:21:59 2021-12-03 11:54:42 2021-12-03 12:27:02 0:32:20 0:19:13 0:13:07 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6541626 2021-12-03 03:22:00 2021-12-03 11:55:12 2021-12-03 12:45:35 0:50:23 0:37:55 0:12:28 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/pggrow thrashosds-health workloads/rbd_fio} 3
pass 6541627 2021-12-03 03:22:00 2021-12-03 11:55:23 2021-12-03 12:20:19 0:24:56 0:12:16 0:12:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6541628 2021-12-03 03:22:01 2021-12-03 11:56:03 2021-12-03 12:19:54 0:23:51 0:10:58 0:12:53 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6541629 2021-12-03 03:22:02 2021-12-03 11:56:23 2021-12-03 12:30:01 0:33:38 0:19:44 0:13:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

"2021-12-03T12:27:28.262016+0000 mon.a (mon.0) 303 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541630 2021-12-03 03:22:03 2021-12-03 11:58:04 2021-12-03 12:23:47 0:25:43 0:11:19 0:14:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6541631 2021-12-03 03:22:04 2021-12-03 11:58:44 2021-12-03 12:32:01 0:33:17 0:19:55 0:13:22 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2021-12-03T12:19:22.685173+0000 mon.a (mon.0) 120 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541632 2021-12-03 03:22:05 2021-12-03 11:59:35 2021-12-03 12:24:35 0:25:00 0:11:42 0:13:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} 3
pass 6541633 2021-12-03 03:22:06 2021-12-03 12:00:25 2021-12-03 12:21:56 0:21:31 0:10:20 0:11:11 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_whole_object_zeroout} 1
pass 6541634 2021-12-03 03:22:06 2021-12-03 12:00:26 2021-12-03 12:32:49 0:32:23 0:20:56 0:11:27 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6541635 2021-12-03 03:22:07 2021-12-03 12:00:46 2021-12-03 12:23:19 0:22:33 0:10:45 0:11:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_kernel} 3
pass 6541636 2021-12-03 03:22:08 2021-12-03 12:31:08 969 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6541637 2021-12-03 03:22:09 2021-12-03 12:03:07 2021-12-03 12:27:36 0:24:29 0:10:44 0:13:45 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6541638 2021-12-03 03:22:10 2021-12-03 12:05:38 2021-12-03 12:33:38 0:28:00 0:15:29 0:12:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} 3
pass 6541639 2021-12-03 03:22:10 2021-12-03 12:06:48 2021-12-03 12:35:19 0:28:31 0:13:09 0:15:22 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6541640 2021-12-03 03:22:11 2021-12-03 12:09:19 2021-12-03 12:50:04 0:40:45 0:27:38 0:13:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_simple_big} 3
pass 6541641 2021-12-03 03:22:12 2021-12-03 12:09:50 2021-12-03 12:31:35 0:21:45 0:09:46 0:11:59 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} 1
fail 6541642 2021-12-03 03:22:13 2021-12-03 12:09:50 2021-12-03 12:42:14 0:32:24 0:18:41 0:13:43 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2021-12-03T12:40:45.932049+0000 mon.a (mon.0) 451 : cluster [WRN] Health check failed: 5 osds down (OSD_DOWN)" in cluster log

pass 6541643 2021-12-03 03:22:14 2021-12-03 12:10:00 2021-12-03 12:49:45 0:39:45 0:28:31 0:11:14 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_fio} 3
pass 6541644 2021-12-03 03:22:15 2021-12-03 12:10:01 2021-12-03 12:43:14 0:33:13 0:19:23 0:13:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_data_pool} 3
pass 6541645 2021-12-03 03:22:16 2021-12-03 12:10:41 2021-12-03 12:33:46 0:23:05 0:10:21 0:12:44 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} 2
pass 6541646 2021-12-03 03:22:17 2021-12-03 12:11:21 2021-12-03 12:45:42 0:34:21 0:22:01 0:12:20 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} 3
fail 6541647 2021-12-03 03:22:17 2021-12-03 12:12:12 2021-12-03 12:48:06 0:35:54 0:19:53 0:16:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2021-12-03T12:35:07.094342+0000 mon.a (mon.0) 116 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6541648 2021-12-03 03:22:18 2021-12-03 12:15:53 2021-12-03 12:48:23 0:32:30 0:19:56 0:12:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2021-12-03T12:35:12.767682+0000 mon.a (mon.0) 119 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541649 2021-12-03 03:22:19 2021-12-03 12:16:23 2021-12-03 12:39:10 0:22:47 0:11:50 0:10:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_fallocate} 3
pass 6541650 2021-12-03 03:22:20 2021-12-03 12:16:54 2021-12-03 12:41:47 0:24:53 0:12:03 0:12:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6541651 2021-12-03 03:22:21 2021-12-03 12:18:04 2021-12-03 13:09:46 0:51:42 0:37:12 0:14:30 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6541652 2021-12-03 03:22:22 2021-12-03 12:18:14 2021-12-03 12:44:22 0:26:08 0:14:13 0:11:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6541653 2021-12-03 03:22:23 2021-12-03 12:18:35 2021-12-03 12:40:43 0:22:08 0:10:09 0:11:59 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_discard} 1
pass 6541654 2021-12-03 03:22:24 2021-12-03 12:18:45 2021-12-03 12:48:44 0:29:59 0:18:39 0:11:20 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6541655 2021-12-03 03:22:25 2021-12-03 12:18:46 2021-12-03 13:40:58 1:22:12 1:09:24 0:12:48 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6541656 2021-12-03 03:22:25 2021-12-03 12:18:56 2021-12-03 12:52:17 0:33:21 0:20:08 0:13:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2021-12-03T12:50:47.027997+0000 mon.a (mon.0) 227 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541657 2021-12-03 03:22:26 2021-12-03 12:19:06 2021-12-03 13:55:49 1:36:43 1:23:46 0:12:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6541658 2021-12-03 03:22:27 2021-12-03 12:19:47 2021-12-03 13:11:42 0:51:55 0:39:08 0:12:47 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_enumerate} 3
fail 6541659 2021-12-03 03:22:28 2021-12-03 12:19:57 2021-12-03 12:52:50 0:32:53 0:20:03 0:12:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2021-12-03T12:50:53.281058+0000 mon.a (mon.0) 300 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541660 2021-12-03 03:22:29 2021-12-03 12:20:27 2021-12-03 12:45:44 0:25:17 0:12:31 0:12:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6541661 2021-12-03 03:22:30 2021-12-03 12:21:18 2021-12-03 12:46:15 0:24:57 0:11:46 0:13:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netns} 3
pass 6541662 2021-12-03 03:22:30 2021-12-03 12:21:48 2021-12-03 12:43:30 0:21:42 0:09:48 0:11:54 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_huge_image} 1
pass 6541663 2021-12-03 03:22:31 2021-12-03 12:21:49 2021-12-03 12:56:24 0:34:35 0:22:35 0:12:00 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6541664 2021-12-03 03:22:32 2021-12-03 12:21:59 2021-12-03 12:47:21 0:25:22 0:12:14 0:13:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6541665 2021-12-03 03:22:33 2021-12-03 12:23:20 2021-12-03 12:48:35 0:25:15 0:12:22 0:12:53 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6541666 2021-12-03 03:22:34 2021-12-03 12:23:50 2021-12-03 12:52:50 0:29:00 0:15:56 0:13:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} 3
fail 6541667 2021-12-03 03:22:35 2021-12-03 12:24:41 2021-12-03 12:58:41 0:34:00 0:19:36 0:14:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2021-12-03T12:56:03.921765+0000 mon.a (mon.0) 281 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541668 2021-12-03 03:22:35 2021-12-03 12:26:11 2021-12-03 12:59:13 0:33:02 0:19:12 0:13:50 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6541669 2021-12-03 03:22:36 2021-12-03 12:27:12 2021-12-03 12:51:19 0:24:07 0:11:23 0:12:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_image_read} 3
pass 6541670 2021-12-03 03:22:37 2021-12-03 12:27:22 2021-12-03 12:47:03 0:19:41 0:09:16 0:10:25 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} 1
fail 6541671 2021-12-03 03:22:38 2021-12-03 12:27:22 2021-12-03 12:59:40 0:32:18 0:19:50 0:12:28 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

"2021-12-03T12:57:11.105864+0000 mon.a (mon.0) 302 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541672 2021-12-03 03:22:39 2021-12-03 12:27:43 2021-12-03 12:54:04 0:26:21 0:10:42 0:15:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} 3
pass 6541673 2021-12-03 03:22:40 2021-12-03 12:30:03 2021-12-03 13:01:12 0:31:09 0:18:01 0:13:08 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6541674 2021-12-03 03:22:41 2021-12-03 12:31:14 2021-12-03 12:53:01 0:21:47 0:10:34 0:11:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6541675 2021-12-03 03:22:41 2021-12-03 12:31:14 2021-12-03 13:00:18 0:29:04 0:15:35 0:13:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_unmap} 3
pass 6541676 2021-12-03 03:22:42 2021-12-03 12:32:05 2021-12-03 12:57:06 0:25:01 0:11:57 0:13:04 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6541677 2021-12-03 03:22:43 2021-12-03 12:32:55 2021-12-03 13:14:29 0:41:34 0:27:34 0:14:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} 3
pass 6541678 2021-12-03 03:22:44 2021-12-03 12:33:46 2021-12-03 12:55:32 0:21:46 0:09:52 0:11:54 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_msgr_segments} 1
pass 6541679 2021-12-03 03:22:45 2021-12-03 12:33:46 2021-12-03 13:08:23 0:34:37 0:22:50 0:11:47 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
fail 6541680 2021-12-03 03:22:46 2021-12-03 12:33:56 2021-12-03 13:07:24 0:33:28 0:19:40 0:13:48 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

"2021-12-03T12:54:49.105449+0000 mon.a (mon.0) 116 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6541681 2021-12-03 03:22:47 2021-12-03 12:35:27 2021-12-03 13:09:53 0:34:26 0:19:34 0:14:52 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2021-12-03T12:56:42.191070+0000 mon.a (mon.0) 118 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6541682 2021-12-03 03:22:48 2021-12-03 12:37:48 2021-12-03 13:09:32 0:31:44 0:19:26 0:12:18 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy} msgr-failures/many tasks/rbd_xfstests} 4
Failure Reason:

"2021-12-03T13:08:22.226868+0000 mon.a (mon.0) 217 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541683 2021-12-03 03:22:48 2021-12-03 12:39:18 2021-12-03 14:30:53 1:51:35 1:38:32 0:13:03 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6541684 2021-12-03 03:22:49 2021-12-03 12:40:09 2021-12-03 13:07:00 0:26:51 0:12:23 0:14:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6541685 2021-12-03 03:22:50 2021-12-03 12:40:49 2021-12-03 13:13:47 0:32:58 0:19:34 0:13:24 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6541686 2021-12-03 03:22:51 2021-12-03 12:41:50 2021-12-03 13:05:36 0:23:46 0:12:24 0:11:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} 3
pass 6541687 2021-12-03 03:22:52 2021-12-03 12:41:50 2021-12-03 13:06:23 0:24:33 0:12:07 0:12:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6541688 2021-12-03 03:22:53 2021-12-03 12:42:20 2021-12-03 13:33:46 0:51:26 0:38:34 0:12:52 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
fail 6541689 2021-12-03 03:22:53 2021-12-03 12:43:21 2021-12-03 13:16:12 0:32:51 0:19:39 0:13:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
Failure Reason:

"2021-12-03T13:13:51.110842+0000 mon.a (mon.0) 283 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541690 2021-12-03 03:22:54 2021-12-03 12:44:01 2021-12-03 13:04:25 0:20:24 0:09:03 0:11:21 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_parent_overlap} 1
pass 6541691 2021-12-03 03:22:55 2021-12-03 12:44:32 2021-12-03 13:17:52 0:33:20 0:19:43 0:13:37 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6541692 2021-12-03 03:22:56 2021-12-03 12:45:42 2021-12-03 13:10:34 0:24:52 0:11:39 0:13:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} 3
pass 6541693 2021-12-03 03:22:57 2021-12-03 12:45:52 2021-12-03 14:15:58 1:30:06 1:18:16 0:11:50 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
fail 6541694 2021-12-03 03:22:57 2021-12-03 12:45:53 2021-12-03 13:18:51 0:32:58 0:19:52 0:13:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_enumerate} 3
Failure Reason:

"2021-12-03T13:17:44.560513+0000 mon.a (mon.0) 242 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541695 2021-12-03 03:22:58 2021-12-03 12:46:23 2021-12-03 13:14:57 0:28:34 0:15:29 0:13:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6541696 2021-12-03 03:22:59 2021-12-03 12:47:14 2021-12-03 13:13:35 0:26:21 0:13:07 0:13:14 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
fail 6541697 2021-12-03 03:23:00 2021-12-03 12:47:24 2021-12-03 13:20:29 0:33:05 0:20:00 0:13:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2021-12-03T13:18:49.685602+0000 mon.a (mon.0) 300 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541698 2021-12-03 03:23:01 2021-12-03 12:48:15 2021-12-03 13:10:57 0:22:42 0:09:55 0:12:47 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} 1
fail 6541699 2021-12-03 03:23:02 2021-12-03 12:48:25 2021-12-03 13:20:43 0:32:18 0:19:35 0:12:43 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2021-12-03T13:18:35.862624+0000 mon.a (mon.0) 269 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6541700 2021-12-03 03:23:02 2021-12-03 12:48:45 2021-12-03 13:13:49 0:25:04 0:12:18 0:12:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6541701 2021-12-03 03:23:03 2021-12-03 12:48:46 2021-12-03 13:15:18 0:26:32 0:11:05 0:15:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6541702 2021-12-03 03:23:04 2021-12-03 12:49:46 2021-12-03 13:18:04 0:28:18 0:15:54 0:12:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_concurrent} 3
pass 6541703 2021-12-03 03:23:05 2021-12-03 12:50:06 2021-12-03 13:15:25 0:25:19 0:11:06 0:14:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6541704 2021-12-03 03:23:06 2021-12-03 12:51:27 2021-12-03 13:23:52 0:32:25 0:20:51 0:11:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
fail 6541705 2021-12-03 03:23:07 2021-12-03 12:52:17 2021-12-03 13:25:39 0:33:22 0:19:27 0:13:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2021-12-03T13:14:19.120577+0000 mon.a (mon.0) 135 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541706 2021-12-03 03:23:07 2021-12-03 12:52:28 2021-12-03 13:14:02 0:21:34 0:09:47 0:11:47 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_whole_object_zeroout} 1
pass 6541707 2021-12-03 03:23:08 2021-12-03 13:21:13 962 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6541708 2021-12-03 03:23:09 2021-12-03 12:52:58 2021-12-03 13:44:59 0:52:01 0:38:08 0:13:53 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} 3
pass 6541709 2021-12-03 03:23:10 2021-12-03 12:52:59 2021-12-03 13:16:56 0:23:57 0:11:00 0:12:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_kernel} 3
pass 6541710 2021-12-03 03:23:11 2021-12-03 12:53:09 2021-12-03 13:22:07 0:28:58 0:16:12 0:12:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6541711 2021-12-03 03:23:11 2021-12-03 12:54:10 2021-12-03 13:18:55 0:24:45 0:10:35 0:14:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6541712 2021-12-03 03:23:12 2021-12-03 12:56:30 2021-12-03 13:25:36 0:29:06 0:15:37 0:13:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_unmap} 3
pass 6541713 2021-12-03 03:23:13 2021-12-03 12:57:11 2021-12-03 13:25:25 0:28:14 0:13:35 0:14:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6541714 2021-12-03 03:23:14 2021-12-03 12:58:51 2021-12-03 13:41:30 0:42:39 0:29:28 0:13:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_simple_big} 3
pass 6541715 2021-12-03 03:23:15 2021-12-03 12:59:22 2021-12-03 13:21:07 0:21:45 0:09:52 0:11:53 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_deep_flatten} 1
pass 6541716 2021-12-03 03:23:15 2021-12-03 12:59:22 2021-12-03 13:35:44 0:36:22 0:25:21 0:11:01 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
fail 6541717 2021-12-03 03:23:16 2021-12-03 12:59:42 2021-12-03 13:30:35 0:30:53 0:19:23 0:11:30 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} 3
Failure Reason:

"2021-12-03T13:29:41.650627+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541718 2021-12-03 03:23:17 2021-12-03 13:00:23 2021-12-03 13:31:09 0:30:46 0:19:03 0:11:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} 3
pass 6541719 2021-12-03 03:23:18 2021-12-03 13:01:13 2021-12-03 13:24:50 0:23:37 0:09:59 0:13:38 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} 2
pass 6541720 2021-12-03 03:23:19 2021-12-03 13:02:54 2021-12-03 13:39:53 0:36:59 0:21:49 0:15:10 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} 3
fail 6541721 2021-12-03 03:23:19 2021-12-03 13:05:45 2021-12-03 13:38:33 0:32:48 0:19:51 0:12:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2021-12-03T13:35:35.309693+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541722 2021-12-03 03:23:20 2021-12-03 13:06:25 2021-12-03 13:39:19 0:32:54 0:19:47 0:13:07 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6541723 2021-12-03 03:23:21 2021-12-03 13:07:06 2021-12-03 13:31:27 0:24:21 0:11:56 0:12:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_fallocate} 3
fail 6541724 2021-12-03 03:23:22 2021-12-03 13:07:26 2021-12-03 13:40:53 0:33:27 0:19:21 0:14:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2021-12-03T13:38:05.731685+0000 mon.a (mon.0) 232 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541725 2021-12-03 03:23:23 2021-12-03 13:08:27 2021-12-03 13:59:40 0:51:13 0:37:28 0:13:45 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6541726 2021-12-03 03:23:23 2021-12-03 13:09:37 2021-12-03 13:35:42 0:26:05 0:14:21 0:11:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6541727 2021-12-03 03:23:24 2021-12-03 13:09:38 2021-12-03 13:33:06 0:23:28 0:09:58 0:13:30 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} 1
fail 6541728 2021-12-03 03:23:25 2021-12-03 13:09:48 2021-12-03 13:40:12 0:30:24 0:18:26 0:11:58 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

"2021-12-03T13:28:56.412832+0000 mon.a (mon.0) 130 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541729 2021-12-03 03:23:26 2021-12-03 13:09:58 2021-12-03 13:35:05 0:25:07 0:12:48 0:12:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_namespaces} 3
fail 6541730 2021-12-03 03:23:27 2021-12-03 13:10:39 2021-12-03 13:43:39 0:33:00 0:20:04 0:12:56 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2021-12-03T13:30:42.191089+0000 mon.a (mon.0) 223 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541731 2021-12-03 03:23:28 2021-12-03 13:10:59 2021-12-03 14:01:50 0:50:51 0:38:31 0:12:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} 3
fail 6541732 2021-12-03 03:23:28 2021-12-03 13:11:50 2021-12-03 13:45:51 0:34:01 0:19:32 0:14:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2021-12-03T13:33:46.724051+0000 mon.a (mon.0) 140 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541733 2021-12-03 03:23:29 2021-12-03 13:13:40 2021-12-03 13:40:51 0:27:11 0:12:24 0:14:47 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6541734 2021-12-03 03:23:30 2021-12-03 13:13:50 2021-12-03 13:37:38 0:23:48 0:12:16 0:11:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netns} 3
pass 6541735 2021-12-03 03:23:31 2021-12-03 13:13:51 2021-12-03 13:35:53 0:22:02 0:09:46 0:12:16 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_huge_image} 1
pass 6541736 2021-12-03 03:23:32 2021-12-03 13:14:11 2021-12-03 13:51:11 0:37:00 0:21:34 0:15:26 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6541737 2021-12-03 03:23:32 2021-12-03 13:14:32 2021-12-03 15:00:56 1:46:24 1:33:19 0:13:05 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
dead 6541738 2021-12-03 03:23:33 2021-12-03 13:15:02 2021-12-03 13:33:16 0:18:14 0:04:55 0:13:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} 3
Failure Reason:

{'Failure object was': {'smithi043.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"}

pass 6541739 2021-12-03 03:23:34 2021-12-03 13:15:22 2021-12-03 13:39:49 0:24:27 0:11:22 0:13:05 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6541740 2021-12-03 03:23:35 2021-12-03 13:15:33 2021-12-03 13:45:58 0:30:25 0:15:52 0:14:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_concurrent} 3
pass 6541741 2021-12-03 03:23:36 2021-12-03 13:16:23 2021-12-03 13:40:51 0:24:28 0:10:53 0:13:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6541742 2021-12-03 03:23:37 2021-12-03 13:17:04 2021-12-03 13:50:02 0:32:58 0:19:16 0:13:42 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6541743 2021-12-03 03:23:37 2021-12-03 13:17:54 2021-12-03 13:42:14 0:24:20 0:11:27 0:12:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} 3
pass 6541744 2021-12-03 03:23:38 2021-12-03 13:18:14 2021-12-03 13:40:44 0:22:30 0:09:59 0:12:31 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_modprobe} 1
pass 6541745 2021-12-03 03:23:39 2021-12-03 13:18:55 2021-12-03 13:51:36 0:32:41 0:18:52 0:13:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6541746 2021-12-03 03:23:40 2021-12-03 13:19:05 2021-12-03 13:42:20 0:23:15 0:10:53 0:12:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_kernel} 3
fail 6541747 2021-12-03 03:23:41 2021-12-03 13:20:06 2021-12-03 13:52:54 0:32:48 0:19:21 0:13:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2021-12-03T13:50:52.526862+0000 mon.a (mon.0) 277 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541748 2021-12-03 03:23:42 2021-12-03 13:20:36 2021-12-03 13:42:36 0:22:00 0:10:33 0:11:27 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6541749 2021-12-03 03:23:42 2021-12-03 13:20:46 2021-12-03 13:49:44 0:28:58 0:15:44 0:13:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} 3
pass 6541750 2021-12-03 03:23:43 2021-12-03 13:45:31 720 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6541751 2021-12-03 03:23:44 2021-12-03 13:21:17 2021-12-03 14:00:16 0:38:59 0:26:34 0:12:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_simple_big} 3
pass 6541752 2021-12-03 03:23:45 2021-12-03 13:22:18 2021-12-03 13:43:39 0:21:21 0:09:12 0:12:09 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} 1
pass 6541753 2021-12-03 03:23:46 2021-12-03 13:24:00 2021-12-03 14:01:02 0:37:02 0:24:49 0:12:13 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6541754 2021-12-03 03:23:46 2021-12-03 13:25:00 2021-12-03 14:10:15 0:45:15 0:31:20 0:13:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_fio} 3
pass 6541755 2021-12-03 03:23:47 2021-12-03 13:25:32 2021-12-03 14:02:26 0:36:54 0:24:00 0:12:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_data_pool} 3
pass 6541756 2021-12-03 03:23:48 2021-12-03 13:25:42 2021-12-03 13:52:08 0:26:26 0:12:04 0:14:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6541757 2021-12-03 03:23:49 2021-12-03 13:25:42 2021-12-03 14:02:28 0:36:46 0:18:58 0:17:48 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6541758 2021-12-03 03:23:50 2021-12-03 13:30:43 2021-12-03 13:55:10 0:24:27 0:12:56 0:11:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_fallocate} 3
pass 6541759 2021-12-03 03:23:51 2021-12-03 13:31:14 2021-12-03 13:55:35 0:24:21 0:12:39 0:11:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6541760 2021-12-03 03:23:52 2021-12-03 13:31:34 2021-12-03 14:24:41 0:53:07 0:38:10 0:14:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6541761 2021-12-03 03:23:52 2021-12-03 13:33:15 2021-12-03 13:58:59 0:25:44 0:15:13 0:10:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6541762 2021-12-03 03:23:53 2021-12-03 13:33:25 2021-12-03 13:54:22 0:20:57 0:09:50 0:11:07 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_parent_overlap} 1
pass 6541763 2021-12-03 03:23:54 2021-12-03 13:33:55 2021-12-03 14:03:32 0:29:37 0:16:48 0:12:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6541764 2021-12-03 03:23:55 2021-12-03 13:35:06 2021-12-03 14:31:31 0:56:25 0:43:35 0:12:50 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/upmap thrashosds-health workloads/rbd_fio} 3
pass 6541765 2021-12-03 03:23:56 2021-12-03 13:35:46 2021-12-03 13:59:59 0:24:13 0:12:02 0:12:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_namespaces} 3
pass 6541766 2021-12-03 03:23:57 2021-12-03 13:35:47 2021-12-03 15:03:59 1:28:12 1:15:28 0:12:44 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6541767 2021-12-03 03:23:58 2021-12-03 13:35:57 2021-12-03 14:33:37 0:57:40 0:44:39 0:13:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6541768 2021-12-03 03:23:58 2021-12-03 13:37:48 2021-12-03 14:04:36 0:26:48 0:15:33 0:11:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6541769 2021-12-03 03:23:59 2021-12-03 13:38:38 2021-12-03 14:03:42 0:25:04 0:13:00 0:12:04 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6541770 2021-12-03 03:24:00 2021-12-03 13:39:29 2021-12-03 14:03:59 0:24:30 0:11:31 0:12:59 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netns} 3
pass 6541771 2021-12-03 03:24:01 2021-12-03 13:39:59 2021-12-03 13:59:42 0:19:43 0:09:37 0:10:06 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_read_only} 1
pass 6541772 2021-12-03 03:24:02 2021-12-03 13:39:59 2021-12-03 14:16:11 0:36:12 0:23:03 0:13:09 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6541773 2021-12-03 03:24:03 2021-12-03 13:40:20 2021-12-03 14:07:16 0:26:56 0:12:38 0:14:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6541774 2021-12-03 03:24:03 2021-12-03 13:40:50 2021-12-03 14:04:46 0:23:56 0:11:28 0:12:28 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6541775 2021-12-03 03:24:04 2021-12-03 13:41:00 2021-12-03 14:11:20 0:30:20 0:17:22 0:12:58 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} 3
pass 6541776 2021-12-03 03:24:05 2021-12-03 13:41:01 2021-12-03 14:05:08 0:24:07 0:11:04 0:13:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6541777 2021-12-03 03:24:06 2021-12-03 13:41:01 2021-12-03 14:15:45 0:34:44 0:22:03 0:12:41 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
fail 6541778 2021-12-03 03:24:07 2021-12-03 13:41:01 2021-12-03 14:15:45 0:34:44 0:20:09 0:14:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2021-12-03T14:12:49.873173+0000 mon.a (mon.0) 221 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6541779 2021-12-03 03:24:08 2021-12-03 13:41:32 2021-12-03 14:04:19 0:22:47 0:10:14 0:12:33 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} 1
pass 6541780 2021-12-03 03:24:09 2021-12-03 13:42:22 2021-12-03 14:12:49 0:30:27 0:17:54 0:12:33 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6541781 2021-12-03 03:24:10 2021-12-03 13:42:23 2021-12-03 14:07:44 0:25:21 0:10:56 0:14:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} 3
pass 6541782 2021-12-03 03:24:10 2021-12-03 13:42:43 2021-12-03 14:13:43 0:31:00 0:16:56 0:14:04 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
fail 6541783 2021-12-03 03:24:11 2021-12-03 13:43:45 2021-12-03 14:15:39 0:31:54 0:19:35 0:12:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2021-12-03T14:03:31.314513+0000 mon.a (mon.0) 132 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6541784 2021-12-03 03:24:12 2021-12-03 13:43:45 2021-12-03 14:13:01 0:29:16 0:16:20 0:12:56 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_unmap} 3
pass 6541785 2021-12-03 03:24:13 2021-12-03 13:45:06 2021-12-03 14:12:06 0:27:00 0:13:28 0:13:32 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
fail 6541786 2021-12-03 03:24:14 2021-12-03 13:45:36 2021-12-03 14:17:46 0:32:10 0:20:15 0:11:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} 3
Failure Reason:

"2021-12-03T14:16:03.439872+0000 mon.a (mon.0) 235 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log