Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 6824124 2022-05-06 03:20:13 2022-05-09 00:03:21 2022-05-09 00:22:31 0:19:10 0:09:14 0:09:56 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_deep_flatten} 1
pass 6824125 2022-05-06 03:20:14 2022-05-09 00:03:21 2022-05-09 00:34:48 0:31:27 0:18:51 0:12:36 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6824126 2022-05-06 03:20:15 2022-05-09 00:05:22 2022-05-09 00:27:47 0:22:25 0:09:48 0:12:37 smithi master krbd/ms_modeless/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_default_map_options} 3
fail 6824127 2022-05-06 03:20:16 2022-05-09 00:06:02 2022-05-09 00:39:25 0:33:23 0:18:46 0:14:37 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_fio} 3
Failure Reason:

"2022-05-09T00:37:39.003079+0000 mon.a (mon.0) 197 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824128 2022-05-06 03:20:17 2022-05-09 00:08:53 2022-05-09 00:39:47 0:30:54 0:18:05 0:12:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_data_pool} 3
fail 6824129 2022-05-06 03:20:18 2022-05-09 00:09:44 2022-05-09 04:49:51 4:40:07 4:26:22 0:13:45 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy} msgr-failures/few tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi114 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude0gsmc5ny -r -- '-g auto -g blockdev -x clone'"

pass 6824130 2022-05-06 03:20:19 2022-05-09 00:12:04 2022-05-09 01:05:06 0:53:02 0:41:21 0:11:41 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/backoff thrashosds-health workloads/rbd_fio} 3
fail 6824131 2022-05-06 03:20:20 2022-05-09 00:13:25 2022-05-09 00:22:25 0:09:00 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major

pass 6824132 2022-05-06 03:20:21 2022-05-09 00:14:45 2022-05-09 00:33:23 0:18:38 0:08:14 0:10:24 smithi master krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} 1
pass 6824133 2022-05-06 03:20:22 2022-05-09 00:15:16 2022-05-09 00:37:44 0:22:28 0:11:17 0:11:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_exclusive_option} 3
fail 6824134 2022-05-06 03:20:23 2022-05-09 00:15:16 2022-05-09 00:44:52 0:29:36 0:18:46 0:10:50 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-05-09T00:32:02.699140+0000 mon.a (mon.0) 165 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824135 2022-05-06 03:20:24 2022-05-06 17:36:36 2022-05-06 18:00:27 0:23:51 0:10:45 0:13:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} 3
fail 6824136 2022-05-06 03:20:25 2022-05-09 00:15:16 2022-05-09 00:45:59 0:30:43 0:18:36 0:12:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-05-09T00:32:50.420701+0000 mon.a (mon.0) 197 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824137 2022-05-06 03:20:26 2022-05-09 00:16:27 2022-05-09 01:06:55 0:50:28 0:36:30 0:13:58 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6824138 2022-05-06 03:20:27 2022-05-09 00:19:08 2022-05-09 00:44:38 0:25:30 0:13:18 0:12:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6824139 2022-05-06 03:20:28 2022-05-09 00:20:38 2022-05-09 00:40:05 0:19:27 0:09:10 0:10:17 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_discard} 1
pass 6824140 2022-05-06 03:20:29 2022-05-09 00:20:49 2022-05-09 00:49:09 0:28:20 0:16:31 0:11:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
fail 6824141 2022-05-06 03:20:30 2022-05-09 00:21:39 2022-05-09 00:52:23 0:30:44 0:18:39 0:12:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2022-05-09T00:50:18.912374+0000 mon.a (mon.0) 248 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6824142 2022-05-06 03:20:31 2022-05-09 00:22:30 2022-05-09 01:45:19 1:22:49 1:12:03 0:10:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2022-05-09T01:23:30.023378+0000 osd.3 (osd.3) 3 : cluster [WRN] 1 slow requests (by type [ 'waiting for sub ops' : 1 ] most affected pool [ 'rbd' : 1 ])" in cluster log

pass 6824143 2022-05-06 03:20:32 2022-05-09 00:23:50 2022-05-09 01:14:30 0:50:40 0:38:24 0:12:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_enumerate} 3
fail 6824144 2022-05-06 03:20:33 2022-05-09 00:25:11 2022-05-09 00:57:53 0:32:42 0:18:36 0:14:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2022-05-09T00:45:07.054545+0000 mon.a (mon.0) 120 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824145 2022-05-06 03:20:34 2022-05-09 00:27:51 2022-05-09 00:49:23 0:21:32 0:11:05 0:10:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
fail 6824146 2022-05-06 03:20:35 2022-05-09 00:27:52 2022-05-09 00:59:10 0:31:18 0:18:15 0:13:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} 3
Failure Reason:

"2022-05-09T00:56:17.344024+0000 mon.a (mon.0) 310 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824147 2022-05-06 03:20:36 2022-05-09 00:29:42 2022-05-09 00:46:48 0:17:06 0:08:08 0:08:58 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} 1
pass 6824148 2022-05-06 03:20:37 2022-05-09 00:29:43 2022-05-09 01:06:17 0:36:34 0:21:07 0:15:27 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6824149 2022-05-06 03:20:38 2022-05-09 00:34:55 2022-05-09 00:58:40 0:23:45 0:11:27 0:12:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_symlinks} 3
fail 6824150 2022-05-06 03:20:39 2022-05-09 00:36:26 2022-05-09 01:08:19 0:31:53 0:18:26 0:13:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
Failure Reason:

"2022-05-09T00:55:43.835673+0000 mon.a (mon.0) 114 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824151 2022-05-06 03:20:40 2022-05-09 00:37:46 2022-05-09 01:05:47 0:28:01 0:15:18 0:12:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_concurrent} 3
pass 6824152 2022-05-06 03:20:41 2022-05-09 00:39:27 2022-05-09 01:02:06 0:22:39 0:10:26 0:12:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6824153 2022-05-06 03:20:42 2022-05-09 00:39:57 2022-05-09 01:11:09 0:31:12 0:19:09 0:12:03 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6824154 2022-05-06 03:20:43 2022-05-09 00:41:30 2022-05-09 01:06:51 0:25:21 0:09:50 0:15:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_image_read} 3
pass 6824155 2022-05-06 03:20:44 2022-05-09 00:44:46 2022-05-09 01:02:18 0:17:32 0:07:58 0:09:34 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_modprobe} 1
pass 6824156 2022-05-06 03:20:45 2022-05-09 00:45:06 2022-05-09 01:11:26 0:26:20 0:15:21 0:10:59 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6824157 2022-05-06 03:20:46 2022-05-09 00:46:06 2022-05-09 01:53:16 1:07:10 0:56:44 0:10:26 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6824158 2022-05-06 03:20:47 2022-05-09 00:46:07 2022-05-09 01:15:31 0:29:24 0:19:11 0:10:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-05-09T01:03:34.588654+0000 mon.a (mon.0) 114 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6824159 2022-05-06 03:20:48 2022-05-09 00:46:07 2022-05-09 01:16:45 0:30:38 0:18:44 0:11:54 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2022-05-09T01:15:12.332177+0000 mon.a (mon.0) 303 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824160 2022-05-06 03:20:50 2022-05-09 00:46:57 2022-05-09 01:09:20 0:22:23 0:10:33 0:11:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} 3
fail 6824161 2022-05-06 03:20:51 2022-05-09 00:47:48 2022-05-09 01:18:45 0:30:57 0:18:30 0:12:27 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_map_unmap} 3
Failure Reason:

"2022-05-09T01:16:09.433996+0000 mon.a (mon.0) 326 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824162 2022-05-06 03:20:52 2022-05-09 00:49:18 2022-05-09 01:11:03 0:21:45 0:11:07 0:10:38 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6824163 2022-05-06 03:20:53 2022-05-09 00:49:29 2022-05-09 01:24:06 0:34:37 0:22:53 0:11:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_simple_big} 3
pass 6824164 2022-05-06 03:20:53 2022-05-09 00:50:29 2022-05-09 01:12:13 0:21:44 0:09:07 0:12:37 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_msgr_segments} 1
pass 6824165 2022-05-06 03:20:54 2022-05-09 00:52:30 2022-05-09 01:36:00 0:43:30 0:24:18 0:19:12 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
fail 6824166 2022-05-06 03:20:55 2022-05-09 00:58:01 2022-05-09 01:30:51 0:32:50 0:18:41 0:14:09 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

"2022-05-09T01:18:32.800678+0000 mon.a (mon.0) 138 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6824167 2022-05-06 03:20:56 2022-05-09 00:58:41 2022-05-09 01:30:31 0:31:50 0:18:30 0:13:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-05-09T01:29:00.357130+0000 mon.a (mon.0) 274 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6824168 2022-05-06 03:20:57 2022-05-09 00:59:12 2022-05-09 01:24:16 0:25:04 0:10:57 0:14:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6824169 2022-05-06 03:20:58 2022-05-09 01:02:13 2022-05-09 01:31:51 0:29:38 0:17:53 0:11:45 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
fail 6824170 2022-05-06 03:20:59 2022-05-09 01:02:23 2022-05-09 01:34:52 0:32:29 0:18:36 0:13:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_fallocate} 3
Failure Reason:

"2022-05-09T01:32:46.411815+0000 mon.a (mon.0) 275 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824171 2022-05-06 03:21:01 2022-05-09 01:05:14 2022-05-09 01:30:32 0:25:18 0:11:43 0:13:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6824172 2022-05-06 03:21:01 2022-05-09 01:05:54 2022-05-09 01:53:50 0:47:56 0:37:39 0:10:17 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6824173 2022-05-06 03:21:03 2022-05-09 01:06:24 2022-05-09 01:33:00 0:26:36 0:14:27 0:12:09 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6824174 2022-05-06 03:21:03 2022-05-09 01:06:55 2022-05-09 01:26:16 0:19:21 0:08:35 0:10:46 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} 1
pass 6824175 2022-05-06 03:21:04 2022-05-09 01:06:55 2022-05-09 01:36:43 0:29:48 0:18:45 0:11:03 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6824176 2022-05-06 03:21:05 2022-05-09 01:06:55 2022-05-09 01:30:41 0:23:46 0:10:47 0:12:59 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_namespaces} 3
pass 6824177 2022-05-06 03:21:06 2022-05-09 01:08:26 2022-05-09 02:15:07 1:06:41 0:55:31 0:11:10 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
fail 6824178 2022-05-06 03:21:08 2022-05-09 01:09:26 2022-05-09 01:40:42 0:31:16 0:18:57 0:12:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} 3
Failure Reason:

"2022-05-09T01:39:08.428631+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824179 2022-05-06 03:21:09 2022-05-09 01:11:07 2022-05-09 01:36:52 0:25:45 0:14:35 0:11:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6824180 2022-05-06 03:21:10 2022-05-09 01:11:17 2022-05-09 01:34:59 0:23:42 0:13:06 0:10:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
fail 6824181 2022-05-06 03:21:11 2022-05-09 01:11:28 2022-05-09 01:41:34 0:30:06 0:18:29 0:11:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-05-09T01:29:16.254503+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824182 2022-05-06 03:21:12 2022-05-09 01:12:08 2022-05-09 01:32:14 0:20:06 0:08:53 0:11:13 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_read_only} 1
pass 6824183 2022-05-06 03:21:13 2022-05-09 01:12:18 2022-05-09 01:44:21 0:32:03 0:17:39 0:14:24 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6824184 2022-05-06 03:21:14 2022-05-09 01:14:19 2022-05-09 02:10:05 0:55:46 0:44:47 0:10:59 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/pggrow thrashosds-health workloads/rbd_fio} 3
pass 6824185 2022-05-06 03:21:15 2022-05-09 01:14:39 2022-05-09 01:39:03 0:24:24 0:10:46 0:13:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6824186 2022-05-06 03:21:16 2022-05-09 01:15:40 2022-05-09 01:38:29 0:22:49 0:10:15 0:12:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6824187 2022-05-06 03:21:17 2022-05-09 01:16:50 2022-05-09 01:50:22 0:33:32 0:18:40 0:14:52 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

"2022-05-09T01:48:15.546166+0000 mon.a (mon.0) 326 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6824188 2022-05-06 03:21:18 2022-05-09 01:17:51 2022-05-09 01:42:06 0:24:15 0:10:14 0:14:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6824189 2022-05-06 03:21:19 2022-05-09 01:18:51 2022-05-09 01:54:07 0:35:16 0:20:11 0:15:05 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
fail 6824190 2022-05-06 03:21:20 2022-05-09 01:22:12 2022-05-09 01:56:12 0:34:00 0:19:22 0:14:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2022-05-09T01:54:25.087879+0000 mon.a (mon.0) 273 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824191 2022-05-06 03:21:21 2022-05-09 01:24:03 2022-05-09 01:45:44 0:21:41 0:09:14 0:12:27 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_whole_object_zeroout} 1
pass 6824192 2022-05-06 03:21:22 2022-05-09 01:24:13 2022-05-09 01:56:33 0:32:20 0:20:17 0:12:03 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6824193 2022-05-06 03:21:23 2022-05-09 01:24:23 2022-05-09 01:46:28 0:22:05 0:10:25 0:11:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_kernel} 3
pass 6824194 2022-05-06 03:21:24 2022-05-09 01:25:14 2022-05-09 01:53:43 0:28:29 0:16:01 0:12:28 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
fail 6824195 2022-05-06 03:21:25 2022-05-09 01:26:24 2022-05-09 01:56:53 0:30:29 0:18:47 0:11:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2022-05-09T01:55:26.493098+0000 mon.a (mon.0) 251 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824196 2022-05-06 03:21:26 2022-05-09 01:27:45 2022-05-09 01:54:47 0:27:02 0:14:46 0:12:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} 3
pass 6824197 2022-05-06 03:21:27 2022-05-09 01:29:25 2022-05-09 01:57:10 0:27:45 0:11:48 0:15:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6824198 2022-05-06 03:21:28 2022-05-09 01:30:36 2022-05-09 02:05:55 0:35:19 0:24:46 0:10:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_simple_big} 3
pass 6824199 2022-05-06 03:21:29 2022-05-09 01:30:36 2022-05-09 01:50:02 0:19:26 0:08:39 0:10:47 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} 1
pass 6824200 2022-05-06 03:21:30 2022-05-09 01:30:37 2022-05-09 02:07:16 0:36:39 0:21:13 0:15:26 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6824201 2022-05-06 03:21:31 2022-05-09 01:30:47 2022-05-09 02:10:58 0:40:11 0:27:09 0:13:02 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_fio} 3
pass 6824202 2022-05-06 03:21:32 2022-05-09 01:30:57 2022-05-09 02:03:09 0:32:12 0:18:55 0:13:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_data_pool} 3
pass 6824203 2022-05-06 03:21:33 2022-05-09 01:31:28 2022-05-09 01:53:10 0:21:42 0:09:04 0:12:38 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} 2
fail 6824204 2022-05-06 03:21:34 2022-05-09 01:31:48 2022-05-09 02:03:34 0:31:46 0:18:46 0:13:00 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} 3
Failure Reason:

"2022-05-09T02:01:09.667640+0000 mon.a (mon.0) 311 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6824205 2022-05-06 03:21:35 2022-05-09 01:31:58 2022-05-09 01:59:09 0:27:11 0:11:55 0:15:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} 3
fail 6824206 2022-05-06 03:21:36 2022-05-09 01:33:09 2022-05-09 02:06:18 0:33:09 0:18:54 0:14:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-05-09T01:53:53.358497+0000 mon.a (mon.0) 226 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824207 2022-05-06 03:21:37 2022-05-09 01:34:30 2022-05-09 01:58:28 0:23:58 0:11:21 0:12:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_fallocate} 3
fail 6824208 2022-05-06 03:21:38 2022-05-09 01:35:00 2022-05-09 02:06:34 0:31:34 0:19:06 0:12:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-05-09T02:03:59.199534+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824209 2022-05-06 03:21:39 2022-05-09 01:35:00 2022-05-09 02:24:08 0:49:08 0:36:28 0:12:40 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6824210 2022-05-06 03:21:40 2022-05-09 01:36:11 2022-05-09 02:02:36 0:26:25 0:13:27 0:12:58 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6824211 2022-05-06 03:21:41 2022-05-09 01:36:51 2022-05-09 01:56:07 0:19:16 0:09:20 0:09:56 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_discard} 1
pass 6824212 2022-05-06 03:21:42 2022-05-09 01:36:51 2022-05-09 02:04:31 0:27:40 0:16:51 0:10:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6824213 2022-05-06 03:21:43 2022-05-09 01:37:02 2022-05-09 02:52:30 1:15:28 1:01:31 0:13:57 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6824214 2022-05-06 03:21:44 2022-05-09 01:38:32 2022-05-09 02:00:44 0:22:12 0:11:37 0:10:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_namespaces} 3
pass 6824215 2022-05-06 03:21:45 2022-05-09 01:39:13 2022-05-09 02:54:13 1:15:00 1:03:49 0:11:11 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6824216 2022-05-06 03:21:46 2022-05-09 01:40:43 2022-05-09 02:31:06 0:50:23 0:38:18 0:12:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_enumerate} 3
fail 6824217 2022-05-06 03:21:48 2022-05-09 01:41:44 2022-05-09 02:11:40 0:29:56 0:18:14 0:11:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2022-05-09T01:59:58.752451+0000 mon.a (mon.0) 162 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824218 2022-05-06 03:21:49 2022-05-09 01:42:14 2022-05-09 02:08:30 0:26:16 0:12:04 0:14:12 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6824219 2022-05-06 03:21:50 2022-05-09 01:44:25 2022-05-09 02:06:36 0:22:11 0:10:36 0:11:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_netns} 3
pass 6824220 2022-05-06 03:21:51 2022-05-09 01:45:25 2022-05-09 02:04:53 0:19:28 0:08:32 0:10:56 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_huge_image} 1
pass 6824221 2022-05-06 03:21:52 2022-05-09 01:45:26 2022-05-09 02:17:22 0:31:56 0:20:37 0:11:19 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6824222 2022-05-06 03:21:53 2022-05-09 01:45:46 2022-05-09 02:08:56 0:23:10 0:12:02 0:11:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_symlinks} 3
fail 6824223 2022-05-06 03:21:54 2022-05-09 01:45:46 2022-05-09 02:15:53 0:30:07 0:18:55 0:11:12 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
Failure Reason:

"2022-05-09T02:14:35.354369+0000 mon.a (mon.0) 267 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824224 2022-05-06 03:21:55 2022-05-09 01:46:37 2022-05-09 02:16:47 0:30:10 0:14:52 0:15:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} 3
fail 6824225 2022-05-06 03:21:56 2022-05-09 01:50:28 2022-05-09 02:25:03 0:34:35 0:19:35 0:15:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2022-05-09T02:23:20.751844+0000 mon.a (mon.0) 246 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824226 2022-05-06 03:21:57 2022-05-09 01:53:18 2022-05-09 02:24:45 0:31:27 0:20:03 0:11:24 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6824227 2022-05-06 03:21:58 2022-05-09 01:53:19 2022-05-09 02:17:14 0:23:55 0:11:32 0:12:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_image_read} 3
pass 6824228 2022-05-06 03:21:59 2022-05-09 01:53:49 2022-05-09 02:13:15 0:19:26 0:08:01 0:11:25 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} 1
pass 6824229 2022-05-06 03:22:00 2022-05-09 01:53:59 2022-05-09 02:27:22 0:33:23 0:21:02 0:12:21 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
fail 6824230 2022-05-06 03:22:01 2022-05-09 01:54:00 2022-05-09 02:25:49 0:31:49 0:19:36 0:12:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-05-09T02:13:03.650365+0000 mon.a (mon.0) 143 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824231 2022-05-06 03:22:02 2022-05-09 01:54:10 2022-05-09 02:24:12 0:30:02 0:18:23 0:11:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6824232 2022-05-06 03:22:03 2022-05-09 01:54:51 2022-05-09 02:17:42 0:22:51 0:10:21 0:12:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_map_snapshot_io} 3
fail 6824233 2022-05-06 03:22:04 2022-05-09 01:56:11 2022-05-09 02:26:30 0:30:19 0:19:00 0:11:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_map_unmap} 3
Failure Reason:

"2022-05-09T02:24:45.395756+0000 mon.a (mon.0) 309 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824234 2022-05-06 03:22:05 2022-05-09 01:56:22 2022-05-09 02:18:52 0:22:30 0:11:33 0:10:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
fail 6824235 2022-05-06 03:22:06 2022-05-09 01:56:42 2022-05-09 02:26:21 0:29:39 0:18:37 0:11:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} 3
Failure Reason:

"2022-05-09T02:24:28.847054+0000 mon.a (mon.0) 251 : cluster [WRN] Health check failed: 3 osds down (OSD_DOWN)" in cluster log

pass 6824236 2022-05-06 03:22:07 2022-05-09 01:57:02 2022-05-09 02:17:43 0:20:41 0:08:50 0:11:51 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_msgr_segments} 1
pass 6824237 2022-05-06 03:22:08 2022-05-09 01:57:13 2022-05-09 02:32:05 0:34:52 0:22:23 0:12:29 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6824238 2022-05-06 03:22:09 2022-05-09 01:58:33 2022-05-09 02:21:21 0:22:48 0:10:51 0:11:57 smithi master krbd/ms_modeless/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_rxbounce} 3
pass 6824239 2022-05-06 03:22:10 2022-05-09 01:59:14 2022-05-09 02:38:31 0:39:17 0:28:02 0:11:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_fio} 3
fail 6824240 2022-05-06 03:22:11 2022-05-09 02:00:54 2022-05-09 02:32:42 0:31:48 0:18:35 0:13:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-05-09T02:20:20.670630+0000 mon.a (mon.0) 131 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6824241 2022-05-06 03:22:12 2022-05-09 02:02:45 2022-05-09 02:33:04 0:30:19 0:18:37 0:11:42 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{secure} msgr-failures/many tasks/rbd_xfstests} 4
Failure Reason:

"2022-05-09T02:31:41.588771+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824242 2022-05-06 03:22:13 2022-05-09 02:03:15 2022-05-09 03:19:06 1:15:51 1:04:40 0:11:11 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6824243 2022-05-06 03:22:14 2022-05-09 02:03:36 2022-05-09 02:26:08 0:22:32 0:10:52 0:11:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_exclusive_option} 3
fail 6824244 2022-05-06 03:22:15 2022-05-09 02:04:36 2022-05-09 02:35:24 0:30:48 0:18:36 0:12:12 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-05-09T02:32:57.504666+0000 mon.a (mon.0) 272 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824245 2022-05-06 03:22:16 2022-05-09 02:05:57 2022-05-09 02:30:08 0:24:11 0:10:45 0:13:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} 3
pass 6824246 2022-05-06 03:22:17 2022-05-09 02:06:27 2022-05-09 02:30:02 0:23:35 0:11:11 0:12:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6824247 2022-05-06 03:22:18 2022-05-09 02:06:37 2022-05-09 02:54:11 0:47:34 0:36:52 0:10:42 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6824248 2022-05-06 03:22:19 2022-05-09 02:06:38 2022-05-09 02:35:51 0:29:13 0:14:23 0:14:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6824249 2022-05-06 03:22:20 2022-05-09 02:07:18 2022-05-09 02:28:41 0:21:23 0:08:20 0:13:03 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_parent_overlap} 1
pass 6824250 2022-05-06 03:22:21 2022-05-09 02:07:18 2022-05-09 02:40:45 0:33:27 0:18:52 0:14:35 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6824251 2022-05-06 03:22:23 2022-05-09 02:08:39 2022-05-09 02:32:19 0:23:40 0:10:36 0:13:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} 3
pass 6824252 2022-05-06 03:22:24 2022-05-09 02:08:59 2022-05-09 03:17:39 1:08:40 0:55:03 0:13:37 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6824253 2022-05-06 03:22:25 2022-05-09 02:10:10 2022-05-09 03:05:09 0:54:59 0:41:38 0:13:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6824254 2022-05-06 03:22:26 2022-05-09 02:11:00 2022-05-09 02:37:08 0:26:08 0:14:33 0:11:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
fail 6824255 2022-05-06 03:22:27 2022-05-09 02:11:51 2022-05-09 02:44:41 0:32:50 0:18:51 0:13:59 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
Failure Reason:

"2022-05-09T02:42:58.400575+0000 mon.a (mon.0) 224 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6824256 2022-05-06 03:22:28 2022-05-09 02:15:11 2022-05-09 02:45:10 0:29:59 0:18:58 0:11:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-05-09T02:32:27.779988+0000 mon.a (mon.0) 143 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824257 2022-05-06 03:22:29 2022-05-09 02:16:02 2022-05-09 02:35:28 0:19:26 0:09:00 0:10:26 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} 1
fail 6824258 2022-05-06 03:22:30 2022-05-09 02:16:02 2022-05-09 02:47:34 0:31:32 0:18:54 0:12:38 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2022-05-09T02:45:28.587870+0000 mon.a (mon.0) 340 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824259 2022-05-06 03:22:31 2022-05-09 02:16:53 2022-05-09 02:39:00 0:22:07 0:10:43 0:11:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6824260 2022-05-06 03:22:32 2022-05-09 02:17:23 2022-05-09 02:36:50 0:19:27 0:10:11 0:09:16 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6824261 2022-05-06 03:22:33 2022-05-09 02:17:23 2022-05-09 05:20:27 3:03:04 2:49:56 0:13:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

Cannot connect to remote host smithi141

pass 6824262 2022-05-06 03:22:34 2022-05-09 02:17:44 2022-05-09 02:41:07 0:23:23 0:10:14 0:13:09 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6824263 2022-05-06 03:22:35 2022-05-09 02:18:54 2022-05-09 02:51:00 0:32:06 0:18:48 0:13:18 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2022-05-09T02:48:32.644711+0000 mon.a (mon.0) 328 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6824264 2022-05-06 03:22:36 2022-05-09 02:21:25 2022-05-09 02:51:30 0:30:05 0:18:24 0:11:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2022-05-09T02:38:31.124240+0000 mon.a (mon.0) 119 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824265 2022-05-06 03:22:37 2022-05-09 02:21:25 2022-05-09 02:40:46 0:19:21 0:09:00 0:10:21 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_whole_object_zeroout} 1
pass 6824266 2022-05-06 03:22:38 2022-05-09 02:21:36 2022-05-09 02:49:37 0:28:01 0:14:30 0:13:31 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6824267 2022-05-06 03:22:39 2022-05-09 02:24:16 2022-05-09 03:12:26 0:48:10 0:36:20 0:11:50 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} 3
pass 6824268 2022-05-06 03:22:40 2022-05-09 02:24:17 2022-05-09 02:46:14 0:21:57 0:09:56 0:12:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_kernel} 3
pass 6824269 2022-05-06 03:22:41 2022-05-09 02:24:47 2022-05-09 02:50:12 0:25:25 0:15:29 0:09:56 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
fail 6824270 2022-05-06 03:22:42 2022-05-09 02:25:07 2022-05-09 02:55:33 0:30:26 0:18:55 0:11:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2022-05-09T02:54:06.894313+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824271 2022-05-06 03:22:43 2022-05-09 02:25:58 2022-05-09 02:51:51 0:25:53 0:14:36 0:11:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_map_unmap} 3
pass 6824272 2022-05-06 03:22:44 2022-05-09 02:26:18 2022-05-09 02:49:41 0:23:23 0:11:44 0:11:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6824273 2022-05-06 03:22:45 2022-05-09 02:26:29 2022-05-09 03:02:47 0:36:18 0:23:36 0:12:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_simple_big} 3
pass 6824274 2022-05-06 03:22:46 2022-05-09 02:26:39 2022-05-09 02:46:49 0:20:10 0:08:42 0:11:28 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_deep_flatten} 1
pass 6824275 2022-05-06 03:22:48 2022-05-09 02:27:29 2022-05-09 03:04:17 0:36:48 0:24:35 0:12:13 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6824276 2022-05-06 03:22:49 2022-05-09 02:28:50 2022-05-09 03:08:08 0:39:18 0:27:14 0:12:04 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} 3
pass 6824277 2022-05-06 03:22:50 2022-05-09 02:30:10 2022-05-09 02:59:34 0:29:24 0:18:43 0:10:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} 3
pass 6824278 2022-05-06 03:22:51 2022-05-09 02:30:11 2022-05-09 02:50:36 0:20:25 0:08:39 0:11:46 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} 2
pass 6824279 2022-05-06 03:22:52 2022-05-09 02:31:11 2022-05-09 03:03:43 0:32:32 0:20:22 0:12:10 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} 3
fail 6824280 2022-05-06 03:22:53 2022-05-09 02:32:12 2022-05-09 03:01:39 0:29:27 0:18:50 0:10:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2022-05-09T02:59:12.514263+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6824281 2022-05-06 03:22:54 2022-05-09 02:32:22 2022-05-09 03:02:38 0:30:16 0:18:30 0:11:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-05-09T03:00:25.779430+0000 mon.a (mon.0) 224 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824282 2022-05-06 03:22:55 2022-05-09 02:32:53 2022-05-09 02:54:36 0:21:43 0:10:31 0:11:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_fallocate} 3
pass 6824283 2022-05-06 03:22:56 2022-05-09 02:33:13 2022-05-09 02:57:00 0:23:47 0:11:09 0:12:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6824284 2022-05-06 03:22:57 2022-05-09 02:35:34 2022-05-09 03:22:59 0:47:25 0:36:55 0:10:30 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6824285 2022-05-06 03:22:58 2022-05-09 02:35:34 2022-05-09 03:00:22 0:24:48 0:12:53 0:11:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6824286 2022-05-06 03:22:59 2022-05-09 02:35:54 2022-05-09 02:56:15 0:20:21 0:08:59 0:11:22 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} 1
pass 6824287 2022-05-06 03:23:00 2022-05-09 02:36:55 2022-05-09 03:02:36 0:25:41 0:14:40 0:11:01 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
fail 6824288 2022-05-06 03:23:01 2022-05-09 02:37:15 2022-05-09 03:08:11 0:30:56 0:18:33 0:12:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2022-05-09T03:05:29.405570+0000 mon.a (mon.0) 262 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824289 2022-05-06 03:23:02 2022-05-09 02:38:36 2022-05-09 03:54:45 1:16:09 1:04:40 0:11:29 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6824290 2022-05-06 03:23:04 2022-05-09 02:39:06 2022-05-09 03:30:50 0:51:44 0:38:24 0:13:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6824291 2022-05-06 03:23:05 2022-05-09 02:40:47 2022-05-09 03:06:11 0:25:24 0:14:23 0:11:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6824292 2022-05-06 03:23:06 2022-05-09 02:40:47 2022-05-09 03:03:16 0:22:29 0:11:27 0:11:02 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6824293 2022-05-06 03:23:07 2022-05-06 03:35:49 2022-05-06 03:57:14 0:21:25 0:10:35 0:10:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_netns} 3
pass 6824294 2022-05-06 03:23:08 2022-05-09 02:41:07 2022-05-09 03:02:13 0:21:06 0:08:10 0:12:56 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_huge_image} 1
pass 6824295 2022-05-06 03:23:09 2022-05-09 02:44:48 2022-05-09 03:14:28 0:29:40 0:20:01 0:09:39 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6824296 2022-05-06 03:23:10 2022-05-09 02:45:18 2022-05-09 03:55:47 1:10:29 0:59:48 0:10:41 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6824297 2022-05-06 03:23:11 2022-05-09 02:46:19 2022-05-09 03:16:19 0:30:00 0:18:30 0:11:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} 3
Failure Reason:

"2022-05-09T03:13:40.749415+0000 mon.a (mon.0) 237 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 6824298 2022-05-06 03:23:12 2022-05-09 02:46:59 2022-05-09 03:18:02 0:31:03 0:18:36 0:12:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
Failure Reason:

"2022-05-09T03:05:32.947919+0000 mon.a (mon.0) 184 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6824299 2022-05-06 03:23:13 2022-05-09 02:47:40 2022-05-09 03:15:05 0:27:25 0:15:06 0:12:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_concurrent} 3
fail 6824300 2022-05-06 03:23:14 2022-05-09 02:49:40 2022-05-09 03:19:11 0:29:31 0:18:42 0:10:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2022-05-09T03:16:45.143695+0000 mon.a (mon.0) 231 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824301 2022-05-06 03:23:15 2022-05-09 02:49:51 2022-05-09 03:19:40 0:29:49 0:18:48 0:11:01 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6824302 2022-05-06 03:23:16 2022-05-09 02:50:21 2022-05-09 03:10:31 0:20:10 0:10:09 0:10:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} 3
pass 6824303 2022-05-06 03:23:17 2022-05-09 02:51:02 2022-05-09 03:08:30 0:17:28 0:08:08 0:09:20 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_modprobe} 1
pass 6824304 2022-05-06 03:23:18 2022-05-09 02:51:02 2022-05-09 03:21:35 0:30:33 0:19:01 0:11:32 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6824305 2022-05-06 03:23:19 2022-05-09 02:51:32 2022-05-09 03:13:28 0:21:56 0:10:16 0:11:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_kernel} 3
pass 6824306 2022-05-06 03:23:20 2022-05-09 02:51:53 2022-05-09 03:20:21 0:28:28 0:16:48 0:11:40 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6824307 2022-05-06 03:23:21 2022-05-09 02:52:33 2022-05-09 03:13:51 0:21:18 0:09:32 0:11:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_snapshot_io} 3
fail 6824308 2022-05-06 03:23:22 2022-05-09 02:54:14 2022-05-09 03:01:54 0:07:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} 3
Failure Reason:

Command failed on smithi123 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main'

pass 6824309 2022-05-06 03:23:24 2022-05-09 02:54:14 2022-05-09 03:16:22 0:22:08 0:11:15 0:10:53 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6824310 2022-05-06 03:23:25 2022-05-09 02:54:44 2022-05-09 03:29:05 0:34:21 0:22:23 0:11:58 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_simple_big} 3
pass 6824312 2022-05-06 03:23:26 2022-05-09 02:55:35 2022-05-09 03:12:31 0:16:56 0:08:30 0:08:26 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} 1
pass 6824314 2022-05-06 03:23:27 2022-05-09 02:55:35 2022-05-09 03:32:40 0:37:05 0:25:40 0:11:25 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
fail 6824316 2022-05-06 03:23:28 2022-05-09 02:57:06 2022-05-09 03:29:03 0:31:57 0:19:00 0:12:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

"2022-05-09T03:26:29.812195+0000 mon.a (mon.0) 235 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6824317 2022-05-06 03:23:29 2022-05-09 02:59:36 2022-05-09 03:30:55 0:31:19 0:18:29 0:12:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-05-09T03:29:09.370072+0000 mon.a (mon.0) 302 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6824318 2022-05-06 03:23:30 2022-05-09 03:00:27 2022-05-09 03:22:54 0:22:27 0:10:41 0:11:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6824319 2022-05-06 03:23:31 2022-05-09 03:01:47 2022-05-09 03:31:28 0:29:41 0:17:44 0:11:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6824320 2022-05-06 03:23:32 2022-05-09 03:01:58 2022-05-09 03:26:02 0:24:04 0:11:18 0:12:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_fallocate} 3
pass 6824321 2022-05-06 03:23:33 2022-05-09 03:02:38 2022-05-09 03:27:08 0:24:30 0:11:10 0:13:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6824322 2022-05-06 03:23:34 2022-05-09 03:02:48 2022-05-09 03:52:27 0:49:39 0:37:48 0:11:51 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6824323 2022-05-06 03:23:35 2022-05-09 03:02:49 2022-05-09 03:29:32 0:26:43 0:14:40 0:12:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6824324 2022-05-06 03:23:36 2022-05-09 03:03:19 2022-05-09 03:22:50 0:19:31 0:08:30 0:11:01 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_parent_overlap} 1
pass 6824325 2022-05-06 03:23:37 2022-05-09 03:03:19 2022-05-09 03:31:21 0:28:02 0:16:09 0:11:53 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6824326 2022-05-06 03:23:38 2022-05-09 03:03:50 2022-05-09 03:55:49 0:51:59 0:40:47 0:11:12 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/upmap thrashosds-health workloads/rbd_fio} 3
pass 6824327 2022-05-06 03:23:39 2022-05-09 03:04:20 2022-05-09 03:27:21 0:23:01 0:10:37 0:12:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_namespaces} 3
pass 6824328 2022-05-06 03:23:40 2022-05-09 03:05:11 2022-05-09 04:11:40 1:06:29 0:54:14 0:12:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6824329 2022-05-06 03:23:41 2022-05-09 03:06:11 2022-05-09 04:02:14 0:56:03 0:42:01 0:14:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6824330 2022-05-06 03:23:42 2022-05-09 03:08:12 2022-05-09 03:33:45 0:25:33 0:14:40 0:10:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6824331 2022-05-06 03:23:43 2022-05-09 03:08:22 2022-05-09 03:34:09 0:25:47 0:13:01 0:12:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
fail 6824332 2022-05-06 03:23:44 2022-05-09 03:10:33 2022-05-09 03:41:38 0:31:05 0:18:30 0:12:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-05-09T03:40:09.861148+0000 mon.a (mon.0) 227 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6824333 2022-05-06 03:23:45 2022-05-09 03:12:33 2022-05-09 03:34:34 0:22:01 0:09:09 0:12:52 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_read_only} 1
pass 6824334 2022-05-06 03:23:46 2022-05-09 03:12:34 2022-05-09 03:47:12 0:34:38 0:21:37 0:13:01 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6824335 2022-05-06 03:23:47 2022-05-09 03:13:34 2022-05-09 03:37:36 0:24:02 0:10:58 0:13:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6824336 2022-05-06 03:23:48 2022-05-09 03:13:55 2022-05-09 03:35:52 0:21:57 0:10:02 0:11:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6824337 2022-05-06 03:23:49 2022-05-09 03:14:35 2022-05-09 03:42:27 0:27:52 0:14:57 0:12:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} 3
pass 6824338 2022-05-06 03:23:50 2022-05-09 03:15:05 2022-05-09 03:37:57 0:22:52 0:10:06 0:12:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6824339 2022-05-06 03:23:51 2022-05-09 03:16:26 2022-05-09 03:49:46 0:33:20 0:20:25 0:12:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
pass 6824340 2022-05-06 03:23:52 2022-05-09 03:16:26 2022-05-09 03:39:14 0:22:48 0:10:29 0:12:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_image_read} 3
pass 6824341 2022-05-06 03:23:53 2022-05-09 03:17:47 2022-05-09 03:39:18 0:21:31 0:09:15 0:12:16 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} 1
pass 6824342 2022-05-06 03:23:54 2022-05-09 03:17:47 2022-05-09 03:50:28 0:32:41 0:18:34 0:14:07 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6824343 2022-05-06 03:23:55 2022-05-09 03:18:08 2022-05-09 03:40:44 0:22:36 0:10:20 0:12:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} 3
pass 6824344 2022-05-06 03:23:57 2022-05-09 03:19:08 2022-05-09 03:48:31 0:29:23 0:16:02 0:13:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6824345 2022-05-06 03:23:58 2022-05-09 03:19:18 2022-05-09 03:41:00 0:21:42 0:10:04 0:11:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6824346 2022-05-06 03:23:59 2022-05-09 03:19:49 2022-05-09 03:48:20 0:28:31 0:15:28 0:13:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_map_unmap} 3
pass 6824347 2022-05-06 03:24:00 2022-05-09 03:20:29 2022-05-09 03:45:40 0:25:11 0:11:34 0:13:37 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6824348 2022-05-06 03:24:01 2022-05-09 03:21:40 2022-05-09 04:02:15 0:40:35 0:27:19 0:13:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} 3