Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 6771416 2022-04-01 03:20:36 2022-04-04 02:24:30 2022-04-04 02:42:27 0:17:57 0:07:45 0:10:12 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_deep_flatten} 1
pass 6771417 2022-04-01 03:20:37 2022-04-04 02:25:11 2022-04-04 02:53:58 0:28:47 0:17:33 0:11:14 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6771418 2022-04-01 03:20:38 2022-04-04 02:26:41 2022-04-04 02:44:47 0:18:06 0:08:31 0:09:35 smithi master krbd/ms_modeless/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_default_map_options} 3
pass 6771419 2022-04-01 03:20:39 2022-04-04 02:27:22 2022-04-04 03:03:11 0:35:49 0:26:29 0:09:20 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_fio} 3
pass 6771420 2022-04-01 03:20:39 2022-04-04 02:27:42 2022-04-04 02:58:04 0:30:22 0:17:04 0:13:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_data_pool} 3
fail 6771421 2022-04-01 03:20:40 2022-04-04 02:30:33 2022-04-04 07:22:09 4:51:36 4:39:10 0:12:26 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy-rxbounce} msgr-failures/few tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi101 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude1o9it2l5 -r -- '-g auto -g blockdev -x clone'"

pass 6771422 2022-04-01 03:20:41 2022-04-04 02:32:23 2022-04-04 03:24:35 0:52:12 0:41:24 0:10:48 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/backoff thrashosds-health workloads/rbd_fio} 3
fail 6771423 2022-04-01 03:20:42 2022-04-04 02:32:34 2022-04-04 02:41:17 0:08:43 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major

pass 6771424 2022-04-01 03:20:43 2022-04-04 02:33:44 2022-04-04 02:51:48 0:18:04 0:07:13 0:10:51 smithi master krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} 1
pass 6771425 2022-04-01 03:20:44 2022-04-04 02:34:34 2022-04-04 02:55:10 0:20:36 0:10:03 0:10:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_exclusive_option} 3
pass 6771426 2022-04-01 03:20:44 2022-04-04 02:35:45 2022-04-04 03:08:39 0:32:54 0:18:07 0:14:47 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6771427 2022-04-01 03:20:45 2022-04-04 02:38:26 2022-04-04 02:59:01 0:20:35 0:10:02 0:10:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} 3
fail 6771428 2022-04-01 03:20:46 2022-04-04 02:39:26 2022-04-04 03:10:00 0:30:34 0:17:47 0:12:47 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-04-04T03:08:35.281523+0000 mon.a (mon.0) 238 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771429 2022-04-01 03:20:47 2022-04-04 02:42:37 2022-04-04 03:30:22 0:47:45 0:35:25 0:12:20 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6771430 2022-04-01 03:20:48 2022-04-04 02:44:57 2022-04-04 03:13:16 0:28:19 0:12:09 0:16:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6771431 2022-04-01 03:20:48 2022-04-04 02:51:59 2022-04-04 03:11:11 0:19:12 0:07:52 0:11:20 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_discard} 1
pass 6771432 2022-04-01 03:20:49 2022-04-04 02:53:59 2022-04-04 03:20:44 0:26:45 0:15:37 0:11:08 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6771433 2022-04-01 03:20:50 2022-04-04 02:55:20 2022-04-04 03:17:09 0:21:49 0:10:10 0:11:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} 3
fail 6771434 2022-04-01 03:20:51 2022-04-04 02:55:50 2022-04-04 03:25:46 0:29:56 0:17:41 0:12:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2022-04-04T03:13:57.614381+0000 mon.a (mon.0) 137 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6771435 2022-04-01 03:20:52 2022-04-04 02:58:11 2022-04-04 03:44:42 0:46:31 0:36:00 0:10:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_enumerate} 3
fail 6771436 2022-04-01 03:20:52 2022-04-04 02:59:11 2022-04-04 03:30:37 0:31:26 0:17:41 0:13:45 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2022-04-04T03:28:54.935911+0000 mon.a (mon.0) 259 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771437 2022-04-01 03:20:53 2022-04-04 03:03:12 2022-04-04 03:24:35 0:21:23 0:10:23 0:11:00 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6771438 2022-04-01 03:20:54 2022-04-04 03:05:02 2022-04-04 03:29:01 0:23:59 0:09:39 0:14:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} 3
pass 6771439 2022-04-01 03:20:55 2022-04-04 03:08:43 2022-04-04 03:25:48 0:17:05 0:07:21 0:09:44 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} 1
fail 6771440 2022-04-01 03:20:55 2022-04-04 03:08:43 2022-04-04 03:37:23 0:28:40 0:17:42 0:10:58 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2022-04-04T03:25:20.568346+0000 mon.a (mon.0) 183 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6771441 2022-04-01 03:20:56 2022-04-04 03:10:04 2022-04-04 03:34:05 0:24:01 0:10:40 0:13:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6771442 2022-04-01 03:20:57 2022-04-04 03:12:25 2022-04-04 03:32:50 0:20:25 0:09:34 0:10:51 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6771443 2022-04-01 03:20:58 2022-04-04 03:13:25 2022-04-04 03:40:38 0:27:13 0:13:45 0:13:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_concurrent} 3
fail 6771444 2022-04-01 03:20:59 2022-04-04 03:17:16 2022-04-04 03:48:12 0:30:56 0:17:41 0:13:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2022-04-04T03:35:59.061459+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6771445 2022-04-01 03:20:59 2022-04-04 03:20:47 2022-04-04 03:52:39 0:31:52 0:17:00 0:14:52 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6771446 2022-04-01 03:21:00 2022-04-04 03:24:37 2022-04-04 03:44:03 0:19:26 0:09:03 0:10:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_image_read} 3
pass 6771447 2022-04-01 03:21:01 2022-04-04 03:24:38 2022-04-04 03:42:53 0:18:15 0:07:17 0:10:58 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_modprobe} 1
pass 6771448 2022-04-01 03:21:02 2022-04-04 03:25:48 2022-04-04 03:49:20 0:23:32 0:13:30 0:10:02 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6771449 2022-04-01 03:21:02 2022-04-04 03:25:49 2022-04-04 04:53:23 1:27:34 1:13:52 0:13:42 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6771450 2022-04-01 03:21:03 2022-04-04 03:29:09 2022-04-04 03:57:54 0:28:45 0:17:45 0:11:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-04-04T03:56:18.347862+0000 mon.a (mon.0) 217 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6771451 2022-04-01 03:21:04 2022-04-04 03:30:30 2022-04-04 03:57:56 0:27:26 0:17:59 0:09:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2022-04-04T03:56:32.837375+0000 mon.a (mon.0) 327 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771452 2022-04-01 03:21:05 2022-04-04 03:30:40 2022-04-04 03:52:16 0:21:36 0:08:44 0:12:52 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6771453 2022-04-01 03:21:06 2022-04-04 03:32:51 2022-04-04 03:56:50 0:23:59 0:13:53 0:10:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_map_unmap} 3
pass 6771454 2022-04-01 03:21:06 2022-04-04 03:33:01 2022-04-04 03:56:01 0:23:00 0:10:01 0:12:59 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6771455 2022-04-01 03:21:07 2022-04-04 03:34:12 2022-04-04 04:08:50 0:34:38 0:21:13 0:13:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_simple_big} 3
pass 6771456 2022-04-01 03:21:08 2022-04-04 03:37:32 2022-04-04 03:54:51 0:17:19 0:07:46 0:09:33 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_msgr_segments} 1
pass 6771457 2022-04-01 03:21:09 2022-04-04 03:37:33 2022-04-04 04:12:03 0:34:30 0:20:55 0:13:35 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6771458 2022-04-01 03:21:10 2022-04-04 03:40:43 2022-04-04 04:20:00 0:39:17 0:26:28 0:12:49 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} 3
fail 6771459 2022-04-01 03:21:10 2022-04-04 03:42:54 2022-04-04 04:11:33 0:28:39 0:17:37 0:11:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-04-04T04:09:46.649825+0000 mon.a (mon.0) 231 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771460 2022-04-01 03:21:11 2022-04-04 03:44:04 2022-04-04 04:04:16 0:20:12 0:10:00 0:10:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6771461 2022-04-01 03:21:12 2022-04-04 03:44:45 2022-04-04 04:15:57 0:31:12 0:16:53 0:14:19 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
fail 6771462 2022-04-01 03:21:13 2022-04-04 03:48:16 2022-04-04 04:19:13 0:30:57 0:17:54 0:13:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_fallocate} 3
Failure Reason:

"2022-04-04T04:16:02.531281+0000 mon.a (mon.0) 317 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771463 2022-04-01 03:21:13 2022-04-04 03:49:26 2022-04-04 04:11:36 0:22:10 0:09:57 0:12:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} 3
fail 6771464 2022-04-01 03:21:14 2022-04-04 03:52:17 2022-04-04 04:20:43 0:28:26 0:17:43 0:10:43 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
Failure Reason:

"2022-04-04T04:09:03.805223+0000 mon.a (mon.0) 165 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6771465 2022-04-01 03:21:15 2022-04-04 03:52:47 2022-04-04 04:18:52 0:26:05 0:12:39 0:13:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6771466 2022-04-01 03:21:16 2022-04-04 03:55:18 2022-04-04 04:12:38 0:17:20 0:07:22 0:09:58 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} 1
pass 6771467 2022-04-01 03:21:17 2022-04-04 03:55:18 2022-04-04 04:23:55 0:28:37 0:16:01 0:12:36 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6771468 2022-04-01 03:21:17 2022-04-04 03:56:09 2022-04-04 04:18:56 0:22:47 0:10:26 0:12:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_namespaces} 3
pass 6771469 2022-04-01 03:21:18 2022-04-04 03:56:59 2022-04-04 05:19:22 1:22:23 1:12:02 0:10:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6771470 2022-04-01 03:21:19 2022-04-04 03:57:59 2022-04-04 04:47:10 0:49:11 0:39:20 0:09:51 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6771471 2022-04-01 03:21:20 2022-04-04 03:58:00 2022-04-04 04:27:50 0:29:50 0:13:35 0:16:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
fail 6771472 2022-04-01 03:21:21 2022-04-04 04:04:21 2022-04-04 04:36:26 0:32:05 0:17:32 0:14:33 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
Failure Reason:

"2022-04-04T04:35:01.835008+0000 mon.a (mon.0) 231 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6771473 2022-04-01 03:21:21 2022-04-04 04:08:52 2022-04-04 04:39:01 0:30:09 0:17:58 0:12:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-04-04T04:37:52.808906+0000 mon.a (mon.0) 286 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771474 2022-04-01 03:21:22 2022-04-04 04:11:43 2022-04-04 04:28:56 0:17:13 0:07:58 0:09:15 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_read_only} 1
pass 6771475 2022-04-01 03:21:23 2022-04-04 04:11:43 2022-04-04 04:38:58 0:27:15 0:16:57 0:10:18 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6771476 2022-04-01 03:21:24 2022-04-04 04:11:43 2022-04-04 04:59:34 0:47:51 0:38:12 0:09:39 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/pggrow thrashosds-health workloads/rbd_fio} 3
pass 6771477 2022-04-01 03:21:24 2022-04-04 04:12:14 2022-04-04 04:35:29 0:23:15 0:10:02 0:13:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6771478 2022-04-01 03:21:25 2022-04-04 04:16:04 2022-04-04 04:36:57 0:20:53 0:09:03 0:11:50 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6771479 2022-04-01 03:21:26 2022-04-04 04:17:35 2022-04-04 04:48:38 0:31:03 0:18:00 0:13:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

"2022-04-04T04:45:42.850872+0000 mon.a (mon.0) 306 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771480 2022-04-01 03:21:27 2022-04-04 04:18:55 2022-04-04 04:39:12 0:20:17 0:09:14 0:11:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6771481 2022-04-01 03:21:28 2022-04-04 04:19:06 2022-04-04 04:48:38 0:29:32 0:19:18 0:10:14 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
fail 6771482 2022-04-01 03:21:28 2022-04-04 04:19:16 2022-04-04 04:48:54 0:29:38 0:18:07 0:11:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2022-04-04T04:46:15.315607+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771483 2022-04-01 03:21:29 2022-04-04 04:19:16 2022-04-04 04:39:03 0:19:47 0:08:13 0:11:34 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_whole_object_zeroout} 1
pass 6771484 2022-04-01 03:21:30 2022-04-04 04:20:07 2022-04-04 04:48:34 0:28:27 0:17:44 0:10:43 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6771485 2022-04-01 03:21:31 2022-04-04 04:20:47 2022-04-04 04:43:50 0:23:03 0:08:57 0:14:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_kernel} 3
pass 6771486 2022-04-01 03:21:32 2022-04-04 04:23:58 2022-04-04 04:51:28 0:27:30 0:14:01 0:13:29 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6771487 2022-04-01 03:21:33 2022-04-04 04:27:59 2022-04-04 04:48:30 0:20:31 0:08:46 0:11:45 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6771488 2022-04-01 03:21:33 2022-04-04 04:28:59 2022-04-04 04:58:53 0:29:54 0:13:49 0:16:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} 3
pass 6771489 2022-04-01 03:21:34 2022-04-04 04:35:31 2022-04-04 04:57:50 0:22:19 0:10:26 0:11:53 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6771490 2022-04-01 03:21:35 2022-04-04 04:36:31 2022-04-04 05:10:31 0:34:00 0:23:12 0:10:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_simple_big} 3
pass 6771491 2022-04-01 03:21:36 2022-04-04 04:37:01 2022-04-04 04:56:13 0:19:12 0:08:01 0:11:11 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} 1
pass 6771492 2022-04-01 03:21:37 2022-04-04 04:39:02 2022-04-04 05:08:24 0:29:22 0:19:50 0:09:32 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6771493 2022-04-01 03:21:37 2022-04-04 04:39:02 2022-04-04 05:17:15 0:38:13 0:26:52 0:11:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_fio} 3
pass 6771494 2022-04-01 03:21:38 2022-04-04 04:39:13 2022-04-04 05:06:27 0:27:14 0:17:30 0:09:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_data_pool} 3
pass 6771495 2022-04-01 03:21:39 2022-04-04 04:39:13 2022-04-04 05:01:39 0:22:26 0:07:36 0:14:50 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} 2
fail 6771496 2022-04-01 03:21:40 2022-04-04 04:43:54 2022-04-04 05:16:37 0:32:43 0:19:36 0:13:07 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} 3
Failure Reason:

"2022-04-04T05:04:13.200501+0000 osd.1 (osd.1) 3 : cluster [WRN] 2 slow requests (by type [ 'waiting for sub ops' : 2 ] most affected pool [ 'rbd' : 2 ])" in cluster log

fail 6771497 2022-04-01 03:21:40 2022-04-04 04:47:15 2022-04-04 05:18:32 0:31:17 0:17:51 0:13:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2022-04-04T05:16:02.245361+0000 mon.a (mon.0) 289 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771498 2022-04-01 03:21:41 2022-04-04 04:48:35 2022-04-04 05:18:05 0:29:30 0:18:20 0:11:10 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6771499 2022-04-01 03:21:42 2022-04-04 04:48:35 2022-04-04 05:10:12 0:21:37 0:10:18 0:11:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_fallocate} 3
fail 6771500 2022-04-01 03:21:43 2022-04-04 04:48:46 2022-04-04 05:18:08 0:29:22 0:18:06 0:11:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-04-04T05:15:35.956627+0000 mon.a (mon.0) 224 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771501 2022-04-01 03:21:44 2022-04-04 04:48:46 2022-04-04 05:36:38 0:47:52 0:36:06 0:11:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6771502 2022-04-01 03:21:45 2022-04-04 04:48:57 2022-04-04 05:14:14 0:25:17 0:13:00 0:12:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6771503 2022-04-01 03:21:45 2022-04-04 04:50:47 2022-04-04 05:07:57 0:17:10 0:08:02 0:09:08 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_discard} 1
pass 6771504 2022-04-01 03:21:46 2022-04-04 04:50:47 2022-04-04 05:18:43 0:27:56 0:16:48 0:11:08 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6771505 2022-04-01 03:21:47 2022-04-04 04:50:48 2022-04-04 06:20:39 1:29:51 1:18:27 0:11:24 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6771506 2022-04-01 03:21:48 2022-04-04 04:50:48 2022-04-04 05:22:27 0:31:39 0:19:13 0:12:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2022-04-04T05:19:39.883020+0000 mon.a (mon.0) 266 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6771507 2022-04-01 03:21:49 2022-04-04 04:50:48 2022-04-04 05:22:22 0:31:34 0:18:55 0:12:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2022-04-04T05:19:20.224584+0000 mon.a (mon.0) 364 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6771508 2022-04-01 03:21:49 2022-04-04 04:50:49 2022-04-04 05:41:00 0:50:11 0:37:18 0:12:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_enumerate} 3
fail 6771509 2022-04-01 03:21:50 2022-04-04 04:50:49 2022-04-04 05:20:09 0:29:20 0:17:51 0:11:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2022-04-04T05:17:37.023422+0000 mon.a (mon.0) 248 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771510 2022-04-01 03:21:51 2022-04-04 04:50:50 2022-04-04 05:15:03 0:24:13 0:11:35 0:12:38 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
fail 6771511 2022-04-01 03:21:52 2022-04-04 04:51:30 2022-04-04 05:23:44 0:32:14 0:17:59 0:14:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_netns} 3
Failure Reason:

"2022-04-04T05:20:31.180661+0000 mon.a (mon.0) 216 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771512 2022-04-01 03:21:53 2022-04-04 04:53:31 2022-04-04 05:12:48 0:19:17 0:08:52 0:10:25 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_huge_image} 1
pass 6771513 2022-04-01 03:21:53 2022-04-04 04:53:31 2022-04-04 05:26:03 0:32:32 0:19:33 0:12:59 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6771514 2022-04-01 03:21:54 2022-04-04 04:56:52 2022-04-04 05:18:12 0:21:20 0:10:44 0:10:36 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6771515 2022-04-01 03:21:55 2022-04-04 04:56:52 2022-04-04 05:18:10 0:21:18 0:09:44 0:11:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6771516 2022-04-01 03:21:56 2022-04-04 04:56:52 2022-04-04 05:23:22 0:26:30 0:14:01 0:12:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} 3
fail 6771517 2022-04-01 03:21:57 2022-04-04 04:59:04 2022-04-04 05:29:01 0:29:57 0:17:40 0:12:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2022-04-04T05:16:02.388248+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6771518 2022-04-01 03:21:58 2022-04-04 04:59:44 2022-04-04 05:28:14 0:28:30 0:17:04 0:11:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
fail 6771519 2022-04-01 03:21:59 2022-04-04 05:01:04 2022-04-04 05:33:05 0:32:01 0:18:02 0:13:59 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_image_read} 3
Failure Reason:

"2022-04-04T05:19:27.690382+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6771520 2022-04-01 03:21:59 2022-04-04 05:01:05 2022-04-04 05:19:54 0:18:49 0:07:40 0:11:09 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} 1
pass 6771521 2022-04-01 03:22:00 2022-04-04 05:01:05 2022-04-04 05:33:11 0:32:06 0:18:39 0:13:27 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6771522 2022-04-01 03:22:01 2022-04-04 05:01:06 2022-04-04 05:22:30 0:21:24 0:09:30 0:11:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} 3
fail 6771523 2022-04-01 03:22:02 2022-04-04 05:01:06 2022-04-04 05:10:37 0:09:31 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

Command failed on smithi117 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main'

dead 6771524 2022-04-01 03:22:03 2022-04-04 05:01:06 2022-04-04 05:10:56 0:09:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_map_snapshot_io} 3
Failure Reason:

SSH connection to smithi117 was lost: 'sudo dpkg -i /tmp/linux-image.deb'

pass 6771525 2022-04-01 03:22:04 2022-04-04 05:01:07 2022-04-04 05:28:59 0:27:52 0:14:38 0:13:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_unmap} 3
pass 6771526 2022-04-01 03:22:04 2022-04-04 05:01:07 2022-04-04 05:25:38 0:24:31 0:10:21 0:14:10 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6771527 2022-04-01 03:22:05 2022-04-04 05:01:48 2022-04-04 05:39:48 0:38:00 0:23:20 0:14:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} 3
pass 6771528 2022-04-01 03:22:06 2022-04-04 05:06:28 2022-04-04 05:25:13 0:18:45 0:07:35 0:11:10 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_msgr_segments} 1
pass 6771529 2022-04-01 03:22:07 2022-04-04 05:07:59 2022-04-04 05:39:53 0:31:54 0:21:49 0:10:05 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6771530 2022-04-01 03:22:08 2022-04-04 05:08:29 2022-04-04 05:31:55 0:23:26 0:10:10 0:13:16 smithi master krbd/ms_modeless/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_rxbounce} 3
dead 6771531 2022-04-01 03:22:09 2022-04-04 05:10:20 2022-04-04 05:18:17 0:07:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

SSH connection to smithi117 was lost: 'sudo dpkg -i /tmp/linux-image.deb'

fail 6771532 2022-04-01 03:22:09 2022-04-04 05:10:40 2022-04-04 05:40:10 0:29:30 0:18:03 0:11:27 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-04-04T05:27:42.692401+0000 mon.a (mon.0) 162 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6771533 2022-04-01 03:22:10 2022-04-04 05:10:41 2022-04-04 05:19:02 0:08:21 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{crc-rxbounce} msgr-failures/many tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi117 with status 2: 'sudo dpkg -i /tmp/linux-image.deb'

pass 6771534 2022-04-01 03:22:11 2022-04-04 05:11:11 2022-04-04 06:47:51 1:36:40 1:24:33 0:12:07 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6771535 2022-04-01 03:22:12 2022-04-04 05:12:32 2022-04-04 05:35:50 0:23:18 0:10:17 0:13:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_exclusive_option} 3
fail 6771536 2022-04-01 03:22:13 2022-04-04 05:14:22 2022-04-04 05:42:51 0:28:29 0:17:43 0:10:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-04-04T05:41:22.273469+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6771537 2022-04-01 03:22:13 2022-04-04 05:15:13 2022-04-04 05:43:52 0:28:39 0:17:59 0:10:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} 3
Failure Reason:

"2022-04-04T05:42:32.031357+0000 mon.a (mon.0) 279 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771538 2022-04-01 03:22:14 2022-04-04 05:16:43 2022-04-04 05:39:23 0:22:40 0:10:11 0:12:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6771539 2022-04-01 03:22:15 2022-04-04 05:17:24 2022-04-04 06:07:38 0:50:14 0:36:29 0:13:45 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6771540 2022-04-01 03:22:16 2022-04-04 05:18:14 2022-04-04 05:43:43 0:25:29 0:13:11 0:12:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6771541 2022-04-01 03:22:17 2022-04-04 05:18:15 2022-04-04 05:37:21 0:19:06 0:07:39 0:11:27 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_parent_overlap} 1
pass 6771542 2022-04-01 03:22:18 2022-04-04 05:18:15 2022-04-04 05:47:45 0:29:30 0:16:44 0:12:46 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6771543 2022-04-01 03:22:18 2022-04-04 05:18:15 2022-04-04 05:41:46 0:23:31 0:10:31 0:13:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} 3
fail 6771544 2022-04-01 03:22:19 2022-04-04 05:18:16 2022-04-04 05:28:33 0:10:17 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

Command failed on smithi117 with status 2: 'sudo dpkg -i /tmp/linux-image.deb'

fail 6771545 2022-04-01 03:22:20 2022-04-04 05:18:26 2022-04-04 05:50:26 0:32:00 0:17:44 0:14:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_enumerate} 3
Failure Reason:

"2022-04-04T05:37:28.545273+0000 mon.a (mon.0) 207 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6771546 2022-04-01 03:22:21 2022-04-04 05:18:36 2022-04-04 05:44:33 0:25:57 0:13:36 0:12:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
fail 6771547 2022-04-01 03:22:22 2022-04-04 05:18:47 2022-04-04 05:50:55 0:32:08 0:18:00 0:14:08 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
Failure Reason:

"2022-04-04T05:37:57.180568+0000 mon.a (mon.0) 188 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6771548 2022-04-01 03:22:22 2022-04-04 05:19:07 2022-04-04 05:50:52 0:31:45 0:18:02 0:13:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-04-04T05:48:40.353976+0000 mon.a (mon.0) 287 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

dead 6771549 2022-04-01 03:22:23 2022-04-04 05:19:27 2022-04-04 05:28:33 0:09:06 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} 1
Failure Reason:

SSH connection to smithi117 was lost: 'sudo dpkg -i /tmp/linux-image.deb'

fail 6771550 2022-04-01 03:22:24 2022-04-04 05:19:28 2022-04-04 05:51:35 0:32:07 0:18:03 0:14:04 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2022-04-04T05:48:48.144127+0000 mon.a (mon.0) 214 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771551 2022-04-01 03:22:25 2022-04-04 05:20:18 2022-04-04 05:45:54 0:25:36 0:10:56 0:14:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6771552 2022-04-01 03:22:26 2022-04-04 05:22:29 2022-04-04 05:43:54 0:21:25 0:10:02 0:11:23 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6771553 2022-04-01 03:22:27 2022-04-04 05:22:29 2022-04-04 05:50:15 0:27:46 0:15:43 0:12:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

"2022-04-04T05:46:30.230205+0000 osd.7 (osd.7) 3 : cluster [WRN] 3 slow requests (by type [ 'started' : 3 ] most affected pool [ 'rbd' : 3 ])" in cluster log

pass 6771554 2022-04-01 03:22:27 2022-04-04 05:22:39 2022-04-04 05:45:01 0:22:22 0:09:38 0:12:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6771555 2022-04-01 03:22:28 2022-04-04 05:23:30 2022-04-04 05:57:53 0:34:23 0:21:25 0:12:58 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
pass 6771556 2022-04-01 03:22:29 2022-04-04 05:23:50 2022-04-04 05:47:34 0:23:44 0:09:26 0:14:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_image_read} 3
pass 6771557 2022-04-01 03:22:30 2022-04-04 05:25:41 2022-04-04 05:45:05 0:19:24 0:07:41 0:11:43 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_whole_object_zeroout} 1
pass 6771558 2022-04-01 03:22:31 2022-04-04 05:25:41 2022-04-04 05:51:39 0:25:58 0:13:31 0:12:27 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6771559 2022-04-01 03:22:32 2022-04-04 05:26:11 2022-04-04 06:13:45 0:47:34 0:35:21 0:12:13 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} 3
pass 6771560 2022-04-01 03:22:33 2022-04-04 05:28:22 2022-04-04 05:47:55 0:19:33 0:09:09 0:10:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_kernel} 3
pass 6771561 2022-04-01 03:22:33 2022-04-04 05:28:42 2022-04-04 05:54:10 0:25:28 0:14:33 0:10:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6771562 2022-04-01 03:22:34 2022-04-04 05:29:03 2022-04-04 05:49:05 0:20:02 0:08:56 0:11:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6771563 2022-04-01 03:22:35 2022-04-04 05:29:03 2022-04-04 05:55:38 0:26:35 0:13:32 0:13:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_map_unmap} 3
fail 6771564 2022-04-01 03:22:36 2022-04-04 05:32:04 2022-04-04 06:03:28 0:31:24 0:17:47 0:13:37 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
Failure Reason:

"2022-04-04T06:00:37.140036+0000 mon.a (mon.0) 299 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6771565 2022-04-01 03:22:37 2022-04-04 05:33:14 2022-04-04 06:03:08 0:29:54 0:17:59 0:11:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_simple_big} 3
Failure Reason:

"2022-04-04T05:49:27.738210+0000 mon.a (mon.0) 115 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6771566 2022-04-01 03:22:38 2022-04-04 05:33:15 2022-04-04 05:52:23 0:19:08 0:08:33 0:10:35 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_deep_flatten} 1
pass 6771567 2022-04-01 03:22:38 2022-04-04 05:33:15 2022-04-04 06:09:24 0:36:09 0:23:29 0:12:40 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6771568 2022-04-01 03:22:39 2022-04-04 05:35:56 2022-04-04 06:14:31 0:38:35 0:26:38 0:11:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} 3
pass 6771569 2022-04-01 03:22:40 2022-04-04 05:36:46 2022-04-04 06:09:25 0:32:39 0:18:06 0:14:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} 3
pass 6771570 2022-04-01 03:22:41 2022-04-04 05:39:27 2022-04-04 05:57:19 0:17:52 0:07:44 0:10:08 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} 2
pass 6771571 2022-04-01 03:22:42 2022-04-04 05:39:57 2022-04-04 06:09:16 0:29:19 0:19:45 0:09:34 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} 3
fail 6771572 2022-04-01 03:22:42 2022-04-04 05:39:58 2022-04-04 06:09:36 0:29:38 0:17:52 0:11:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2022-04-04T06:06:33.214504+0000 mon.a (mon.0) 242 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6771573 2022-04-01 03:22:43 2022-04-04 05:40:18 2022-04-04 06:11:18 0:31:00 0:18:21 0:12:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-04-04T06:09:15.752446+0000 mon.a (mon.0) 244 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771574 2022-04-01 03:22:44 2022-04-04 05:41:08 2022-04-04 06:02:16 0:21:08 0:10:14 0:10:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_fallocate} 3
pass 6771575 2022-04-01 03:22:45 2022-04-04 05:41:49 2022-04-04 06:04:26 0:22:37 0:10:14 0:12:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} 3
fail 6771576 2022-04-01 03:22:46 2022-04-04 05:42:59 2022-04-04 06:13:17 0:30:18 0:18:13 0:12:05 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
Failure Reason:

"2022-04-04T06:10:26.688337+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771577 2022-04-01 03:22:46 2022-04-04 05:43:50 2022-04-04 06:07:37 0:23:47 0:12:46 0:11:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6771578 2022-04-01 03:22:47 2022-04-04 05:44:00 2022-04-04 06:03:10 0:19:10 0:08:22 0:10:48 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} 1
pass 6771579 2022-04-01 03:22:48 2022-04-04 05:44:01 2022-04-04 06:07:08 0:23:07 0:13:58 0:09:09 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
fail 6771580 2022-04-01 03:22:49 2022-04-04 05:44:01 2022-04-04 06:12:27 0:28:26 0:17:55 0:10:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2022-04-04T06:00:34.253275+0000 mon.a (mon.0) 130 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6771581 2022-04-01 03:22:50 2022-04-04 05:44:41 2022-04-04 06:12:36 0:27:55 0:17:57 0:09:58 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2022-04-04T06:11:12.692166+0000 mon.a (mon.0) 236 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6771582 2022-04-01 03:22:51 2022-04-04 05:45:02 2022-04-04 06:35:38 0:50:36 0:38:36 0:12:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6771583 2022-04-01 03:22:52 2022-04-04 05:46:02 2022-04-04 06:11:33 0:25:31 0:13:36 0:11:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6771584 2022-04-01 03:22:52 2022-04-04 05:55:48 2022-04-04 06:20:08 0:24:20 0:10:53 0:13:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6771585 2022-04-01 03:22:53 2022-04-04 05:57:58 2022-04-04 06:21:48 0:23:50 0:09:55 0:13:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_netns} 3
pass 6771586 2022-04-01 03:22:54 2022-04-04 06:02:19 2022-04-04 06:19:39 0:17:20 0:07:36 0:09:44 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_huge_image} 1
pass 6771587 2022-04-01 03:22:55 2022-04-04 06:02:19 2022-04-04 06:33:11 0:30:52 0:18:55 0:11:57 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6771588 2022-04-01 03:22:56 2022-04-04 06:03:10 2022-04-04 07:41:40 1:38:30 1:27:05 0:11:25 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6771589 2022-04-01 03:22:57 2022-04-04 06:03:30 2022-04-04 06:32:11 0:28:41 0:17:45 0:10:56 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} 3
Failure Reason:

"2022-04-04T06:31:00.046634+0000 mon.a (mon.0) 233 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6771590 2022-04-01 03:22:57 2022-04-04 06:04:31 2022-04-04 06:36:29 0:31:58 0:17:52 0:14:06 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
Failure Reason:

"2022-04-04T06:33:20.455190+0000 mon.a (mon.0) 306 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771591 2022-04-01 03:22:58 2022-04-04 06:07:11 2022-04-04 06:33:02 0:25:51 0:14:06 0:11:45 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_concurrent} 3
fail 6771592 2022-04-01 03:22:59 2022-04-04 06:07:42 2022-04-04 06:37:11 0:29:29 0:17:48 0:11:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2022-04-04T06:34:35.057785+0000 mon.a (mon.0) 297 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6771593 2022-04-01 03:23:00 2022-04-04 06:07:42 2022-04-04 06:36:42 0:29:00 0:17:45 0:11:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6771594 2022-04-01 03:23:01 2022-04-04 08:26:08 2022-04-04 08:45:26 0:19:18 0:09:13 0:10:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} 3
pass 6771595 2022-04-01 03:23:01 2022-04-04 08:26:08 2022-04-04 08:42:55 0:16:47 0:07:40 0:09:07 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_modprobe} 1
pass 6771596 2022-04-01 03:23:02 2022-04-04 08:26:08 2022-04-04 08:56:30 0:30:22 0:18:14 0:12:08 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
fail 6771597 2022-04-01 03:23:03 2022-04-04 08:26:28 2022-04-04 08:55:37 0:29:09 0:17:54 0:11:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-04-04T08:54:04.333520+0000 mon.a (mon.0) 226 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6771598 2022-04-01 03:23:04 2022-04-04 08:27:59 2022-04-04 08:56:09 0:28:10 0:17:34 0:10:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2022-04-04T08:44:06.644320+0000 mon.a (mon.0) 163 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6771599 2022-04-01 03:23:05 2022-04-04 08:29:00 2022-04-04 08:49:02 0:20:02 0:08:42 0:11:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6771600 2022-04-01 03:23:05 2022-04-04 08:29:40 2022-04-04 08:54:57 0:25:17 0:14:01 0:11:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} 3
pass 6771601 2022-04-01 03:23:06 2022-04-04 08:31:31 2022-04-04 08:53:30 0:21:59 0:10:10 0:11:49 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6771602 2022-04-01 03:23:07 2022-04-04 08:32:21 2022-04-04 09:09:36 0:37:15 0:21:51 0:15:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_simple_big} 3
pass 6771603 2022-04-01 03:23:08 2022-04-04 08:37:22 2022-04-04 08:55:08 0:17:46 0:07:46 0:10:00 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} 1
fail 6771604 2022-04-01 03:23:09 2022-04-04 08:38:12 2022-04-04 09:14:30 0:36:18 0:25:50 0:10:28 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2022-04-04T09:02:29.639376+0000 osd.0 (osd.0) 3 : cluster [WRN] 1 slow requests (by type [ 'waiting for sub ops' : 1 ] most affected pool [ 'pool_client.0' : 1 ])" in cluster log

fail 6771605 2022-04-01 03:23:09 2022-04-04 08:39:13 2022-04-04 09:08:02 0:28:49 0:17:37 0:11:12 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

"2022-04-04T08:56:19.036197+0000 mon.a (mon.0) 153 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6771606 2022-04-01 03:23:10 2022-04-04 08:40:23 2022-04-04 09:12:22 0:31:59 0:17:49 0:14:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-04-04T09:09:24.936335+0000 mon.a (mon.0) 254 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771607 2022-04-01 03:23:11 2022-04-04 08:42:24 2022-04-04 09:05:57 0:23:33 0:10:00 0:13:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6771608 2022-04-01 03:23:12 2022-04-04 08:44:24 2022-04-04 09:12:55 0:28:31 0:17:17 0:11:14 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6771609 2022-04-01 03:23:13 2022-04-04 08:45:35 2022-04-04 09:08:23 0:22:48 0:10:39 0:12:09 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_fallocate} 3
pass 6771610 2022-04-01 03:23:14 2022-04-04 08:46:55 2022-04-04 09:10:25 0:23:30 0:10:01 0:13:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6771611 2022-04-01 03:23:15 2022-04-04 08:49:06 2022-04-04 09:42:58 0:53:52 0:36:15 0:17:37 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
fail 6771612 2022-04-01 03:23:15 2022-04-04 08:53:37 2022-04-04 09:23:38 0:30:01 0:17:50 0:12:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
Failure Reason:

"2022-04-04T09:22:23.271045+0000 mon.a (mon.0) 218 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771613 2022-04-01 03:23:16 2022-04-04 08:54:17 2022-04-04 09:14:35 0:20:18 0:08:02 0:12:16 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_parent_overlap} 1
pass 6771614 2022-04-01 03:23:17 2022-04-04 08:54:58 2022-04-04 09:22:28 0:27:30 0:14:17 0:13:13 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6771615 2022-04-01 03:23:18 2022-04-04 08:55:08 2022-04-04 09:49:08 0:54:00 0:41:00 0:13:00 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/upmap thrashosds-health workloads/rbd_fio} 3
pass 6771616 2022-04-01 03:23:19 2022-04-04 08:55:38 2022-04-04 09:17:30 0:21:52 0:09:39 0:12:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_namespaces} 3
pass 6771617 2022-04-01 03:23:19 2022-04-04 08:56:19 2022-04-04 10:23:01 1:26:42 1:12:18 0:14:24 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6771618 2022-04-01 03:23:20 2022-04-04 08:56:39 2022-04-04 09:54:03 0:57:24 0:42:58 0:14:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_enumerate} 3
fail 6771619 2022-04-01 03:23:21 2022-04-04 08:58:40 2022-04-04 09:32:52 0:34:12 0:17:42 0:16:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2022-04-04T09:29:55.335874+0000 mon.a (mon.0) 202 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771620 2022-04-01 03:23:22 2022-04-04 09:03:21 2022-04-04 09:27:12 0:23:51 0:11:11 0:12:40 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
fail 6771621 2022-04-01 03:23:22 2022-04-04 09:05:51 2022-04-04 09:35:46 0:29:55 0:17:52 0:12:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-04-04T09:32:40.067923+0000 mon.a (mon.0) 254 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771622 2022-04-01 03:23:23 2022-04-04 09:06:02 2022-04-04 09:25:26 0:19:24 0:08:13 0:11:11 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_read_only} 1
pass 6771623 2022-04-01 03:23:24 2022-04-04 09:08:12 2022-04-04 09:38:00 0:29:48 0:19:34 0:10:14 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
fail 6771624 2022-04-01 03:23:25 2022-04-04 09:08:33 2022-04-04 09:39:56 0:31:23 0:18:07 0:13:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_symlinks} 3
Failure Reason:

"2022-04-04T09:36:52.641019+0000 mon.a (mon.0) 214 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771625 2022-04-01 03:23:26 2022-04-04 09:09:43 2022-04-04 09:29:51 0:20:08 0:09:14 0:10:54 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
dead 6771626 2022-04-01 03:23:26 2022-04-04 09:10:33 2022-04-04 12:17:27 3:06:54 2:55:08 0:11:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

SSH connection to smithi178 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

pass 6771627 2022-04-01 03:23:27 2022-04-04 09:11:34 2022-04-04 09:34:28 0:22:54 0:09:26 0:13:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6771628 2022-04-01 03:23:28 2022-04-04 09:12:15 2022-04-04 09:41:43 0:29:28 0:17:51 0:11:37 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2022-04-04T09:39:00.545432+0000 mon.a (mon.0) 234 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771629 2022-04-01 03:23:29 2022-04-04 09:12:15 2022-04-04 09:34:12 0:21:57 0:09:55 0:12:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_image_read} 3
pass 6771630 2022-04-01 03:23:29 2022-04-04 09:12:25 2022-04-04 09:31:14 0:18:49 0:08:25 0:10:24 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} 1
pass 6771631 2022-04-01 03:23:30 2022-04-04 09:12:26 2022-04-04 09:38:27 0:26:01 0:15:38 0:10:23 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6771632 2022-04-01 03:23:31 2022-04-04 09:13:06 2022-04-04 09:32:51 0:19:45 0:09:33 0:10:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} 3
pass 6771633 2022-04-01 03:23:32 2022-04-04 09:13:06 2022-04-04 09:40:12 0:27:06 0:14:39 0:12:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
fail 6771634 2022-04-01 03:23:33 2022-04-04 09:14:37 2022-04-04 09:47:11 0:32:34 0:17:45 0:14:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2022-04-04T09:44:18.237047+0000 mon.a (mon.0) 289 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6771635 2022-04-01 03:23:33 2022-04-04 09:17:38 2022-04-04 09:48:01 0:30:23 0:13:48 0:16:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_unmap} 3
pass 6771636 2022-04-01 03:23:34 2022-04-04 09:27:16 2022-04-04 09:53:18 0:26:02 0:12:55 0:13:07 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6771637 2022-04-01 03:23:35 2022-04-04 09:29:17 2022-04-04 10:07:18 0:38:01 0:27:17 0:10:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} 3