Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 6719082 2022-03-04 03:20:45 2022-03-07 16:24:37 2022-03-07 16:43:46 0:19:09 0:10:35 0:08:34 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_deep_flatten} 1
pass 6719083 2022-03-04 03:20:46 2022-03-07 16:24:38 2022-03-07 16:54:45 0:30:07 0:19:20 0:10:47 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6719084 2022-03-04 03:20:46 2022-03-07 16:25:18 2022-03-07 16:48:42 0:23:24 0:10:34 0:12:50 smithi master krbd/ms_modeless/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_default_map_options} 3
pass 6719085 2022-03-04 03:20:47 2022-03-07 16:25:19 2022-03-07 17:06:51 0:41:32 0:28:37 0:12:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_fio} 3
pass 6719086 2022-03-04 03:20:48 2022-03-07 16:25:39 2022-03-07 16:58:31 0:32:52 0:18:37 0:14:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_data_pool} 3
fail 6719087 2022-03-04 03:20:49 2022-03-07 16:27:20 2022-03-07 21:04:04 4:36:44 4:21:42 0:15:02 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy-rxbounce} msgr-failures/few tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi159 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/exclude5mk524_n -r -- '-g auto -g blockdev -x clone'"

pass 6719088 2022-03-04 03:20:50 2022-03-07 16:27:50 2022-03-07 17:25:33 0:57:43 0:43:42 0:14:01 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/backoff thrashosds-health workloads/rbd_fio} 3
fail 6719089 2022-03-04 03:20:50 2022-03-07 16:28:11 2022-03-07 16:35:56 0:07:45 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major

pass 6719090 2022-03-04 03:20:51 2022-03-07 16:28:41 2022-03-07 16:52:12 0:23:31 0:09:11 0:14:20 smithi master krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} 1
fail 6719091 2022-03-04 03:20:52 2022-03-07 16:28:41 2022-03-07 17:05:27 0:36:46 0:19:23 0:17:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2022-03-07T17:02:36.344734+0000 mon.a (mon.0) 226 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719092 2022-03-04 03:20:53 2022-03-07 16:29:32 2022-03-07 17:05:37 0:36:05 0:20:16 0:15:49 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6719093 2022-03-04 03:20:54 2022-03-07 16:30:12 2022-03-07 16:56:48 0:26:36 0:11:18 0:15:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} 3
fail 6719094 2022-03-04 03:20:54 2022-03-07 16:30:43 2022-03-07 17:07:14 0:36:31 0:19:58 0:16:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-03-07T17:04:15.953036+0000 mon.a (mon.0) 221 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719095 2022-03-04 03:20:55 2022-03-07 16:31:43 2022-03-07 17:24:44 0:53:01 0:37:38 0:15:23 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6719096 2022-03-04 03:20:56 2022-03-07 16:32:34 2022-03-07 17:00:25 0:27:51 0:14:21 0:13:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6719097 2022-03-04 03:20:57 2022-03-07 16:32:34 2022-03-07 16:55:25 0:22:51 0:10:15 0:12:36 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_discard} 1
pass 6719098 2022-03-04 03:20:58 2022-03-07 16:32:34 2022-03-07 17:02:57 0:30:23 0:16:37 0:13:46 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6719099 2022-03-04 03:20:59 2022-03-07 16:35:25 2022-03-07 17:01:00 0:25:35 0:13:06 0:12:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} 3
pass 6719100 2022-03-04 03:20:59 2022-03-07 16:35:35 2022-03-07 18:11:50 1:36:15 1:24:30 0:11:45 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6719101 2022-03-04 03:21:00 2022-03-07 16:36:06 2022-03-07 17:27:18 0:51:12 0:38:19 0:12:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6719102 2022-03-04 03:21:01 2022-03-07 16:37:56 2022-03-07 17:03:41 0:25:45 0:15:28 0:10:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6719103 2022-03-04 03:21:02 2022-03-07 16:38:07 2022-03-07 17:02:35 0:24:28 0:11:50 0:12:38 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6719104 2022-03-04 03:21:02 2022-03-07 16:41:07 2022-03-07 17:05:29 0:24:22 0:11:40 0:12:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} 3
pass 6719105 2022-03-04 03:21:03 2022-03-07 16:41:58 2022-03-07 17:01:30 0:19:32 0:09:02 0:10:30 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} 1
fail 6719106 2022-03-04 03:21:04 2022-03-07 16:41:58 2022-03-07 17:16:20 0:34:22 0:20:02 0:14:20 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2022-03-07T17:13:58.233601+0000 mon.a (mon.0) 334 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719107 2022-03-04 03:21:05 2022-03-07 16:42:09 2022-03-07 17:06:09 0:24:00 0:11:43 0:12:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6719108 2022-03-04 03:21:06 2022-03-07 16:42:49 2022-03-07 17:07:47 0:24:58 0:11:26 0:13:32 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6719109 2022-03-04 03:21:06 2022-03-07 16:44:10 2022-03-07 17:13:18 0:29:08 0:16:20 0:12:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_concurrent} 3
fail 6719110 2022-03-04 03:21:07 2022-03-07 16:44:10 2022-03-07 17:18:29 0:34:19 0:19:31 0:14:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2022-03-07T17:16:34.029690+0000 mon.a (mon.0) 248 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719111 2022-03-04 03:21:08 2022-03-07 16:45:30 2022-03-07 17:20:29 0:34:59 0:18:55 0:16:04 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6719112 2022-03-04 03:21:09 2022-03-07 16:47:21 2022-03-07 17:10:05 0:22:44 0:11:03 0:11:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_image_read} 3
pass 6719113 2022-03-04 03:21:09 2022-03-07 16:48:22 2022-03-07 17:08:04 0:19:42 0:09:09 0:10:33 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_modprobe} 1
pass 6719114 2022-03-04 03:21:10 2022-03-07 16:48:52 2022-03-07 17:18:52 0:30:00 0:16:15 0:13:45 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6719115 2022-03-04 03:21:11 2022-03-07 16:49:52 2022-03-07 18:17:55 1:28:03 1:14:39 0:13:24 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6719116 2022-03-04 03:21:12 2022-03-07 16:52:13 2022-03-07 17:15:59 0:23:46 0:10:24 0:13:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_kernel} 3
pass 6719117 2022-03-04 03:21:13 2022-03-07 16:54:24 2022-03-07 17:24:37 0:30:13 0:18:11 0:12:02 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6719118 2022-03-04 03:21:13 2022-03-07 16:54:54 2022-03-07 17:17:54 0:23:00 0:10:10 0:12:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6719119 2022-03-04 03:21:14 2022-03-07 16:55:55 2022-03-07 17:22:27 0:26:32 0:15:17 0:11:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_map_unmap} 3
pass 6719120 2022-03-04 03:21:15 2022-03-07 16:56:45 2022-03-07 17:21:17 0:24:32 0:11:24 0:13:08 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6719121 2022-03-04 03:21:16 2022-03-07 16:56:55 2022-03-07 17:31:59 0:35:04 0:22:38 0:12:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_simple_big} 3
pass 6719122 2022-03-04 03:21:16 2022-03-07 16:58:36 2022-03-07 17:17:39 0:19:03 0:09:34 0:09:29 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_msgr_segments} 1
dead 6719123 2022-03-04 03:21:17 2022-03-07 16:58:36 2022-03-07 17:01:51 0:03:15 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
Failure Reason:

Error reimaging machines: Failed to power on smithi029

pass 6719124 2022-03-04 03:21:18 2022-03-07 17:00:27 2022-03-07 17:40:44 0:40:17 0:28:20 0:11:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} 3
pass 6719125 2022-03-04 03:21:19 2022-03-07 17:01:07 2022-03-07 17:35:17 0:34:10 0:23:26 0:10:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} 3
pass 6719126 2022-03-04 03:21:20 2022-03-07 17:01:38 2022-03-07 17:25:50 0:24:12 0:11:35 0:12:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6719127 2022-03-04 03:21:20 2022-03-07 17:01:58 2022-03-07 17:31:49 0:29:51 0:18:15 0:11:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
fail 6719128 2022-03-04 03:21:21 2022-03-07 17:02:09 2022-03-07 17:34:02 0:31:53 0:19:35 0:12:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_fallocate} 3
Failure Reason:

"2022-03-07T17:32:17.842958+0000 mon.a (mon.0) 281 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719129 2022-03-04 03:21:22 2022-03-07 17:02:39 2022-03-07 17:28:30 0:25:51 0:11:24 0:14:27 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} 3
fail 6719130 2022-03-04 03:21:23 2022-03-07 17:02:59 2022-03-07 17:35:28 0:32:29 0:19:26 0:13:03 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
Failure Reason:

"2022-03-07T17:33:32.199541+0000 mon.a (mon.0) 228 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719131 2022-03-04 03:21:24 2022-03-07 17:03:50 2022-03-07 17:32:59 0:29:09 0:14:36 0:14:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6719132 2022-03-04 03:21:24 2022-03-07 17:05:30 2022-03-07 17:26:47 0:21:17 0:08:52 0:12:25 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} 1
pass 6719133 2022-03-04 03:21:25 2022-03-07 17:05:31 2022-03-07 17:35:00 0:29:29 0:17:47 0:11:42 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6719134 2022-03-04 03:21:26 2022-03-07 17:05:31 2022-03-07 17:29:14 0:23:43 0:11:43 0:12:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_namespaces} 3
pass 6719135 2022-03-04 03:21:27 2022-03-07 17:05:41 2022-03-07 18:35:37 1:29:56 1:16:51 0:13:05 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6719136 2022-03-04 03:21:28 2022-03-07 17:06:12 2022-03-07 18:00:15 0:54:03 0:40:24 0:13:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6719137 2022-03-04 03:21:28 2022-03-07 17:06:52 2022-03-07 17:34:46 0:27:54 0:15:14 0:12:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6719138 2022-03-04 03:21:29 2022-03-07 17:07:23 2022-03-07 17:31:38 0:24:15 0:12:57 0:11:18 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6719139 2022-03-04 03:21:30 2022-03-07 17:07:53 2022-03-07 17:33:57 0:26:04 0:13:53 0:12:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netns} 3
pass 6719140 2022-03-04 03:21:31 2022-03-07 17:10:20 2022-03-07 17:29:35 0:19:15 0:09:31 0:09:44 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_read_only} 1
pass 6719141 2022-03-04 03:21:31 2022-03-07 17:10:20 2022-03-07 17:42:36 0:32:16 0:19:20 0:12:56 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6719142 2022-03-04 03:21:32 2022-03-07 17:13:22 2022-03-07 18:07:29 0:54:07 0:42:53 0:11:14 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/pggrow thrashosds-health workloads/rbd_fio} 3
pass 6719143 2022-03-04 03:21:33 2022-03-07 17:14:03 2022-03-07 17:37:37 0:23:34 0:11:46 0:11:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6719144 2022-03-04 03:21:34 2022-03-07 17:16:06 2022-03-07 17:38:26 0:22:20 0:10:54 0:11:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6719145 2022-03-04 03:21:34 2022-03-07 17:16:26 2022-03-07 19:43:17 2:26:51 2:15:06 0:11:45 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

"2022-03-07T18:46:21.094260+0000 osd.3 (osd.3) 3 : cluster [WRN] 1 slow requests (by type [ 'started' : 1 ] most affected pool [ 'rbd' : 1 ])" in cluster log

pass 6719146 2022-03-04 03:21:35 2022-03-07 17:22:35 2022-03-07 17:46:56 0:24:21 0:11:14 0:13:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6719147 2022-03-04 03:21:36 2022-03-07 17:24:45 2022-03-07 17:56:19 0:31:34 0:21:26 0:10:08 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2022-03-07T17:46:11.001693+0000 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 0 slow ops, oldest one blocked for 34 sec, osd.4 has slow ops (SLOW_OPS)" in cluster log

pass 6719148 2022-03-04 03:21:37 2022-03-07 17:24:46 2022-03-07 17:46:54 0:22:08 0:11:12 0:10:56 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} 3
pass 6719149 2022-03-04 03:21:38 2022-03-07 17:25:36 2022-03-07 17:44:54 0:19:18 0:09:42 0:09:36 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_whole_object_zeroout} 1
pass 6719150 2022-03-04 03:21:38 2022-03-07 17:25:36 2022-03-07 17:55:54 0:30:18 0:18:49 0:11:29 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6719151 2022-03-04 03:21:39 2022-03-07 17:25:57 2022-03-07 17:48:14 0:22:17 0:10:43 0:11:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_kernel} 3
pass 6719152 2022-03-04 03:21:40 2022-03-07 17:26:57 2022-03-07 17:54:52 0:27:55 0:16:01 0:11:54 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6719153 2022-03-04 03:21:41 2022-03-07 17:27:28 2022-03-07 17:50:06 0:22:38 0:10:24 0:12:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6719154 2022-03-04 03:21:41 2022-03-07 17:28:38 2022-03-07 17:56:47 0:28:09 0:15:43 0:12:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} 3
pass 6719155 2022-03-04 03:21:42 2022-03-07 17:29:19 2022-03-07 17:53:54 0:24:35 0:12:31 0:12:04 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6719156 2022-03-04 03:21:43 2022-03-07 17:29:39 2022-03-07 18:07:24 0:37:45 0:25:28 0:12:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_simple_big} 3
pass 6719157 2022-03-04 03:21:44 2022-03-07 17:31:40 2022-03-07 17:51:33 0:19:53 0:09:44 0:10:09 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} 1
pass 6719158 2022-03-04 03:21:45 2022-03-07 17:31:50 2022-03-07 18:05:25 0:33:35 0:22:42 0:10:53 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6719159 2022-03-04 03:21:45 2022-03-07 17:32:00 2022-03-07 18:10:27 0:38:27 0:27:28 0:10:59 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_fio} 3
pass 6719160 2022-03-04 03:21:46 2022-03-07 17:33:01 2022-03-07 18:05:48 0:32:47 0:18:41 0:14:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_data_pool} 3
pass 6719161 2022-03-04 03:21:47 2022-03-07 17:34:01 2022-03-07 17:55:43 0:21:42 0:09:37 0:12:05 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} 2
fail 6719162 2022-03-04 03:21:48 2022-03-07 17:34:02 2022-03-07 18:05:30 0:31:28 0:19:16 0:12:12 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} 3
Failure Reason:

"2022-03-07T18:02:39.915774+0000 mon.a (mon.0) 288 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719163 2022-03-04 03:21:49 2022-03-07 17:34:12 2022-03-07 17:58:27 0:24:15 0:11:41 0:12:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} 3
fail 6719164 2022-03-04 03:21:49 2022-03-07 17:34:52 2022-03-07 18:06:42 0:31:50 0:20:16 0:11:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-03-07T17:55:19.893786+0000 mon.a (mon.0) 159 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log

pass 6719165 2022-03-04 03:21:50 2022-03-07 17:35:03 2022-03-07 17:59:08 0:24:05 0:11:44 0:12:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_fallocate} 3
pass 6719166 2022-03-04 03:21:52 2022-03-07 17:35:23 2022-03-07 17:59:00 0:23:37 0:12:24 0:11:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6719167 2022-03-04 03:21:53 2022-03-07 17:35:33 2022-03-07 18:24:32 0:48:59 0:36:47 0:12:12 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6719168 2022-03-04 03:21:53 2022-03-07 17:36:24 2022-03-07 18:01:15 0:24:51 0:13:45 0:11:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6719169 2022-03-04 03:21:54 2022-03-07 17:37:45 2022-03-07 17:56:44 0:18:59 0:09:52 0:09:07 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_discard} 1
pass 6719170 2022-03-04 03:21:55 2022-03-07 17:37:45 2022-03-07 18:06:47 0:29:02 0:17:19 0:11:43 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6719171 2022-03-04 03:21:56 2022-03-07 17:38:35 2022-03-07 19:18:23 1:39:48 1:27:29 0:12:19 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6719172 2022-03-04 03:21:57 2022-03-07 17:40:46 2022-03-07 18:06:01 0:25:15 0:12:11 0:13:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_namespaces} 3
pass 6719173 2022-03-04 03:21:57 2022-03-07 17:42:47 2022-03-07 19:21:17 1:38:30 1:26:37 0:11:53 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6719174 2022-03-04 03:21:58 2022-03-07 17:43:47 2022-03-07 18:32:23 0:48:36 0:37:44 0:10:52 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6719175 2022-03-04 03:21:59 2022-03-07 17:44:58 2022-03-07 18:15:39 0:30:41 0:15:59 0:14:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6719176 2022-03-04 03:22:00 2022-03-07 17:46:48 2022-03-07 18:09:02 0:22:14 0:12:13 0:10:01 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6719177 2022-03-04 03:22:01 2022-03-07 17:46:59 2022-03-07 18:08:25 0:21:26 0:12:05 0:09:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_netns} 3
pass 6719178 2022-03-04 03:22:01 2022-03-07 17:46:59 2022-03-07 18:07:35 0:20:36 0:09:06 0:11:30 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_huge_image} 1
pass 6719179 2022-03-04 03:22:02 2022-03-07 17:48:19 2022-03-07 18:21:51 0:33:32 0:21:59 0:11:33 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6719180 2022-03-04 03:22:03 2022-03-07 17:50:10 2022-03-07 18:14:09 0:23:59 0:12:43 0:11:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6719181 2022-03-04 03:22:04 2022-03-07 17:50:41 2022-03-07 18:13:20 0:22:39 0:11:09 0:11:30 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6719182 2022-03-04 03:22:05 2022-03-07 17:51:41 2022-03-07 18:24:26 0:32:45 0:16:45 0:16:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} 3
pass 6719183 2022-03-04 03:22:05 2022-03-07 17:54:02 2022-03-07 18:20:59 0:26:57 0:11:15 0:15:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6719184 2022-03-04 03:22:06 2022-03-07 17:54:52 2022-03-07 18:28:27 0:33:35 0:20:09 0:13:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6719185 2022-03-04 03:22:07 2022-03-07 17:55:03 2022-03-07 18:21:52 0:26:49 0:11:33 0:15:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_image_read} 3
pass 6719186 2022-03-04 03:22:08 2022-03-07 17:56:03 2022-03-07 18:19:38 0:23:35 0:09:52 0:13:43 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} 1
fail 6719187 2022-03-04 03:22:09 2022-03-07 17:56:03 2022-03-07 18:29:49 0:33:46 0:20:03 0:13:43 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

"2022-03-07T18:18:13.244948+0000 mon.a (mon.0) 191 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6719188 2022-03-04 03:22:09 2022-03-07 17:56:24 2022-03-07 18:30:19 0:33:55 0:19:56 0:13:59 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-03-07T18:19:06.394093+0000 mon.a (mon.0) 147 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log

pass 6719189 2022-03-04 03:22:10 2022-03-07 17:56:54 2022-03-07 18:30:39 0:33:45 0:18:39 0:15:06 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6719190 2022-03-04 03:22:11 2022-03-07 18:23:23 666 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6719191 2022-03-04 03:22:12 2022-03-07 17:57:45 2022-03-07 18:28:13 0:30:28 0:16:21 0:14:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_unmap} 3
pass 6719192 2022-03-04 03:22:13 2022-03-07 18:24:44 759 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6719193 2022-03-04 03:22:14 2022-03-07 17:59:06 2022-03-07 18:36:57 0:37:51 0:26:03 0:11:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} 3
pass 6719194 2022-03-04 03:22:14 2022-03-07 17:59:16 2022-03-07 18:21:32 0:22:16 0:09:34 0:12:42 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_msgr_segments} 1
pass 6719195 2022-03-04 03:22:15 2022-03-07 18:00:17 2022-03-07 18:35:57 0:35:40 0:23:20 0:12:20 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6719196 2022-03-04 03:22:16 2022-03-07 18:00:27 2022-03-07 18:24:51 0:24:24 0:12:29 0:11:55 smithi master krbd/ms_modeless/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_rxbounce} 3
pass 6719197 2022-03-04 03:22:17 2022-03-07 18:01:18 2022-03-07 18:43:05 0:41:47 0:30:26 0:11:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_fio} 3
fail 6719198 2022-03-04 03:22:18 2022-03-07 18:01:38 2022-03-07 18:35:22 0:33:44 0:19:35 0:14:09 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-03-07T18:32:48.664875+0000 mon.a (mon.0) 307 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 6719199 2022-03-04 03:22:19 2022-03-07 18:03:29 2022-03-07 18:34:54 0:31:25 0:18:58 0:12:27 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{secure} msgr-failures/many tasks/rbd_xfstests} 4
Failure Reason:

"2022-03-07T18:33:39.245066+0000 mon.a (mon.0) 187 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719200 2022-03-04 03:22:19 2022-03-07 18:05:29 2022-03-07 19:41:16 1:35:47 1:24:33 0:11:14 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6719201 2022-03-04 03:22:20 2022-03-07 18:05:40 2022-03-07 18:29:38 0:23:58 0:12:06 0:11:52 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6719202 2022-03-04 03:22:21 2022-03-07 18:05:50 2022-03-07 18:37:29 0:31:39 0:18:40 0:12:59 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
fail 6719203 2022-03-04 03:22:22 2022-03-07 18:06:10 2022-03-07 18:43:01 0:36:51 0:19:37 0:17:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} 3
Failure Reason:

"2022-03-07T18:30:13.489839+0000 mon.a (mon.0) 170 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6719204 2022-03-04 03:22:23 2022-03-07 18:06:51 2022-03-07 18:32:14 0:25:23 0:11:46 0:13:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6719205 2022-03-04 03:22:23 2022-03-07 18:06:51 2022-03-07 18:59:03 0:52:12 0:37:26 0:14:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6719206 2022-03-04 03:22:24 2022-03-07 18:07:32 2022-03-07 18:37:19 0:29:47 0:14:43 0:15:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6719207 2022-03-04 03:22:25 2022-03-07 18:07:32 2022-03-07 18:31:01 0:23:29 0:08:57 0:14:32 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_parent_overlap} 1
pass 6719208 2022-03-04 03:22:26 2022-03-07 18:07:42 2022-03-07 18:39:56 0:32:14 0:17:15 0:14:59 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6719209 2022-03-04 03:22:27 2022-03-07 18:08:33 2022-03-07 18:35:08 0:26:35 0:11:43 0:14:52 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} 3
pass 6719210 2022-03-04 03:22:27 2022-03-07 18:09:03 2022-03-07 19:38:04 1:29:01 1:14:01 0:15:00 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
fail 6719211 2022-03-04 03:22:28 2022-03-07 18:10:34 2022-03-07 18:43:32 0:32:58 0:19:21 0:13:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_enumerate} 3
Failure Reason:

"2022-03-07T18:30:58.221751+0000 mon.a (mon.0) 135 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6719212 2022-03-04 03:22:29 2022-03-07 18:11:54 2022-03-07 18:43:14 0:31:20 0:19:13 0:12:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2022-03-07T18:42:00.005364+0000 mon.a (mon.0) 238 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6719213 2022-03-04 03:22:30 2022-03-07 18:15:45 2022-03-07 18:41:40 0:25:55 0:12:48 0:13:07 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6719214 2022-03-04 03:22:31 2022-03-07 18:18:05 2022-03-07 18:44:40 0:26:35 0:11:52 0:14:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} 3
pass 6719215 2022-03-04 03:22:31 2022-03-07 18:19:46 2022-03-07 18:40:12 0:20:26 0:09:45 0:10:41 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} 1
pass 6719216 2022-03-04 03:22:32 2022-03-07 18:21:06 2022-03-07 18:59:59 0:38:53 0:26:24 0:12:29 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6719217 2022-03-04 03:22:33 2022-03-07 18:21:37 2022-03-07 18:45:48 0:24:11 0:12:45 0:11:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6719218 2022-03-04 03:22:34 2022-03-07 18:21:57 2022-03-07 18:45:26 0:23:29 0:11:32 0:11:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6719219 2022-03-04 03:22:34 2022-03-07 18:21:57 2022-03-07 22:55:25 4:33:28 4:21:42 0:11:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

Command failed (workunit test rbd/concurrent.sh) on smithi146 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f735432033a131a8101d9f67b8c22b3d82a51a79 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh'

pass 6719220 2022-03-04 03:22:35 2022-03-07 18:23:28 2022-03-07 18:46:44 0:23:16 0:10:55 0:12:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6719221 2022-03-04 03:22:36 2022-03-07 18:24:28 2022-03-07 18:54:33 0:30:05 0:19:26 0:10:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2022-03-07T18:43:03.522715+0000 mon.a (mon.0) 114 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6719222 2022-03-04 03:22:37 2022-03-07 18:24:39 2022-03-07 18:46:24 0:21:45 0:11:10 0:10:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_image_read} 3
pass 6719223 2022-03-04 03:22:38 2022-03-07 18:24:49 2022-03-07 18:44:34 0:19:45 0:09:54 0:09:51 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_whole_object_zeroout} 1
pass 6719224 2022-03-04 03:22:38 2022-03-07 18:25:00 2022-03-07 18:53:50 0:28:50 0:14:57 0:13:53 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6719225 2022-03-04 03:22:39 2022-03-07 18:28:21 2022-03-07 19:15:56 0:47:35 0:36:33 0:11:02 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} 3
pass 6719226 2022-03-04 03:22:40 2022-03-07 18:28:31 2022-03-07 18:51:31 0:23:00 0:10:32 0:12:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_kernel} 3
pass 6719227 2022-03-04 03:22:41 2022-03-07 18:29:41 2022-03-07 18:57:18 0:27:37 0:16:22 0:11:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6719228 2022-03-04 03:22:42 2022-03-07 18:29:52 2022-03-07 18:51:39 0:21:47 0:10:23 0:11:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6719229 2022-03-04 03:22:42 2022-03-07 18:30:22 2022-03-07 18:59:38 0:29:16 0:15:26 0:13:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_map_unmap} 3
pass 6719230 2022-03-04 03:22:43 2022-03-07 18:30:43 2022-03-07 18:56:35 0:25:52 0:13:29 0:12:23 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6719231 2022-03-04 03:22:44 2022-03-07 18:31:03 2022-03-07 19:07:48 0:36:45 0:24:19 0:12:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_simple_big} 3
pass 6719232 2022-03-04 03:22:45 2022-03-07 18:32:24 2022-03-07 18:51:49 0:19:25 0:09:32 0:09:53 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_deep_flatten} 1
pass 6719233 2022-03-04 03:22:46 2022-03-07 18:32:24 2022-03-07 19:07:57 0:35:33 0:25:13 0:10:20 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6719234 2022-03-04 03:22:46 2022-03-07 18:32:24 2022-03-07 19:12:11 0:39:47 0:28:01 0:11:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} 3
pass 6719235 2022-03-04 03:22:47 2022-03-07 18:34:55 2022-03-07 19:04:23 0:29:28 0:19:19 0:10:09 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} 3
pass 6719236 2022-03-04 03:22:48 2022-03-07 18:34:56 2022-03-07 18:57:22 0:22:26 0:09:23 0:13:03 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} 2
pass 6719237 2022-03-04 03:22:49 2022-03-07 18:35:16 2022-03-07 19:11:15 0:35:59 0:22:01 0:13:58 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} 3
fail 6719238 2022-03-04 03:22:49 2022-03-07 18:35:26 2022-03-07 19:07:05 0:31:39 0:19:38 0:12:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2022-03-07T19:05:18.415007+0000 mon.a (mon.0) 275 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6719239 2022-03-04 03:22:50 2022-03-07 18:35:27 2022-03-07 19:07:04 0:31:37 0:19:28 0:12:09 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-03-07T19:05:10.997414+0000 mon.a (mon.0) 243 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719240 2022-03-04 03:22:51 2022-03-07 18:35:47 2022-03-07 18:59:23 0:23:36 0:11:13 0:12:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_fallocate} 3
pass 6719241 2022-03-04 03:22:52 2022-03-07 18:35:57 2022-03-07 19:00:49 0:24:52 0:12:16 0:12:36 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6719242 2022-03-04 03:22:53 2022-03-07 18:37:08 2022-03-07 19:27:11 0:50:03 0:38:11 0:11:52 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6719243 2022-03-04 03:22:53 2022-03-07 18:37:28 2022-03-07 19:02:54 0:25:26 0:14:29 0:10:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6719244 2022-03-04 03:22:54 2022-03-07 18:37:39 2022-03-07 18:59:03 0:21:24 0:09:47 0:11:37 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} 1
pass 6719245 2022-03-04 03:22:55 2022-03-07 18:39:39 2022-03-07 19:07:22 0:27:43 0:16:46 0:10:57 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
fail 6719246 2022-03-04 03:22:56 2022-03-07 18:40:00 2022-03-07 19:11:28 0:31:28 0:19:15 0:12:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2022-03-07T19:09:38.597244+0000 mon.a (mon.0) 326 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6719247 2022-03-04 03:22:57 2022-03-07 18:41:50 2022-03-07 20:16:40 1:34:50 1:22:44 0:12:06 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
fail 6719248 2022-03-04 03:22:57 2022-03-07 18:43:11 2022-03-07 19:15:13 0:32:02 0:19:23 0:12:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} 3
Failure Reason:

"2022-03-07T19:12:28.218322+0000 mon.a (mon.0) 185 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6719249 2022-03-04 03:22:58 2022-03-07 18:43:11 2022-03-07 19:12:53 0:29:42 0:19:13 0:10:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2022-03-07T19:11:13.064121+0000 mon.a (mon.0) 250 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719250 2022-03-04 03:22:59 2022-03-07 18:43:22 2022-03-07 19:07:09 0:23:47 0:11:52 0:11:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6719251 2022-03-04 03:23:00 2022-03-07 18:43:42 2022-03-07 19:09:41 0:25:59 0:11:59 0:14:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_netns} 3
pass 6719252 2022-03-04 03:23:00 2022-03-07 18:44:42 2022-03-07 19:04:13 0:19:31 0:08:54 0:10:37 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_huge_image} 1
pass 6719253 2022-03-04 03:23:01 2022-03-07 18:44:43 2022-03-07 19:14:59 0:30:16 0:19:27 0:10:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6719254 2022-03-04 03:23:02 2022-03-07 18:45:33 2022-03-07 20:17:59 1:32:26 1:21:05 0:11:21 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6719255 2022-03-04 03:23:03 2022-03-07 18:45:54 2022-03-07 19:10:14 0:24:20 0:11:45 0:12:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6719256 2022-03-04 03:23:04 2022-03-07 18:46:34 2022-03-07 19:09:05 0:22:31 0:11:14 0:11:17 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6719257 2022-03-04 03:23:04 2022-03-07 18:46:44 2022-03-07 19:17:49 0:31:05 0:15:25 0:15:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_concurrent} 3
pass 6719258 2022-03-04 03:23:05 2022-03-07 18:51:35 2022-03-07 19:13:13 0:21:38 0:11:00 0:10:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6719259 2022-03-04 03:23:06 2022-03-07 18:51:46 2022-03-07 19:23:31 0:31:45 0:19:33 0:12:12 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6719260 2022-03-04 03:23:07 2022-03-07 18:53:56 2022-03-07 19:16:27 0:22:31 0:10:31 0:12:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} 3
pass 6719261 2022-03-04 03:23:08 2022-03-07 18:54:37 2022-03-07 19:14:01 0:19:24 0:09:01 0:10:23 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_modprobe} 1
pass 6719262 2022-03-04 03:23:08 2022-03-07 18:54:37 2022-03-07 19:26:05 0:31:28 0:19:39 0:11:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
fail 6719263 2022-03-04 03:23:09 2022-03-07 18:56:38 2022-03-07 19:27:41 0:31:03 0:19:27 0:11:36 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-03-07T19:25:57.491690+0000 mon.a (mon.0) 318 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 6719264 2022-03-04 03:23:10 2022-03-07 18:57:28 2022-03-07 19:27:01 0:29:33 0:19:15 0:10:18 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2022-03-07T19:25:00.043815+0000 mon.a (mon.0) 254 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719265 2022-03-04 03:23:11 2022-03-07 18:57:29 2022-03-07 19:20:36 0:23:07 0:10:20 0:12:47 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_snapshot_io} 3
fail 6719266 2022-03-04 03:23:11 2022-03-07 18:59:09 2022-03-07 19:28:55 0:29:46 0:19:23 0:10:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} 3
Failure Reason:

"2022-03-07T19:27:19.733530+0000 mon.a (mon.0) 224 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6719267 2022-03-04 03:23:12 2022-03-07 18:59:30 2022-03-07 19:24:45 0:25:15 0:11:41 0:13:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6719268 2022-03-04 03:23:13 2022-03-07 18:59:40 2022-03-07 19:34:10 0:34:30 0:23:15 0:11:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_simple_big} 3
pass 6719269 2022-03-04 03:23:14 2022-03-07 19:00:00 2022-03-07 19:19:29 0:19:29 0:09:04 0:10:25 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} 1
pass 6719270 2022-03-04 03:23:15 2022-03-07 19:00:01 2022-03-07 19:40:36 0:40:35 0:29:33 0:11:02 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6719271 2022-03-04 03:23:15 2022-03-07 19:00:51 2022-03-07 19:42:39 0:41:48 0:28:33 0:13:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_fio} 3
pass 6719272 2022-03-04 03:23:16 2022-03-07 19:03:02 2022-03-07 19:37:56 0:34:54 0:22:39 0:12:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_data_pool} 3
pass 6719273 2022-03-04 03:23:17 2022-03-07 19:04:32 2022-03-07 19:32:51 0:28:19 0:13:07 0:15:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6719274 2022-03-04 03:23:18 2022-03-07 19:07:13 2022-03-07 19:38:42 0:31:29 0:19:06 0:12:23 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6719275 2022-03-04 03:23:19 2022-03-07 19:07:13 2022-03-07 19:32:24 0:25:11 0:12:09 0:13:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_fallocate} 3
pass 6719276 2022-03-04 03:23:19 2022-03-07 19:07:14 2022-03-07 19:32:50 0:25:36 0:12:28 0:13:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6719277 2022-03-04 03:23:20 2022-03-07 19:07:24 2022-03-07 19:59:18 0:51:54 0:38:36 0:13:18 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6719278 2022-03-04 03:23:21 2022-03-07 19:07:54 2022-03-07 19:35:41 0:27:47 0:15:45 0:12:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6719279 2022-03-04 03:23:22 2022-03-07 19:08:05 2022-03-07 19:29:42 0:21:37 0:10:38 0:10:59 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_parent_overlap} 1
pass 6719280 2022-03-04 03:23:22 2022-03-07 19:08:05 2022-03-07 19:39:35 0:31:30 0:16:16 0:15:14 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6719281 2022-03-04 03:23:23 2022-03-07 19:09:16 2022-03-07 20:08:44 0:59:28 0:44:30 0:14:58 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_fio} 3
pass 6719282 2022-03-04 03:23:24 2022-03-07 19:09:46 2022-03-07 19:33:52 0:24:06 0:11:27 0:12:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_namespaces} 3
pass 6719283 2022-03-04 03:23:25 2022-03-07 19:10:16 2022-03-07 20:39:14 1:28:58 1:15:11 0:13:47 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
fail 6719284 2022-03-04 03:23:26 2022-03-07 19:11:17 2022-03-07 20:07:09 0:55:52 0:42:52 0:13:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_enumerate} 3
Failure Reason:

"2022-03-07T19:54:47.797012+0000 osd.1 (osd.1) 3 : cluster [WRN] 2 slow requests (by type [ 'delayed' : 1 'started' : 1 ] most affected pool [ 'rbd' : 2 ])" in cluster log

fail 6719285 2022-03-04 03:23:26 2022-03-07 19:11:37 2022-03-07 19:43:40 0:32:03 0:20:20 0:11:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2022-03-07T19:31:32.104586+0000 mon.a (mon.0) 202 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6719286 2022-03-04 03:23:27 2022-03-07 19:12:18 2022-03-07 19:38:29 0:26:11 0:14:16 0:11:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
fail 6719287 2022-03-04 03:23:28 2022-03-07 19:12:58 2022-03-07 19:44:52 0:31:54 0:20:06 0:11:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-03-07T19:31:53.806046+0000 mon.a (mon.0) 122 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6719288 2022-03-04 03:23:29 2022-03-07 19:13:19 2022-03-07 19:34:48 0:21:29 0:10:38 0:10:51 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_read_only} 1
pass 6719289 2022-03-04 03:23:30 2022-03-07 19:13:19 2022-03-07 19:48:39 0:35:20 0:22:23 0:12:57 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6719290 2022-03-04 03:23:31 2022-03-07 19:15:09 2022-03-07 19:37:17 0:22:08 0:11:33 0:10:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6719291 2022-03-04 03:23:31 2022-03-07 19:15:20 2022-03-07 19:39:07 0:23:47 0:10:29 0:13:18 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6719292 2022-03-04 03:23:32 2022-03-07 19:16:00 2022-03-07 23:34:47 4:18:47 4:07:29 0:11:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

Command failed (workunit test rbd/concurrent.sh) on smithi173 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f735432033a131a8101d9f67b8c22b3d82a51a79 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh'

pass 6719293 2022-03-04 03:23:33 2022-03-07 19:16:31 2022-03-07 19:39:43 0:23:12 0:10:57 0:12:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6719294 2022-03-04 03:23:34 2022-03-07 19:17:51 2022-03-07 19:49:59 0:32:08 0:20:36 0:11:32 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
fail 6719295 2022-03-04 03:23:34 2022-03-07 19:18:32 2022-03-07 19:52:08 0:33:36 0:19:26 0:14:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2022-03-07T19:39:14.141425+0000 mon.a (mon.0) 157 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6719296 2022-03-04 03:23:35 2022-03-07 19:20:42 2022-03-07 19:40:20 0:19:38 0:09:51 0:09:47 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} 1
pass 6719297 2022-03-04 03:23:36 2022-03-07 19:20:43 2022-03-07 19:52:51 0:32:08 0:19:49 0:12:19 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6719298 2022-03-04 03:23:37 2022-03-07 19:21:23 2022-03-07 19:45:03 0:23:40 0:10:21 0:13:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} 3
pass 6719299 2022-03-04 03:23:38 2022-03-07 19:23:34 2022-03-07 19:54:02 0:30:28 0:16:30 0:13:58 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6719300 2022-03-04 03:23:39 2022-03-07 19:24:54 2022-03-07 19:47:43 0:22:49 0:10:17 0:12:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6719301 2022-03-04 03:23:39 2022-03-07 19:26:15 2022-03-07 19:52:32 0:26:17 0:15:27 0:10:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_unmap} 3
pass 6719302 2022-03-04 03:23:40 2022-03-07 19:38:14 2022-03-07 20:06:12 0:27:58 0:16:26 0:11:32 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
fail 6719303 2022-03-04 03:23:41 2022-03-07 19:38:34 2022-03-07 20:10:26 0:31:52 0:20:27 0:11:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} 3
Failure Reason:

"2022-03-07T20:07:59.196124+0000 mon.a (mon.0) 302 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log