Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 6600266 2022-01-07 03:20:46 2022-01-09 20:27:41 2022-01-09 20:48:05 0:20:24 0:09:25 0:10:59 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_deep_flatten} 1
pass 6600267 2022-01-07 03:20:47 2022-01-09 20:27:41 2022-01-09 20:58:22 0:30:41 0:19:42 0:10:59 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6600268 2022-01-07 03:20:47 2022-01-09 20:27:41 2022-01-09 21:08:00 0:40:19 0:28:13 0:12:06 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_fio} 3
pass 6600269 2022-01-07 03:20:48 2022-01-09 20:27:52 2022-01-09 20:59:27 0:31:35 0:18:47 0:12:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_data_pool} 3
fail 6600270 2022-01-07 03:20:49 2022-01-09 20:29:02 2022-01-10 01:10:36 4:41:34 4:29:02 0:12:32 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{prefer-crc} msgr-failures/few tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi138 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludea8oq5idv -r -- '-g auto -g blockdev -x clone'"

pass 6600271 2022-01-07 03:20:50 2022-01-09 20:29:33 2022-01-09 21:18:35 0:49:02 0:37:35 0:11:27 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/backoff thrashosds-health workloads/rbd_fio} 3
fail 6600272 2022-01-07 03:20:51 2022-01-09 20:30:03 2022-01-09 20:37:44 0:07:41 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major

pass 6600273 2022-01-07 03:20:51 2022-01-09 20:30:03 2022-01-09 20:50:50 0:20:47 0:09:20 0:11:27 smithi master krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} 1
pass 6600274 2022-01-07 03:20:52 2022-01-09 20:30:54 2022-01-09 20:54:11 0:23:17 0:11:41 0:11:36 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_exclusive_option} 3
pass 6600275 2022-01-07 03:20:53 2022-01-09 20:31:14 2022-01-09 21:03:29 0:32:15 0:19:30 0:12:45 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6600276 2022-01-07 03:20:54 2022-01-09 20:33:05 2022-01-09 20:55:51 0:22:46 0:11:31 0:11:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} 3
fail 6600277 2022-01-07 03:20:55 2022-01-09 20:33:55 2022-01-09 21:05:34 0:31:39 0:19:38 0:12:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-01-09T20:53:37.257963+0000 mon.a (mon.0) 138 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6600278 2022-01-07 03:20:55 2022-01-09 20:35:26 2022-01-09 21:25:40 0:50:14 0:37:53 0:12:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6600279 2022-01-07 03:20:56 2022-01-09 20:37:46 2022-01-09 21:01:43 0:23:57 0:13:54 0:10:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6600280 2022-01-07 03:20:57 2022-01-09 20:37:47 2022-01-09 20:57:28 0:19:41 0:09:43 0:09:58 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_discard} 1
pass 6600281 2022-01-07 03:20:58 2022-01-09 20:37:47 2022-01-09 21:06:40 0:28:53 0:17:00 0:11:53 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
fail 6600282 2022-01-07 03:20:59 2022-01-09 20:38:57 2022-01-09 21:10:14 0:31:17 0:19:20 0:11:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2022-01-09T21:08:46.120990+0000 mon.a (mon.0) 235 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600283 2022-01-07 03:21:00 2022-01-09 20:40:28 2022-01-09 22:16:45 1:36:17 1:24:47 0:11:30 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6600284 2022-01-07 03:21:00 2022-01-09 20:40:38 2022-01-09 21:29:26 0:48:48 0:37:20 0:11:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6600285 2022-01-07 03:21:01 2022-01-09 20:40:59 2022-01-09 21:07:41 0:26:42 0:15:30 0:11:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6600286 2022-01-07 03:21:02 2022-01-09 20:41:29 2022-01-09 21:06:31 0:25:02 0:12:10 0:12:52 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6600287 2022-01-07 03:21:03 2022-01-09 20:42:30 2022-01-09 21:05:47 0:23:17 0:11:27 0:11:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} 3
pass 6600288 2022-01-07 03:21:04 2022-01-09 20:43:50 2022-01-09 21:03:20 0:19:30 0:09:23 0:10:07 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} 1
pass 6600289 2022-01-07 03:21:05 2022-01-09 20:43:51 2022-01-09 21:19:10 0:35:19 0:24:04 0:11:15 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6600290 2022-01-07 03:21:05 2022-01-09 20:44:41 2022-01-09 21:09:06 0:24:25 0:11:53 0:12:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6600291 2022-01-07 03:21:06 2022-01-09 20:46:42 2022-01-09 21:09:19 0:22:37 0:10:45 0:11:52 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6600292 2022-01-07 03:21:07 2022-01-09 20:47:22 2022-01-09 21:14:50 0:27:28 0:15:38 0:11:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_concurrent} 3
fail 6600293 2022-01-07 03:21:08 2022-01-09 20:48:53 2022-01-09 21:19:20 0:30:27 0:19:34 0:10:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2022-01-09T21:18:23.857279+0000 mon.a (mon.0) 288 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600294 2022-01-07 03:21:09 2022-01-09 20:48:53 2022-01-09 21:21:29 0:32:36 0:18:53 0:13:43 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6600295 2022-01-07 03:21:09 2022-01-09 20:51:04 2022-01-09 21:15:23 0:24:19 0:10:44 0:13:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_image_read} 3
pass 6600296 2022-01-07 03:21:10 2022-01-09 20:52:54 2022-01-09 21:14:52 0:21:58 0:09:08 0:12:50 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_modprobe} 1
pass 6600297 2022-01-07 03:21:11 2022-01-09 20:53:35 2022-01-09 21:22:24 0:28:49 0:16:04 0:12:45 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6600298 2022-01-07 03:21:12 2022-01-09 20:54:15 2022-01-09 22:19:58 1:25:43 1:12:12 0:13:31 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6600299 2022-01-07 03:21:13 2022-01-09 20:55:46 2022-01-09 21:27:53 0:32:07 0:19:59 0:12:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-01-09T21:15:09.479887+0000 mon.a (mon.0) 154 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6600300 2022-01-07 03:21:14 2022-01-09 20:55:56 2022-01-09 21:27:07 0:31:11 0:17:17 0:13:54 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
fail 6600301 2022-01-07 03:21:14 2022-01-09 20:58:27 2022-01-09 21:30:55 0:32:28 0:19:30 0:12:58 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2022-01-09T21:29:42.266350+0000 mon.a (mon.0) 252 : cluster [WRN] Health check failed: 3 osds down (OSD_DOWN)" in cluster log

fail 6600302 2022-01-07 03:21:15 2022-01-09 20:59:37 2022-01-09 21:32:50 0:33:13 0:19:57 0:13:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_unmap} 3
Failure Reason:

"2022-01-09T21:29:50.830512+0000 mon.a (mon.0) 261 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600303 2022-01-07 03:21:16 2022-01-09 21:01:08 2022-01-09 21:23:45 0:22:37 0:11:45 0:10:52 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6600304 2022-01-07 03:21:17 2022-01-09 21:01:48 2022-01-09 21:37:36 0:35:48 0:23:36 0:12:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_simple_big} 3
pass 6600305 2022-01-07 03:21:18 2022-01-09 21:03:39 2022-01-09 21:23:50 0:20:11 0:09:20 0:10:51 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_msgr_segments} 1
pass 6600306 2022-01-07 03:21:18 2022-01-09 21:03:39 2022-01-09 21:39:29 0:35:50 0:23:42 0:12:08 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6600307 2022-01-07 03:21:19 2022-01-09 21:44:00 1677 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} 3
pass 6600308 2022-01-07 03:21:20 2022-01-09 21:05:50 2022-01-09 21:40:27 0:34:37 0:23:29 0:11:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} 3
pass 6600309 2022-01-07 03:21:21 2022-01-09 21:06:40 2022-01-09 21:30:38 0:23:58 0:11:22 0:12:36 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6600310 2022-01-07 03:21:22 2022-01-09 21:06:41 2022-01-09 21:38:04 0:31:23 0:19:50 0:11:33 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6600311 2022-01-07 03:21:22 2022-01-09 21:07:51 2022-01-09 21:29:58 0:22:07 0:11:56 0:10:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_fallocate} 3
pass 6600312 2022-01-07 03:21:23 2022-01-09 21:08:02 2022-01-09 21:31:37 0:23:35 0:11:24 0:12:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} 3
fail 6600313 2022-01-07 03:21:24 2022-01-09 21:09:12 2022-01-09 21:57:21 0:48:09 0:38:02 0:10:07 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
Failure Reason:

"2022-01-09T21:43:58.940379+0000 osd.1 (osd.1) 3 : cluster [WRN] slow request osd_op(client.4421.0:139884 2.4 2:2c9743e0:::rbd_data.117cbdb0d3ba.00000000000004c1:head [set-alloc-hint object_size 4194304 write_size 4194304,write 1638400~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected e20) initiated 2022-01-09T21:43:28.895303+0000 currently waiting for sub ops" in cluster log

fail 6600314 2022-01-07 03:21:25 2022-01-09 21:09:22 2022-01-09 21:39:57 0:30:35 0:19:27 0:11:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
Failure Reason:

"2022-01-09T21:28:22.880754+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6600315 2022-01-07 03:21:26 2022-01-09 21:10:23 2022-01-09 21:31:59 0:21:36 0:09:03 0:12:33 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} 1
fail 6600316 2022-01-07 03:21:26 2022-01-09 21:12:03 2022-01-09 21:46:09 0:34:06 0:18:37 0:15:29 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

"2022-01-09T21:34:23.912464+0000 mon.a (mon.0) 115 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6600317 2022-01-07 03:21:27 2022-01-09 21:14:54 2022-01-09 21:37:49 0:22:55 0:10:59 0:11:56 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_namespaces} 3
pass 6600318 2022-01-07 03:21:28 2022-01-09 21:15:25 2022-01-09 22:42:26 1:27:01 1:14:31 0:12:30 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6600319 2022-01-07 03:21:29 2022-01-09 21:16:25 2022-01-09 22:08:48 0:52:23 0:39:01 0:13:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6600320 2022-01-07 03:21:30 2022-01-09 21:18:46 2022-01-09 21:45:41 0:26:55 0:15:08 0:11:47 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
fail 6600321 2022-01-07 03:21:31 2022-01-09 21:19:16 2022-01-09 21:51:46 0:32:30 0:19:50 0:12:40 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
Failure Reason:

"2022-01-09T21:49:03.149559+0000 mon.a (mon.0) 231 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600322 2022-01-07 03:21:31 2022-01-09 21:19:26 2022-01-09 21:44:06 0:24:40 0:11:28 0:13:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netns} 3
pass 6600323 2022-01-07 03:21:32 2022-01-09 21:21:37 2022-01-09 21:42:36 0:20:59 0:09:32 0:11:27 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_read_only} 1
pass 6600324 2022-01-07 03:21:33 2022-01-09 21:22:27 2022-01-09 21:51:59 0:29:32 0:17:36 0:11:56 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6600325 2022-01-07 03:21:34 2022-01-09 21:23:48 2022-01-09 22:11:53 0:48:05 0:37:11 0:10:54 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/pggrow thrashosds-health workloads/rbd_fio} 3
pass 6600326 2022-01-07 03:21:34 2022-01-09 21:23:58 2022-01-09 21:47:42 0:23:44 0:11:21 0:12:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6600327 2022-01-07 03:21:35 2022-01-09 21:25:49 2022-01-09 21:49:50 0:24:01 0:10:48 0:13:13 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6600328 2022-01-07 03:21:36 2022-01-09 21:27:09 2022-01-10 01:39:54 4:12:45 4:01:37 0:11:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

Command failed (workunit test rbd/concurrent.sh) on smithi195 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f2313edc67106699e6ab91f50fa91928e579f7ac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh'

pass 6600329 2022-01-07 03:21:37 2022-01-09 21:28:00 2022-01-09 21:51:26 0:23:26 0:10:26 0:13:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6600330 2022-01-07 03:21:38 2022-01-09 21:29:30 2022-01-09 22:02:12 0:32:42 0:20:31 0:12:11 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
pass 6600331 2022-01-07 03:21:39 2022-01-09 21:30:01 2022-01-09 21:52:38 0:22:37 0:10:58 0:11:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} 3
pass 6600332 2022-01-07 03:21:39 2022-01-09 21:30:41 2022-01-09 21:51:20 0:20:39 0:09:54 0:10:45 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_whole_object_zeroout} 1
pass 6600333 2022-01-07 03:21:40 2022-01-09 21:31:02 2022-01-09 22:02:08 0:31:06 0:19:08 0:11:58 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6600334 2022-01-07 03:21:41 2022-01-09 21:31:42 2022-01-09 21:53:59 0:22:17 0:10:26 0:11:51 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_kernel} 3
pass 6600335 2022-01-07 03:21:42 2022-01-09 21:32:02 2022-01-09 22:00:27 0:28:25 0:16:11 0:12:14 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
fail 6600336 2022-01-07 03:21:42 2022-01-09 21:32:53 2022-01-09 22:07:41 0:34:48 0:19:40 0:15:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2022-01-09T21:56:54.263334+0000 mon.a (mon.0) 119 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log

pass 6600337 2022-01-07 03:21:43 2022-01-09 21:37:44 2022-01-09 22:04:19 0:26:35 0:15:37 0:10:58 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} 3
fail 6600338 2022-01-07 03:21:44 2022-01-09 21:37:54 2022-01-09 22:10:17 0:32:23 0:19:49 0:12:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
Failure Reason:

"2022-01-09T22:07:21.014879+0000 mon.a (mon.0) 263 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600339 2022-01-07 03:21:45 2022-01-09 21:38:04 2022-01-09 22:15:32 0:37:28 0:25:13 0:12:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_simple_big} 3
pass 6600340 2022-01-07 03:21:45 2022-01-09 21:39:35 2022-01-09 21:59:49 0:20:14 0:09:57 0:10:17 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} 1
fail 6600341 2022-01-07 03:21:46 2022-01-09 21:40:05 2022-01-09 22:10:22 0:30:17 0:18:39 0:11:38 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2022-01-09T22:08:53.406953+0000 mon.a (mon.0) 212 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600342 2022-01-07 03:21:47 2022-01-09 21:40:36 2022-01-09 22:22:54 0:42:18 0:28:18 0:14:00 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_fio} 3
pass 6600343 2022-01-07 03:21:48 2022-01-09 21:42:46 2022-01-09 22:14:31 0:31:45 0:18:48 0:12:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_data_pool} 3
pass 6600344 2022-01-07 03:21:49 2022-01-09 21:44:07 2022-01-09 22:04:03 0:19:56 0:09:37 0:10:19 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} 2
pass 6600345 2022-01-07 03:21:49 2022-01-09 21:44:07 2022-01-09 22:18:16 0:34:09 0:21:14 0:12:55 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} 3
pass 6600346 2022-01-07 03:21:50 2022-01-09 21:45:48 2022-01-09 22:08:50 0:23:02 0:11:33 0:11:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} 3
fail 6600347 2022-01-07 03:21:51 2022-01-09 21:46:18 2022-01-09 22:17:43 0:31:25 0:19:32 0:11:53 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-01-09T22:05:36.028940+0000 mon.a (mon.0) 177 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6600348 2022-01-07 03:21:52 2022-01-09 21:47:49 2022-01-09 22:14:40 0:26:51 0:11:36 0:15:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_fallocate} 3
fail 6600349 2022-01-07 03:21:52 2022-01-09 21:49:59 2022-01-09 22:23:56 0:33:57 0:19:50 0:14:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-01-09T22:21:19.102122+0000 mon.a (mon.0) 224 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600350 2022-01-07 03:21:53 2022-01-09 21:51:30 2022-01-09 22:42:09 0:50:39 0:38:39 0:12:00 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6600351 2022-01-07 03:21:54 2022-01-09 21:51:50 2022-01-09 22:17:58 0:26:08 0:14:20 0:11:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6600352 2022-01-07 03:21:55 2022-01-09 21:52:01 2022-01-09 22:15:18 0:23:17 0:10:31 0:12:46 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_discard} 1
pass 6600353 2022-01-07 03:21:56 2022-01-09 21:52:01 2022-01-09 22:22:39 0:30:38 0:18:04 0:12:34 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6600354 2022-01-07 03:21:56 2022-01-09 21:52:41 2022-01-09 23:24:21 1:31:40 1:18:58 0:12:42 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6600355 2022-01-07 03:21:57 2022-01-09 21:54:02 2022-01-09 22:27:11 0:33:09 0:19:31 0:13:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2022-01-09T22:15:22.027238+0000 mon.a (mon.0) 150 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6600356 2022-01-07 03:21:58 2022-01-09 21:57:23 2022-01-09 22:30:20 0:32:57 0:19:32 0:13:25 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2022-01-09T22:29:07.186865+0000 mon.a (mon.0) 237 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600357 2022-01-07 03:21:59 2022-01-09 22:00:34 2022-01-09 22:50:27 0:49:53 0:37:19 0:12:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6600358 2022-01-07 03:21:59 2022-01-09 22:02:14 2022-01-09 22:28:40 0:26:26 0:15:15 0:11:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6600359 2022-01-07 03:22:00 2022-01-09 22:02:15 2022-01-09 22:28:15 0:26:00 0:12:03 0:13:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6600360 2022-01-07 03:22:01 2022-01-09 22:04:05 2022-01-09 22:26:51 0:22:46 0:11:36 0:11:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netns} 3
pass 6600361 2022-01-07 03:22:02 2022-01-09 22:04:26 2022-01-09 22:23:52 0:19:26 0:09:09 0:10:17 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_huge_image} 1
pass 6600362 2022-01-07 03:22:03 2022-01-09 22:04:26 2022-01-09 22:41:34 0:37:08 0:22:39 0:14:29 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6600363 2022-01-07 03:22:03 2022-01-09 22:07:47 2022-01-09 22:34:17 0:26:30 0:11:57 0:14:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_symlinks} 3
fail 6600364 2022-01-07 03:22:04 2022-01-09 22:08:57 2022-01-09 22:39:23 0:30:26 0:19:35 0:10:51 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
Failure Reason:

"2022-01-09T22:27:44.262472+0000 mon.a (mon.0) 134 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6600365 2022-01-07 03:22:05 2022-01-09 22:08:58 2022-01-09 22:36:21 0:27:23 0:15:51 0:11:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} 3
pass 6600366 2022-01-07 03:22:06 2022-01-09 22:10:18 2022-01-09 22:32:02 0:21:44 0:10:36 0:11:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6600367 2022-01-07 03:22:07 2022-01-09 22:10:28 2022-01-09 22:41:56 0:31:28 0:19:11 0:12:17 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6600368 2022-01-07 03:22:07 2022-01-09 22:11:59 2022-01-09 22:37:04 0:25:05 0:10:52 0:14:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_image_read} 3
pass 6600369 2022-01-07 03:22:08 2022-01-09 22:14:40 2022-01-09 22:35:29 0:20:49 0:08:59 0:11:50 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} 1
pass 6600370 2022-01-07 03:22:09 2022-01-09 22:14:50 2022-01-09 22:46:38 0:31:48 0:19:44 0:12:04 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6600371 2022-01-07 03:22:10 2022-01-09 22:15:20 2022-01-09 22:37:41 0:22:21 0:10:45 0:11:36 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} 3
pass 6600372 2022-01-07 03:22:10 2022-01-09 22:15:41 2022-01-09 22:45:03 0:29:22 0:17:22 0:12:00 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6600373 2022-01-07 03:22:11 2022-01-09 22:16:51 2022-01-09 22:39:37 0:22:46 0:10:22 0:12:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6600374 2022-01-07 03:22:12 2022-01-09 22:17:52 2022-01-09 22:43:59 0:26:07 0:15:26 0:10:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_unmap} 3
pass 6600375 2022-01-07 03:22:13 2022-01-09 22:18:02 2022-01-09 22:42:48 0:24:46 0:11:50 0:12:56 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6600376 2022-01-07 03:22:14 2022-01-09 22:18:22 2022-01-09 22:56:00 0:37:38 0:25:00 0:12:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} 3
pass 6600377 2022-01-07 03:22:14 2022-01-09 22:20:03 2022-01-09 22:42:24 0:22:21 0:09:23 0:12:58 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_msgr_segments} 1
pass 6600378 2022-01-07 03:22:15 2022-01-09 22:22:44 2022-01-09 22:56:50 0:34:06 0:23:27 0:10:39 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
fail 6600379 2022-01-07 03:22:16 2022-01-09 22:23:04 2022-01-09 22:53:51 0:30:47 0:19:31 0:11:16 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

"2022-01-09T22:52:32.994743+0000 mon.a (mon.0) 215 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600380 2022-01-07 03:22:17 2022-01-09 22:23:54 2022-01-09 22:58:17 0:34:23 0:22:51 0:11:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_data_pool} 3
fail 6600381 2022-01-07 03:22:18 2022-01-09 22:24:05 2022-01-09 22:57:45 0:33:40 0:18:54 0:14:46 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy} msgr-failures/many tasks/rbd_xfstests} 4
Failure Reason:

"2022-01-09T22:46:01.686733+0000 mon.a (mon.0) 122 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6600382 2022-01-07 03:22:18 2022-01-09 22:27:16 2022-01-10 00:04:18 1:37:02 1:24:39 0:12:23 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6600383 2022-01-07 03:22:19 2022-01-09 22:28:16 2022-01-09 22:51:11 0:22:55 0:11:23 0:11:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6600384 2022-01-07 03:22:20 2022-01-09 22:28:46 2022-01-09 23:00:01 0:31:15 0:18:51 0:12:24 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6600385 2022-01-07 03:22:21 2022-01-09 22:30:27 2022-01-09 22:53:48 0:23:21 0:11:58 0:11:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} 3
pass 6600386 2022-01-07 03:22:22 2022-01-09 22:32:08 2022-01-09 22:57:36 0:25:28 0:11:38 0:13:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6600387 2022-01-07 03:22:22 2022-01-09 22:34:18 2022-01-09 23:26:44 0:52:26 0:37:55 0:14:31 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
fail 6600388 2022-01-07 03:22:23 2022-01-09 22:35:39 2022-01-09 23:08:42 0:33:03 0:19:47 0:13:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
Failure Reason:

"2022-01-09T23:05:47.253397+0000 mon.a (mon.0) 250 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600389 2022-01-07 03:22:24 2022-01-09 22:36:29 2022-01-09 22:57:38 0:21:09 0:09:09 0:12:00 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_parent_overlap} 1
pass 6600390 2022-01-07 03:22:25 2022-01-09 22:37:10 2022-01-09 23:07:47 0:30:37 0:18:33 0:12:04 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6600391 2022-01-07 03:22:26 2022-01-09 22:37:50 2022-01-09 23:01:18 0:23:28 0:11:31 0:11:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} 3
pass 6600392 2022-01-07 03:22:26 2022-01-09 22:39:31 2022-01-10 00:03:34 1:24:03 1:13:43 0:10:20 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6600393 2022-01-07 03:22:27 2022-01-09 22:39:41 2022-01-09 23:33:56 0:54:15 0:41:03 0:13:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6600394 2022-01-07 03:22:28 2022-01-09 22:41:42 2022-01-09 23:08:03 0:26:21 0:15:18 0:11:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6600395 2022-01-07 03:22:29 2022-01-09 22:42:02 2022-01-09 23:08:41 0:26:39 0:13:30 0:13:09 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6600396 2022-01-07 03:22:30 2022-01-09 22:42:12 2022-01-09 23:04:30 0:22:18 0:11:32 0:10:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} 3
pass 6600397 2022-01-07 03:22:30 2022-01-09 22:42:33 2022-01-09 23:03:52 0:21:19 0:09:26 0:11:53 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} 1
pass 6600398 2022-01-07 03:22:31 2022-01-09 22:42:33 2022-01-09 23:21:20 0:38:47 0:26:06 0:12:41 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6600399 2022-01-07 03:22:32 2022-01-09 22:42:54 2022-01-09 23:06:14 0:23:20 0:11:35 0:11:45 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6600400 2022-01-07 03:22:33 2022-01-09 22:44:04 2022-01-09 23:07:18 0:23:14 0:10:44 0:12:30 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
dead 6600401 2022-01-07 03:22:34 2022-01-09 22:45:04 2022-01-10 01:26:13 2:41:09 2:22:15 0:18:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

SSH connection to smithi145 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

pass 6600402 2022-01-07 03:22:34 2022-01-09 22:46:45 2022-01-09 23:12:45 0:26:00 0:10:48 0:15:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6600403 2022-01-07 03:22:35 2022-01-09 22:50:36 2022-01-09 23:23:17 0:32:41 0:21:15 0:11:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
pass 6600404 2022-01-07 03:22:36 2022-01-09 22:51:16 2022-01-09 23:15:41 0:24:25 0:10:44 0:13:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_image_read} 3
pass 6600405 2022-01-07 03:22:37 2022-01-09 22:53:57 2022-01-09 23:13:38 0:19:41 0:09:50 0:09:51 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_whole_object_zeroout} 1
pass 6600406 2022-01-07 03:22:38 2022-01-09 22:53:57 2022-01-09 23:19:54 0:25:57 0:15:02 0:10:55 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6600407 2022-01-07 03:22:38 2022-01-09 22:53:58 2022-01-09 23:44:11 0:50:13 0:37:38 0:12:35 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} 3
pass 6600408 2022-01-07 03:22:39 2022-01-09 22:56:08 2022-01-09 23:18:55 0:22:47 0:10:10 0:12:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_kernel} 3
pass 6600409 2022-01-07 03:22:40 2022-01-09 22:56:59 2022-01-09 23:26:09 0:29:10 0:15:53 0:13:17 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6600410 2022-01-07 03:22:41 2022-01-09 22:57:39 2022-01-09 23:20:06 0:22:27 0:10:27 0:12:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6600411 2022-01-07 03:22:42 2022-01-09 22:57:50 2022-01-09 23:27:08 0:29:18 0:15:58 0:13:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_unmap} 3
fail 6600412 2022-01-07 03:22:42 2022-01-09 22:57:50 2022-01-09 23:28:46 0:30:56 0:19:27 0:11:29 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
Failure Reason:

"2022-01-09T23:27:30.983925+0000 mon.a (mon.0) 336 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6600413 2022-01-07 03:22:43 2022-01-09 22:58:20 2022-01-09 23:35:52 0:37:32 0:24:44 0:12:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_simple_big} 3
pass 6600414 2022-01-07 03:22:44 2022-01-09 23:00:11 2022-01-09 23:21:45 0:21:34 0:09:42 0:11:52 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_deep_flatten} 1
pass 6600415 2022-01-07 03:22:45 2022-01-09 23:01:21 2022-01-09 23:39:59 0:38:38 0:24:23 0:14:15 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6600416 2022-01-07 03:22:46 2022-01-09 23:04:02 2022-01-09 23:44:29 0:40:27 0:28:38 0:11:49 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} 3
pass 6600417 2022-01-07 03:22:46 2022-01-09 23:04:32 2022-01-09 23:36:33 0:32:01 0:19:15 0:12:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} 3
pass 6600418 2022-01-07 03:22:47 2022-01-09 23:06:23 2022-01-09 23:27:19 0:20:56 0:09:27 0:11:29 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} 2
pass 6600419 2022-01-07 03:22:48 2022-01-09 23:07:23 2022-01-09 23:40:08 0:32:45 0:21:23 0:11:22 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} 3
pass 6600420 2022-01-07 03:22:49 2022-01-09 23:07:54 2022-01-09 23:30:14 0:22:20 0:11:58 0:10:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_exclusive_option} 3
pass 6600421 2022-01-07 03:22:50 2022-01-09 23:08:04 2022-01-09 23:38:44 0:30:40 0:19:26 0:11:14 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6600422 2022-01-07 03:22:50 2022-01-09 23:08:44 2022-01-09 23:33:27 0:24:43 0:11:56 0:12:47 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_fallocate} 3
fail 6600423 2022-01-07 03:22:51 2022-01-09 23:08:45 2022-01-09 23:42:51 0:34:06 0:19:17 0:14:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-01-09T23:31:09.383549+0000 mon.a (mon.0) 185 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6600424 2022-01-07 03:22:52 2022-01-09 23:12:56 2022-01-10 00:03:47 0:50:51 0:37:12 0:13:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6600425 2022-01-07 03:22:53 2022-01-09 23:15:46 2022-01-09 23:43:07 0:27:21 0:13:54 0:13:27 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6600426 2022-01-07 03:22:54 2022-01-09 23:18:57 2022-01-09 23:38:38 0:19:41 0:10:04 0:09:37 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} 1
pass 6600427 2022-01-07 03:22:54 2022-01-09 23:18:57 2022-01-09 23:48:08 0:29:11 0:17:50 0:11:21 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6600428 2022-01-07 03:22:55 2022-01-09 23:19:58 2022-01-09 23:44:34 0:24:36 0:12:02 0:12:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_namespaces} 3
pass 6600429 2022-01-07 03:22:56 2022-01-09 23:20:08 2022-01-10 00:56:28 1:36:20 1:22:25 0:13:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6600430 2022-01-07 03:22:57 2022-01-09 23:21:29 2022-01-10 00:11:34 0:50:05 0:37:04 0:13:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6600431 2022-01-07 03:22:57 2022-01-09 23:23:19 2022-01-09 23:50:26 0:27:07 0:15:18 0:11:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6600432 2022-01-07 03:22:58 2022-01-09 23:24:30 2022-01-09 23:50:36 0:26:06 0:12:20 0:13:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6600433 2022-01-07 03:22:59 2022-01-09 23:26:10 2022-01-09 23:49:17 0:23:07 0:11:27 0:11:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netns} 3
pass 6600434 2022-01-07 03:23:00 2022-01-09 23:26:51 2022-01-09 23:47:16 0:20:25 0:09:29 0:10:56 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_huge_image} 1
pass 6600435 2022-01-07 03:23:01 2022-01-09 23:26:51 2022-01-10 00:00:30 0:33:39 0:20:58 0:12:41 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6600436 2022-01-07 03:23:01 2022-01-09 23:27:11 2022-01-10 00:57:24 1:30:13 1:20:00 0:10:13 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6600437 2022-01-07 03:23:02 2022-01-09 23:27:22 2022-01-09 23:51:09 0:23:47 0:11:45 0:12:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6600438 2022-01-07 03:23:03 2022-01-09 23:28:52 2022-01-09 23:52:19 0:23:27 0:11:07 0:12:20 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6600439 2022-01-07 03:23:04 2022-01-09 23:30:23 2022-01-10 00:00:15 0:29:52 0:15:51 0:14:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_concurrent} 3
pass 6600440 2022-01-07 03:23:05 2022-01-09 23:33:34 2022-01-09 23:56:29 0:22:55 0:10:53 0:12:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_huge_tickets} 3
fail 6600441 2022-01-07 03:23:05 2022-01-09 23:34:04 2022-01-10 00:05:37 0:31:33 0:19:35 0:11:58 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2022-01-10T00:04:04.152909+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600442 2022-01-07 03:23:06 2022-01-09 23:35:55 2022-01-09 23:58:45 0:22:50 0:10:44 0:12:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} 3
pass 6600443 2022-01-07 03:23:07 2022-01-09 23:36:35 2022-01-09 23:58:26 0:21:51 0:09:10 0:12:41 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_modprobe} 1
pass 6600444 2022-01-07 03:23:08 2022-01-09 23:38:46 2022-01-10 00:10:54 0:32:08 0:20:14 0:11:54 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
fail 6600445 2022-01-07 03:23:08 2022-01-09 23:38:46 2022-01-10 00:10:08 0:31:22 0:19:15 0:12:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-01-10T00:08:28.549496+0000 mon.a (mon.0) 252 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600446 2022-01-07 03:23:09 2022-01-09 23:40:07 2022-01-10 00:08:28 0:28:21 0:16:53 0:11:28 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6600447 2022-01-07 03:23:10 2022-01-09 23:40:17 2022-01-10 00:04:56 0:24:39 0:10:28 0:14:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6600448 2022-01-07 03:23:11 2022-01-09 23:42:58 2022-01-10 00:08:54 0:25:56 0:15:30 0:10:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} 3
pass 6600449 2022-01-07 03:23:11 2022-01-09 23:43:08 2022-01-10 00:06:19 0:23:11 0:11:42 0:11:29 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6600450 2022-01-07 03:23:12 2022-01-09 23:44:18 2022-01-10 00:19:03 0:34:45 0:23:38 0:11:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_simple_big} 3
pass 6600451 2022-01-07 03:23:13 2022-01-09 23:44:39 2022-01-10 00:04:50 0:20:11 0:09:11 0:11:00 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} 1
pass 6600452 2022-01-07 03:23:14 2022-01-09 23:44:39 2022-01-10 00:23:20 0:38:41 0:26:09 0:12:32 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6600453 2022-01-07 03:23:15 2022-01-09 23:47:20 2022-01-10 00:27:57 0:40:37 0:28:34 0:12:03 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_fio} 3
fail 6600454 2022-01-07 03:23:15 2022-01-09 23:48:10 2022-01-10 00:20:01 0:31:51 0:19:19 0:12:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-01-10T00:08:22.298141+0000 mon.a (mon.0) 158 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6600455 2022-01-07 03:23:16 2022-01-09 23:49:21 2022-01-10 00:12:28 0:23:07 0:11:18 0:11:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6600456 2022-01-07 03:23:17 2022-01-09 23:50:31 2022-01-10 00:19:11 0:28:40 0:18:20 0:10:20 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6600457 2022-01-07 03:23:18 2022-01-09 23:50:42 2022-01-10 00:15:13 0:24:31 0:12:03 0:12:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_fallocate} 3
pass 6600458 2022-01-07 03:23:19 2022-01-09 23:51:12 2022-01-10 00:14:19 0:23:07 0:11:49 0:11:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6600459 2022-01-07 03:23:19 2022-01-09 23:52:22 2022-01-10 00:44:32 0:52:10 0:37:36 0:14:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6600460 2022-01-07 03:23:20 2022-01-09 23:56:33 2022-01-10 00:25:01 0:28:28 0:14:17 0:14:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6600461 2022-01-07 03:23:21 2022-01-09 23:58:34 2022-01-10 00:19:07 0:20:33 0:09:17 0:11:16 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_parent_overlap} 1
pass 6600462 2022-01-07 03:23:22 2022-01-09 23:58:54 2022-01-10 00:29:07 0:30:13 0:16:31 0:13:42 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6600463 2022-01-07 03:23:23 2022-01-10 00:00:25 2022-01-10 00:49:55 0:49:30 0:36:37 0:12:53 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/upmap thrashosds-health workloads/rbd_fio} 3
pass 6600464 2022-01-07 03:23:23 2022-01-10 00:00:35 2022-01-10 00:25:30 0:24:55 0:11:07 0:13:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_namespaces} 3
pass 6600465 2022-01-07 03:23:24 2022-01-10 00:03:36 2022-01-10 01:30:12 1:26:36 1:14:25 0:12:11 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
fail 6600466 2022-01-07 03:23:25 2022-01-10 00:03:56 2022-01-10 00:36:29 0:32:33 0:19:38 0:12:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_enumerate} 3
Failure Reason:

"2022-01-10T00:33:42.241955+0000 mon.a (mon.0) 264 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600467 2022-01-07 03:23:26 2022-01-10 00:04:27 2022-01-10 00:31:05 0:26:38 0:15:12 0:11:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6600468 2022-01-07 03:23:27 2022-01-10 00:04:57 2022-01-10 00:28:44 0:23:47 0:12:51 0:10:56 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6600469 2022-01-07 03:23:27 2022-01-10 00:04:57 2022-01-10 00:27:13 0:22:16 0:11:38 0:10:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netns} 3
pass 6600470 2022-01-07 03:23:28 2022-01-10 00:05:38 2022-01-10 00:26:35 0:20:57 0:09:23 0:11:34 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_read_only} 1
pass 6600471 2022-01-07 03:23:29 2022-01-10 00:06:28 2022-01-10 00:42:37 0:36:09 0:22:20 0:13:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6600472 2022-01-07 03:23:30 2022-01-10 00:08:29 2022-01-10 00:30:55 0:22:26 0:11:36 0:10:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6600473 2022-01-07 03:23:30 2022-01-10 00:08:59 2022-01-10 00:31:58 0:22:59 0:10:59 0:12:00 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6600474 2022-01-07 03:23:31 2022-01-10 00:10:10 2022-01-10 00:41:13 0:31:03 0:19:29 0:11:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

"2022-01-10T00:40:05.525519+0000 mon.a (mon.0) 232 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600475 2022-01-07 03:23:32 2022-01-10 00:11:00 2022-01-10 00:33:37 0:22:37 0:10:23 0:12:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6600476 2022-01-07 03:23:33 2022-01-10 00:11:41 2022-01-10 00:44:42 0:33:01 0:19:33 0:13:28 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2022-01-10T00:31:15.095898+0000 mon.a (mon.0) 181 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6600477 2022-01-07 03:23:34 2022-01-10 00:12:31 2022-01-10 00:44:32 0:32:01 0:19:26 0:12:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2022-01-10T00:43:20.679155+0000 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600478 2022-01-07 03:23:34 2022-01-10 00:14:22 2022-01-10 00:34:03 0:19:41 0:09:33 0:10:08 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} 1
fail 6600479 2022-01-07 03:23:35 2022-01-10 00:14:22 2022-01-10 00:45:44 0:31:22 0:19:40 0:11:42 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

"2022-01-10T00:44:28.562099+0000 mon.a (mon.0) 280 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600480 2022-01-07 03:23:36 2022-01-10 00:15:23 2022-01-10 00:41:38 0:26:15 0:10:34 0:15:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} 3
pass 6600481 2022-01-07 03:23:37 2022-01-10 00:19:14 2022-01-10 00:47:39 0:28:25 0:16:36 0:11:49 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6600482 2022-01-07 03:23:38 2022-01-10 00:19:14 2022-01-10 00:42:50 0:23:36 0:10:21 0:13:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6600483 2022-01-07 03:23:38 2022-01-10 00:20:04 2022-01-10 00:49:50 0:29:46 0:15:31 0:14:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_unmap} 3
fail 6600484 2022-01-07 03:23:39 2022-01-10 00:23:25 2022-01-10 00:57:07 0:33:42 0:19:40 0:14:02 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
Failure Reason:

"2022-01-10T00:54:04.243270+0000 mon.a (mon.0) 265 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6600485 2022-01-07 03:23:40 2022-01-10 00:25:06 2022-01-10 01:05:35 0:40:29 0:28:50 0:11:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} 3