Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 6654043 2022-02-02 01:08:20 2022-02-02 01:45:49 2022-02-02 02:08:09 0:22:20 0:11:10 0:11:10 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_deep_flatten} 1
pass 6654044 2022-02-02 01:08:21 2022-02-02 01:46:19 2022-02-02 02:19:10 0:32:51 0:19:54 0:12:57 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6654045 2022-02-02 01:08:22 2022-02-02 01:46:20 2022-02-02 02:26:18 0:39:58 0:28:56 0:11:02 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_fio} 3
pass 6654046 2022-02-02 01:08:23 2022-02-02 01:46:40 2022-02-02 02:16:25 0:29:45 0:19:48 0:09:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_data_pool} 3
pass 6654047 2022-02-02 01:08:24 2022-02-02 01:46:50 2022-02-02 02:11:22 0:24:32 0:12:03 0:12:29 smithi master krbd/rxbounce/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_rxbounce} 3
fail 6654048 2022-02-02 01:08:25 2022-02-02 01:48:01 2022-02-02 06:21:12 4:33:11 4:19:56 0:13:15 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy-rxbounce} msgr-failures/few tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi151 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excluderkh5_jk8 -r -- '-g auto -g blockdev -x clone'"

pass 6654049 2022-02-02 01:08:26 2022-02-02 01:48:51 2022-02-02 02:38:22 0:49:31 0:39:02 0:10:29 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/backoff thrashosds-health workloads/rbd_fio} 3
fail 6654050 2022-02-02 01:08:27 2022-02-02 01:48:52 2022-02-02 01:56:47 0:07:55 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major

pass 6654051 2022-02-02 01:08:28 2022-02-02 01:49:02 2022-02-02 02:08:12 0:19:10 0:09:54 0:09:16 smithi master krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} 1
fail 6654052 2022-02-02 01:08:29 2022-02-02 01:49:02 2022-02-02 02:21:28 0:32:26 0:19:56 0:12:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2022-02-02T02:18:34.358215+0000 mon.a (mon.0) 223 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6654053 2022-02-02 01:08:30 2022-02-02 01:50:03 2022-02-02 02:23:00 0:32:57 0:19:55 0:13:02 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-02-02T02:19:57.170919+0000 mon.a (mon.0) 265 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654054 2022-02-02 01:08:31 2022-02-02 01:51:23 2022-02-02 02:17:55 0:26:32 0:11:42 0:14:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} 3
fail 6654055 2022-02-02 01:08:32 2022-02-02 01:54:14 2022-02-02 02:26:30 0:32:16 0:20:01 0:12:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-02-02T02:25:26.285669+0000 mon.a (mon.0) 262 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654056 2022-02-02 01:08:33 2022-02-02 01:54:14 2022-02-02 02:44:29 0:50:15 0:37:46 0:12:29 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6654057 2022-02-02 01:08:34 2022-02-02 01:54:35 2022-02-02 02:20:48 0:26:13 0:14:21 0:11:52 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6654058 2022-02-02 01:08:35 2022-02-02 01:54:45 2022-02-02 02:17:41 0:22:56 0:10:30 0:12:26 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_discard} 1
pass 6654059 2022-02-02 01:08:36 2022-02-02 01:56:06 2022-02-02 02:26:35 0:30:29 0:18:35 0:11:54 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
fail 6654060 2022-02-02 01:08:38 2022-02-02 01:56:56 2022-02-02 02:29:05 0:32:09 0:19:54 0:12:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2022-02-02T02:16:35.537571+0000 mon.a (mon.0) 120 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6654061 2022-02-02 01:08:39 2022-02-02 01:56:57 2022-02-02 03:31:01 1:34:04 1:21:32 0:12:32 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
fail 6654062 2022-02-02 01:08:40 2022-02-02 01:57:07 2022-02-02 02:29:26 0:32:19 0:19:59 0:12:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_enumerate} 3
Failure Reason:

"2022-02-02T02:27:01.585317+0000 mon.a (mon.0) 228 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654063 2022-02-02 01:08:41 2022-02-02 01:57:48 2022-02-02 02:25:31 0:27:43 0:15:30 0:12:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6654064 2022-02-02 01:08:42 2022-02-02 01:58:58 2022-02-02 02:23:59 0:25:01 0:12:25 0:12:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6654065 2022-02-02 01:08:43 2022-02-02 01:59:18 2022-02-02 02:24:45 0:25:27 0:11:22 0:14:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} 3
pass 6654066 2022-02-02 01:08:44 2022-02-02 02:03:09 2022-02-02 02:22:31 0:19:22 0:09:13 0:10:09 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} 1
fail 6654067 2022-02-02 01:08:45 2022-02-02 02:03:10 2022-02-02 02:34:36 0:31:26 0:20:09 0:11:17 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2022-02-02T02:33:02.431445+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654068 2022-02-02 01:08:46 2022-02-02 02:05:00 2022-02-02 02:27:24 0:22:24 0:12:22 0:10:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks} 3
fail 6654069 2022-02-02 01:08:47 2022-02-02 02:05:51 2022-02-02 02:38:22 0:32:31 0:19:47 0:12:44 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
Failure Reason:

"2022-02-02T02:35:34.854487+0000 mon.a (mon.0) 221 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654070 2022-02-02 01:08:48 2022-02-02 02:06:21 2022-02-02 02:35:23 0:29:02 0:16:09 0:12:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_concurrent} 3
fail 6654071 2022-02-02 01:08:49 2022-02-02 02:07:01 2022-02-02 02:40:01 0:33:00 0:20:13 0:12:47 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2022-02-02T02:37:39.411622+0000 mon.a (mon.0) 201 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654072 2022-02-02 01:08:50 2022-02-02 02:08:12 2022-02-02 02:39:50 0:31:38 0:19:54 0:11:44 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6654073 2022-02-02 01:08:51 2022-02-02 02:08:12 2022-02-02 02:30:18 0:22:06 0:11:21 0:10:45 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_image_read} 3
pass 6654074 2022-02-02 01:08:52 2022-02-02 02:08:43 2022-02-02 02:28:17 0:19:34 0:09:13 0:10:21 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_modprobe} 1
pass 6654075 2022-02-02 01:08:53 2022-02-02 02:08:43 2022-02-02 02:38:34 0:29:51 0:16:09 0:13:42 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6654076 2022-02-02 01:08:54 2022-02-02 02:10:24 2022-02-02 03:36:12 1:25:48 1:14:30 0:11:18 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6654077 2022-02-02 01:08:55 2022-02-02 02:10:24 2022-02-02 02:40:47 0:30:23 0:19:43 0:10:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-02-02T02:39:35.316221+0000 mon.a (mon.0) 218 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6654078 2022-02-02 01:08:56 2022-02-02 02:11:25 2022-02-02 02:44:24 0:32:59 0:19:44 0:13:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2022-02-02T02:41:51.669142+0000 mon.a (mon.0) 234 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654079 2022-02-02 01:08:57 2022-02-02 02:12:15 2022-02-02 02:37:55 0:25:40 0:10:42 0:14:58 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6654080 2022-02-02 01:08:58 2022-02-02 02:16:26 2022-02-02 02:45:21 0:28:55 0:16:02 0:12:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_map_unmap} 3
pass 6654081 2022-02-02 01:08:59 2022-02-02 02:16:36 2022-02-02 02:41:23 0:24:47 0:12:11 0:12:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6654082 2022-02-02 01:09:00 2022-02-02 02:17:47 2022-02-02 02:51:41 0:33:54 0:23:14 0:10:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_simple_big} 3
pass 6654083 2022-02-02 01:09:01 2022-02-02 02:17:57 2022-02-02 02:39:21 0:21:24 0:09:43 0:11:41 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_msgr_segments} 1
pass 6654084 2022-02-02 01:09:02 2022-02-02 02:17:57 2022-02-02 02:56:04 0:38:07 0:25:16 0:12:51 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6654085 2022-02-02 01:09:04 2022-02-02 02:19:18 2022-02-02 03:00:57 0:41:39 0:29:18 0:12:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} 3
fail 6654086 2022-02-02 01:09:05 2022-02-02 02:20:48 2022-02-02 02:53:18 0:32:30 0:20:06 0:12:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-02-02T02:50:40.296071+0000 mon.a (mon.0) 294 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6654087 2022-02-02 01:09:06 2022-02-02 02:21:39 2022-02-02 02:44:36 0:22:57 0:11:46 0:11:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6654088 2022-02-02 01:09:07 2022-02-02 02:23:09 2022-02-02 02:54:46 0:31:37 0:19:01 0:12:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
fail 6654089 2022-02-02 01:09:08 2022-02-02 02:24:00 2022-02-02 02:54:40 0:30:40 0:19:23 0:11:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_fallocate} 3
Failure Reason:

"2022-02-02T02:53:12.728241+0000 mon.a (mon.0) 252 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654090 2022-02-02 01:09:09 2022-02-02 02:24:50 2022-02-02 02:50:13 0:25:23 0:11:59 0:13:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6654091 2022-02-02 01:09:10 2022-02-02 02:25:41 2022-02-02 03:17:58 0:52:17 0:39:02 0:13:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6654092 2022-02-02 01:09:11 2022-02-02 02:26:01 2022-02-02 02:54:01 0:28:00 0:15:16 0:12:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6654093 2022-02-02 01:09:12 2022-02-02 02:26:22 2022-02-02 02:47:44 0:21:22 0:09:43 0:11:39 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} 1
dead 6654094 2022-02-02 01:09:13 2022-02-02 02:26:22 2022-02-02 02:44:06 0:17:44 0:04:50 0:12:54 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

{'Failure object was': {'smithi107.front.sepia.ceph.com': {'stdout': '', 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6198 (apt-get)\\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\\n', 'rc': 100, 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': True, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': ['ceph', 'ceph-common', 'libcephfs1', 'radosgw', 'python-ceph', 'python-rados', 'python-cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph-fs-common', 'openmpi-common'], 'autoclean': False, 'install_recommends': 'None', 'name': ['ceph', 'ceph-common', 'libcephfs1', 'radosgw', 'python-ceph', 'python-rados', 'python-cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph-fs-common', 'openmpi-common'], 'purge': False, 'allow_unauthenticated': False, 'state': 'absent', 'upgrade': 'None', 'update_cache': 'None', 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, 'msg': "'apt-get remove 'librbd1' 'librados2'' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6198 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n", 'stdout_lines': [], 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6198 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'invocation')"}

pass 6654095 2022-02-02 01:09:14 2022-02-02 02:26:32 2022-02-02 02:50:17 0:23:45 0:12:25 0:11:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_namespaces} 3
pass 6654096 2022-02-02 01:09:15 2022-02-02 02:26:43 2022-02-02 03:53:13 1:26:30 1:13:55 0:12:35 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
fail 6654097 2022-02-02 01:09:16 2022-02-02 02:27:33 2022-02-02 03:01:09 0:33:36 0:20:01 0:13:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} 3
Failure Reason:

"2022-02-02T02:47:48.631339+0000 mon.a (mon.0) 116 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6654098 2022-02-02 01:09:17 2022-02-02 02:29:14 2022-02-02 02:55:10 0:25:56 0:15:55 0:10:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6654099 2022-02-02 01:09:18 2022-02-02 02:29:34 2022-02-02 02:53:57 0:24:23 0:13:46 0:10:37 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
fail 6654100 2022-02-02 01:09:19 2022-02-02 02:30:24 2022-02-02 03:04:25 0:34:01 0:19:58 0:14:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-02-02T03:02:57.435239+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654101 2022-02-02 01:09:20 2022-02-02 02:34:45 2022-02-02 02:54:20 0:19:35 0:10:00 0:09:35 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_read_only} 1
pass 6654102 2022-02-02 01:09:21 2022-02-02 02:34:46 2022-02-02 03:05:43 0:30:57 0:19:04 0:11:53 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6654103 2022-02-02 01:09:22 2022-02-02 02:35:26 2022-02-02 03:27:34 0:52:08 0:38:02 0:14:06 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/pggrow thrashosds-health workloads/rbd_fio} 3
pass 6654104 2022-02-02 01:09:23 2022-02-02 02:37:57 2022-02-02 03:02:11 0:24:14 0:11:42 0:12:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6654105 2022-02-02 01:09:24 2022-02-02 02:38:27 2022-02-02 03:02:00 0:23:33 0:11:07 0:12:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
dead 6654106 2022-02-02 01:09:25 2022-02-02 02:38:27 2022-02-02 05:37:18 2:58:51 2:45:46 0:13:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

SSH connection to smithi185 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

pass 6654107 2022-02-02 01:09:26 2022-02-02 02:38:38 2022-02-02 03:03:30 0:24:52 0:11:27 0:13:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6654108 2022-02-02 01:09:27 2022-02-02 02:39:58 2022-02-02 03:13:48 0:33:50 0:21:29 0:12:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
fail 6654109 2022-02-02 01:09:28 2022-02-02 02:40:09 2022-02-02 03:12:06 0:31:57 0:20:33 0:11:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2022-02-02T03:10:18.341306+0000 mon.a (mon.0) 280 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6654110 2022-02-02 01:09:29 2022-02-02 02:40:49 2022-02-02 03:02:23 0:21:34 0:10:56 0:10:38 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_whole_object_zeroout} 1
pass 6654111 2022-02-02 01:09:30 2022-02-02 02:40:50 2022-02-02 03:11:22 0:30:32 0:19:09 0:11:23 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6654112 2022-02-02 01:09:31 2022-02-02 02:41:30 2022-02-02 03:05:49 0:24:19 0:11:19 0:13:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_kernel} 3
pass 6654113 2022-02-02 01:09:32 2022-02-02 02:44:11 2022-02-02 03:14:19 0:30:08 0:17:16 0:12:52 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6654114 2022-02-02 01:09:33 2022-02-02 02:44:31 2022-02-02 03:08:26 0:23:55 0:11:18 0:12:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6654115 2022-02-02 01:09:34 2022-02-02 02:44:32 2022-02-02 03:12:21 0:27:49 0:16:31 0:11:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} 3
pass 6654116 2022-02-02 01:09:35 2022-02-02 02:44:42 2022-02-02 03:12:07 0:27:25 0:13:15 0:14:10 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6654117 2022-02-02 01:09:36 2022-02-02 02:45:23 2022-02-02 03:26:00 0:40:37 0:25:45 0:14:52 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_simple_big} 3
pass 6654118 2022-02-02 01:09:38 2022-02-02 02:50:24 2022-02-02 03:09:49 0:19:25 0:09:50 0:09:35 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} 1
pass 6654119 2022-02-02 01:09:39 2022-02-02 02:50:24 2022-02-02 03:27:00 0:36:36 0:23:38 0:12:58 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6654120 2022-02-02 01:09:40 2022-02-02 02:50:24 2022-02-02 03:31:28 0:41:04 0:28:39 0:12:25 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_fio} 3
pass 6654121 2022-02-02 01:09:41 2022-02-02 02:51:45 2022-02-02 03:20:58 0:29:13 0:19:21 0:09:52 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_data_pool} 3
pass 6654122 2022-02-02 01:09:42 2022-02-02 02:51:45 2022-02-02 03:14:42 0:22:57 0:09:35 0:13:22 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} 2
pass 6654123 2022-02-02 01:09:43 2022-02-02 02:53:26 2022-02-02 03:27:46 0:34:20 0:21:32 0:12:48 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} 3
pass 6654124 2022-02-02 01:09:44 2022-02-02 02:54:06 2022-02-02 03:19:37 0:25:31 0:12:13 0:13:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} 3
pass 6654125 2022-02-02 01:09:45 2022-02-02 02:54:07 2022-02-02 03:28:15 0:34:08 0:20:11 0:13:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6654126 2022-02-02 01:09:46 2022-02-02 02:54:27 2022-02-02 03:19:37 0:25:10 0:12:14 0:12:56 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_fallocate} 3
pass 6654127 2022-02-02 01:09:47 2022-02-02 02:54:47 2022-02-02 03:20:31 0:25:44 0:12:26 0:13:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6654128 2022-02-02 01:09:48 2022-02-02 02:54:48 2022-02-02 03:47:05 0:52:17 0:38:10 0:14:07 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6654129 2022-02-02 01:09:49 2022-02-02 02:55:18 2022-02-02 03:22:54 0:27:36 0:14:24 0:13:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6654130 2022-02-02 01:09:50 2022-02-02 02:56:09 2022-02-02 03:22:27 0:26:18 0:10:11 0:16:07 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_discard} 1
pass 6654131 2022-02-02 01:09:51 2022-02-02 03:01:00 2022-02-02 03:31:07 0:30:07 0:17:54 0:12:13 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6654132 2022-02-02 01:09:52 2022-02-02 03:01:10 2022-02-02 04:35:45 1:34:35 1:22:14 0:12:21 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6654133 2022-02-02 01:09:54 2022-02-02 03:02:10 2022-02-02 03:28:24 0:26:14 0:13:52 0:12:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_namespaces} 3
fail 6654134 2022-02-02 01:09:55 2022-02-02 03:02:21 2022-02-02 03:36:44 0:34:23 0:22:23 0:12:00 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2022-02-02T03:35:03.929382+0000 mon.a (mon.0) 205 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654135 2022-02-02 01:09:56 2022-02-02 03:02:31 2022-02-02 03:55:13 0:52:42 0:40:42 0:12:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6654136 2022-02-02 01:09:57 2022-02-02 03:03:32 2022-02-02 03:34:08 0:30:36 0:19:43 0:10:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6654137 2022-02-02 01:09:58 2022-02-02 03:04:32 2022-02-02 03:31:39 0:27:07 0:15:23 0:11:44 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6654138 2022-02-02 01:09:59 2022-02-02 03:05:53 2022-02-02 03:32:06 0:26:13 0:15:05 0:11:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_netns} 3
pass 6654139 2022-02-02 01:10:00 2022-02-02 03:05:53 2022-02-02 03:30:34 0:24:41 0:09:48 0:14:53 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_huge_image} 1
pass 6654140 2022-02-02 01:10:01 2022-02-02 03:08:34 2022-02-02 03:43:29 0:34:55 0:22:19 0:12:36 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6654141 2022-02-02 01:10:02 2022-02-02 03:09:54 2022-02-02 03:41:13 0:31:19 0:16:45 0:14:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_symlinks} 3
fail 6654142 2022-02-02 01:10:03 2022-02-02 03:11:25 2022-02-02 03:47:03 0:35:38 0:21:21 0:14:17 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
Failure Reason:

"2022-02-02T03:35:00.618922+0000 mon.a (mon.0) 124 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6654143 2022-02-02 01:10:04 2022-02-02 03:12:15 2022-02-02 03:45:31 0:33:16 0:20:48 0:12:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} 3
pass 6654144 2022-02-02 01:10:05 2022-02-02 03:12:15 2022-02-02 03:40:08 0:27:53 0:16:28 0:11:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6654145 2022-02-02 01:10:07 2022-02-02 03:12:26 2022-02-02 03:49:53 0:37:27 0:23:08 0:14:19 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6654146 2022-02-02 01:10:08 2022-02-02 03:13:56 2022-02-02 03:40:10 0:26:14 0:14:19 0:11:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_image_read} 3
pass 6654147 2022-02-02 01:10:09 2022-02-02 03:14:27 2022-02-02 03:36:10 0:21:43 0:10:52 0:10:51 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} 1
fail 6654148 2022-02-02 01:10:10 2022-02-02 03:14:47 2022-02-02 03:52:08 0:37:21 0:21:47 0:15:34 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

"2022-02-02T03:39:06.834174+0000 mon.a (mon.0) 133 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6654149 2022-02-02 01:10:11 2022-02-02 03:18:08 2022-02-02 03:55:01 0:36:53 0:23:25 0:13:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2022-02-02T03:53:06.483543+0000 mon.a (mon.0) 216 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6654150 2022-02-02 01:10:12 2022-02-02 03:18:48 2022-02-02 03:54:11 0:35:23 0:24:59 0:10:24 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2022-02-02T03:42:08.410748+0000 mon.a (mon.0) 111 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6654151 2022-02-02 01:10:13 2022-02-02 03:18:49 2022-02-02 03:46:32 0:27:43 0:15:27 0:12:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6654152 2022-02-02 01:10:14 2022-02-02 03:18:49 2022-02-02 03:52:55 0:34:06 0:21:13 0:12:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_unmap} 3
pass 6654153 2022-02-02 01:10:15 2022-02-02 03:19:19 2022-02-02 03:48:26 0:29:07 0:15:11 0:13:56 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6654154 2022-02-02 01:10:16 2022-02-02 03:19:40 2022-02-02 04:01:12 0:41:32 0:30:01 0:11:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} 3
pass 6654155 2022-02-02 01:10:17 2022-02-02 03:19:40 2022-02-02 03:42:57 0:23:17 0:11:57 0:11:20 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_msgr_segments} 1
pass 6654156 2022-02-02 01:10:18 2022-02-02 03:19:40 2022-02-02 04:00:29 0:40:49 0:28:10 0:12:39 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6654157 2022-02-02 01:10:19 2022-02-02 03:20:41 2022-02-02 04:02:16 0:41:35 0:30:04 0:11:31 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_fio} 3
pass 6654158 2022-02-02 01:10:20 2022-02-02 03:21:01 2022-02-02 04:01:52 0:40:51 0:27:17 0:13:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_data_pool} 3
fail 6654159 2022-02-02 01:10:21 2022-02-02 03:23:02 2022-02-02 08:15:58 4:52:56 4:38:39 0:14:17 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy} msgr-failures/many tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi197 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludewz83nba2 -r -- '-g auto -g blockdev -x clone'"

pass 6654160 2022-02-02 01:10:22 2022-02-02 03:26:03 2022-02-02 05:03:39 1:37:36 1:23:52 0:13:44 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6654161 2022-02-02 01:10:23 2022-02-02 03:27:03 2022-02-02 03:53:18 0:26:15 0:16:02 0:10:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6654162 2022-02-02 01:10:24 2022-02-02 03:27:43 2022-02-02 04:01:30 0:33:47 0:23:45 0:10:02 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6654163 2022-02-02 01:10:26 2022-02-02 03:27:54 2022-02-02 03:56:18 0:28:24 0:17:40 0:10:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} 3
pass 6654164 2022-02-02 01:10:27 2022-02-02 03:28:24 2022-02-02 03:56:41 0:28:17 0:17:21 0:10:56 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6654165 2022-02-02 01:10:28 2022-02-02 03:28:35 2022-02-02 04:26:56 0:58:21 0:44:05 0:14:16 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
fail 6654166 2022-02-02 01:10:29 2022-02-02 03:31:05 2022-02-02 04:09:25 0:38:20 0:25:40 0:12:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
Failure Reason:

"2022-02-02T03:56:39.745834+0000 mon.a (mon.0) 149 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6654167 2022-02-02 01:10:30 2022-02-02 03:31:16 2022-02-02 03:55:28 0:24:12 0:11:15 0:12:57 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_parent_overlap} 1
pass 6654168 2022-02-02 01:10:31 2022-02-02 03:31:16 2022-02-02 04:07:21 0:36:05 0:24:21 0:11:44 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6654169 2022-02-02 01:10:32 2022-02-02 03:31:36 2022-02-02 03:59:37 0:28:01 0:15:37 0:12:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} 3
pass 6654170 2022-02-02 01:10:33 2022-02-02 03:31:47 2022-02-02 05:04:04 1:32:17 1:19:18 0:12:59 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6654171 2022-02-02 01:10:34 2022-02-02 03:32:07 2022-02-02 04:30:07 0:58:00 0:44:23 0:13:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6654172 2022-02-02 01:10:35 2022-02-02 03:34:18 2022-02-02 04:06:02 0:31:44 0:19:44 0:12:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6654173 2022-02-02 01:10:36 2022-02-02 03:36:18 2022-02-02 04:06:42 0:30:24 0:18:38 0:11:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6654174 2022-02-02 01:10:37 2022-02-02 03:36:49 2022-02-02 04:05:49 0:29:00 0:15:10 0:13:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} 3
pass 6654175 2022-02-02 01:10:38 2022-02-02 03:40:10 2022-02-02 04:01:38 0:21:28 0:11:23 0:10:05 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} 1
pass 6654176 2022-02-02 01:10:39 2022-02-02 03:40:10 2022-02-02 04:22:12 0:42:02 0:32:03 0:09:59 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6654177 2022-02-02 01:10:40 2022-02-02 03:40:20 2022-02-02 04:07:10 0:26:50 0:15:53 0:10:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6654178 2022-02-02 01:10:41 2022-02-02 03:41:21 2022-02-02 04:11:12 0:29:51 0:17:15 0:12:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6654179 2022-02-02 01:10:42 2022-02-02 03:43:32 2022-02-02 04:18:56 0:35:24 0:22:55 0:12:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_concurrent} 3
pass 6654180 2022-02-02 01:10:43 2022-02-02 03:45:32 2022-02-02 04:14:25 0:28:53 0:15:27 0:13:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6654181 2022-02-02 01:10:44 2022-02-02 03:46:33 2022-02-02 04:26:57 0:40:24 0:28:15 0:12:09 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
fail 6654182 2022-02-02 01:10:45 2022-02-02 03:47:13 2022-02-02 04:25:52 0:38:39 0:25:36 0:13:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2022-02-02T04:23:20.924632+0000 mon.a (mon.0) 298 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654183 2022-02-02 01:10:47 2022-02-02 03:47:13 2022-02-02 04:10:46 0:23:33 0:12:52 0:10:41 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_whole_object_zeroout} 1
pass 6654184 2022-02-02 01:10:48 2022-02-02 03:47:14 2022-02-02 04:21:19 0:34:05 0:19:58 0:14:07 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6654185 2022-02-02 01:10:49 2022-02-02 03:48:34 2022-02-02 04:41:46 0:53:12 0:40:22 0:12:50 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} 3
pass 6654186 2022-02-02 01:10:50 2022-02-02 03:49:55 2022-02-02 04:20:02 0:30:07 0:16:28 0:13:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_kernel} 3
pass 6654187 2022-02-02 01:10:51 2022-02-02 03:52:15 2022-02-02 04:24:31 0:32:16 0:20:30 0:11:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
fail 6654188 2022-02-02 01:10:52 2022-02-02 03:52:56 2022-02-02 04:30:53 0:37:57 0:26:11 0:11:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2022-02-02T04:18:30.892301+0000 mon.a (mon.0) 112 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6654189 2022-02-02 01:10:53 2022-02-02 03:53:16 2022-02-02 04:26:59 0:33:43 0:21:53 0:11:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_map_unmap} 3
fail 6654190 2022-02-02 01:10:54 2022-02-02 03:53:26 2022-02-02 04:31:40 0:38:14 0:26:56 0:11:18 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
Failure Reason:

"2022-02-02T04:19:52.653734+0000 mon.a (mon.0) 146 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6654191 2022-02-02 01:10:55 2022-02-02 04:37:22 1884 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_simple_big} 3
pass 6654192 2022-02-02 01:10:56 2022-02-02 03:55:07 2022-02-02 04:18:07 0:23:00 0:12:11 0:10:49 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_deep_flatten} 1
pass 6654193 2022-02-02 01:10:57 2022-02-02 03:55:08 2022-02-02 04:40:54 0:45:46 0:34:38 0:11:08 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6654194 2022-02-02 01:10:58 2022-02-02 03:55:18 2022-02-02 04:40:22 0:45:04 0:33:29 0:11:35 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} 3
pass 6654195 2022-02-02 01:10:59 2022-02-02 03:56:29 2022-02-02 04:30:55 0:34:26 0:22:36 0:11:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} 3
pass 6654196 2022-02-02 01:11:01 2022-02-02 03:56:49 2022-02-02 04:23:24 0:26:35 0:13:23 0:13:12 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} 2
pass 6654197 2022-02-02 01:11:02 2022-02-02 03:59:40 2022-02-02 04:36:19 0:36:39 0:26:15 0:10:24 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} 3
pass 6654198 2022-02-02 01:11:03 2022-02-02 04:00:30 2022-02-02 04:26:52 0:26:22 0:15:22 0:11:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_exclusive_option} 3
pass 6654199 2022-02-02 01:11:04 2022-02-02 04:01:21 2022-02-02 04:37:08 0:35:47 0:25:29 0:10:18 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6654200 2022-02-02 01:11:05 2022-02-02 04:01:31 2022-02-02 04:29:49 0:28:18 0:16:55 0:11:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_fallocate} 3
pass 6654201 2022-02-02 01:11:06 2022-02-02 04:01:41 2022-02-02 04:32:48 0:31:07 0:17:37 0:13:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6654202 2022-02-02 01:11:07 2022-02-02 04:02:02 2022-02-02 04:53:33 0:51:31 0:40:42 0:10:49 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6654203 2022-02-02 01:11:08 2022-02-02 04:02:22 2022-02-02 04:33:32 0:31:10 0:17:06 0:14:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6654204 2022-02-02 01:11:09 2022-02-02 04:05:53 2022-02-02 04:27:53 0:22:00 0:11:03 0:10:57 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} 1
pass 6654205 2022-02-02 01:11:10 2022-02-02 04:06:03 2022-02-02 04:40:44 0:34:41 0:23:12 0:11:29 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6654206 2022-02-02 01:11:11 2022-02-02 04:06:44 2022-02-02 04:35:08 0:28:24 0:17:21 0:11:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_namespaces} 3
pass 6654207 2022-02-02 01:11:12 2022-02-02 04:07:14 2022-02-02 05:43:06 1:35:52 1:26:16 0:09:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6654208 2022-02-02 01:11:13 2022-02-02 04:07:24 2022-02-02 05:03:40 0:56:16 0:43:23 0:12:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6654209 2022-02-02 01:11:14 2022-02-02 04:09:35 2022-02-02 04:42:31 0:32:56 0:21:53 0:11:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6654210 2022-02-02 01:11:16 2022-02-02 04:10:55 2022-02-02 04:38:51 0:27:56 0:17:32 0:10:24 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6654211 2022-02-02 01:11:17 2022-02-02 04:11:16 2022-02-02 04:40:22 0:29:06 0:16:07 0:12:59 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_netns} 3
pass 6654212 2022-02-02 01:11:18 2022-02-02 04:14:27 2022-02-02 04:37:16 0:22:49 0:09:52 0:12:57 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_huge_image} 1
pass 6654213 2022-02-02 01:11:19 2022-02-02 04:18:17 2022-02-02 04:54:24 0:36:07 0:25:06 0:11:01 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6654214 2022-02-02 01:11:20 2022-02-02 04:18:58 2022-02-02 05:58:13 1:39:15 1:26:15 0:13:00 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6654215 2022-02-02 01:11:21 2022-02-02 04:20:09 2022-02-02 04:50:15 0:30:06 0:15:58 0:14:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6654216 2022-02-02 01:11:22 2022-02-02 04:21:29 2022-02-02 04:48:13 0:26:44 0:15:11 0:11:33 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6654217 2022-02-02 01:11:23 2022-02-02 04:22:20 2022-02-02 04:58:21 0:36:01 0:23:30 0:12:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_concurrent} 3
pass 6654218 2022-02-02 01:11:24 2022-02-02 04:24:41 2022-02-02 04:52:40 0:27:59 0:14:32 0:13:27 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6654219 2022-02-02 01:11:25 2022-02-02 04:26:01 2022-02-02 04:58:37 0:32:36 0:21:17 0:11:19 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6654220 2022-02-02 01:11:26 2022-02-02 04:27:01 2022-02-02 04:54:38 0:27:37 0:17:04 0:10:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} 3
pass 6654221 2022-02-02 01:11:27 2022-02-02 04:27:02 2022-02-02 04:50:54 0:23:52 0:13:32 0:10:20 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_modprobe} 1
pass 6654222 2022-02-02 01:11:28 2022-02-02 04:27:02 2022-02-02 05:02:42 0:35:40 0:24:37 0:11:03 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6654223 2022-02-02 01:11:29 2022-02-02 04:27:02 2022-02-02 04:54:48 0:27:46 0:16:16 0:11:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_kernel} 3
pass 6654224 2022-02-02 01:11:30 2022-02-02 04:27:03 2022-02-02 05:06:02 0:38:59 0:24:54 0:14:05 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6654225 2022-02-02 01:11:31 2022-02-02 04:29:53 2022-02-02 04:56:07 0:26:14 0:15:31 0:10:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_map_snapshot_io} 3
fail 6654226 2022-02-02 01:11:32 2022-02-02 04:30:14 2022-02-02 05:08:19 0:38:05 0:25:57 0:12:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} 3
Failure Reason:

"2022-02-02T04:55:49.855814+0000 mon.a (mon.0) 129 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log

pass 6654227 2022-02-02 01:11:33 2022-02-02 04:30:54 2022-02-02 04:59:35 0:28:41 0:17:02 0:11:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6654228 2022-02-02 01:11:34 2022-02-02 04:31:05 2022-02-02 05:09:07 0:38:02 0:27:59 0:10:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_simple_big} 3
pass 6654229 2022-02-02 01:11:35 2022-02-02 04:31:45 2022-02-02 04:55:34 0:23:49 0:12:55 0:10:54 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} 1
pass 6654230 2022-02-02 01:11:36 2022-02-02 04:31:45 2022-02-02 05:15:46 0:44:01 0:30:09 0:13:52 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6654231 2022-02-02 01:11:37 2022-02-02 04:32:56 2022-02-02 05:17:22 0:44:26 0:32:22 0:12:04 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_fio} 3
pass 6654232 2022-02-02 01:11:38 2022-02-02 04:33:36 2022-02-02 05:15:12 0:41:36 0:28:46 0:12:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_data_pool} 3
pass 6654233 2022-02-02 01:11:39 2022-02-02 04:35:17 2022-02-02 05:01:15 0:25:58 0:15:13 0:10:45 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6654234 2022-02-02 01:11:40 2022-02-02 04:35:47 2022-02-02 05:06:16 0:30:29 0:19:59 0:10:30 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6654235 2022-02-02 01:11:41 2022-02-02 04:36:28 2022-02-02 05:04:55 0:28:27 0:17:28 0:10:59 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_fallocate} 3
pass 6654236 2022-02-02 01:11:42 2022-02-02 04:37:18 2022-02-02 05:03:50 0:26:32 0:15:42 0:10:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_huge_osdmap} 3
fail 6654237 2022-02-02 01:11:43 2022-02-02 04:37:28 2022-02-02 05:14:34 0:37:06 0:25:01 0:12:05 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
Failure Reason:

"2022-02-02T05:01:38.427871+0000 mon.a (mon.0) 162 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6654238 2022-02-02 01:11:45 2022-02-02 04:38:59 2022-02-02 05:14:23 0:35:24 0:22:19 0:13:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
Failure Reason:

"2022-02-02T05:01:37.363834+0000 mon.a (mon.0) 183 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6654239 2022-02-02 01:11:46 2022-02-02 04:40:29 2022-02-02 05:06:28 0:25:59 0:14:02 0:11:57 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_parent_overlap} 1
pass 6654240 2022-02-02 01:11:47 2022-02-02 04:40:30 2022-02-02 05:14:10 0:33:40 0:22:08 0:11:32 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6654241 2022-02-02 01:11:48 2022-02-02 04:40:30 2022-02-02 05:35:01 0:54:31 0:43:09 0:11:22 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/upmap thrashosds-health workloads/rbd_fio} 3
pass 6654242 2022-02-02 01:11:49 2022-02-02 04:40:50 2022-02-02 05:04:32 0:23:42 0:13:25 0:10:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_namespaces} 3
pass 6654243 2022-02-02 01:11:50 2022-02-02 04:41:01 2022-02-02 06:07:49 1:26:48 1:15:12 0:11:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6654244 2022-02-02 01:11:51 2022-02-02 04:41:51 2022-02-02 05:40:19 0:58:28 0:46:25 0:12:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6654245 2022-02-02 01:11:52 2022-02-02 04:42:42 2022-02-02 05:14:17 0:31:35 0:17:49 0:13:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6654246 2022-02-02 01:11:53 2022-02-02 04:46:43 2022-02-02 05:14:10 0:27:27 0:14:26 0:13:01 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
fail 6654247 2022-02-02 01:11:54 2022-02-02 04:48:23 2022-02-02 05:21:52 0:33:29 0:20:13 0:13:16 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-02-02T05:19:00.589782+0000 mon.a (mon.0) 260 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654248 2022-02-02 01:11:55 2022-02-02 04:50:24 2022-02-02 05:11:54 0:21:30 0:10:23 0:11:07 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_read_only} 1
pass 6654249 2022-02-02 01:11:56 2022-02-02 04:50:24 2022-02-02 05:29:26 0:39:02 0:22:58 0:16:04 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6654250 2022-02-02 01:11:57 2022-02-02 04:52:45 2022-02-02 05:16:47 0:24:02 0:12:05 0:11:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6654251 2022-02-02 01:11:58 2022-02-02 04:53:35 2022-02-02 05:15:49 0:22:14 0:10:47 0:11:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6654252 2022-02-02 01:11:59 2022-02-02 04:54:26 2022-02-02 05:22:24 0:27:58 0:16:36 0:11:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} 3
pass 6654253 2022-02-02 01:12:00 2022-02-02 04:54:46 2022-02-02 05:16:43 0:21:57 0:11:18 0:10:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6654254 2022-02-02 01:12:01 2022-02-02 04:54:57 2022-02-02 05:28:27 0:33:30 0:21:04 0:12:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
pass 6654255 2022-02-02 01:12:02 2022-02-02 04:55:37 2022-02-02 05:18:04 0:22:27 0:11:15 0:11:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_image_read} 3
pass 6654256 2022-02-02 01:12:03 2022-02-02 04:56:18 2022-02-02 05:17:58 0:21:40 0:10:02 0:11:38 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} 1
fail 6654257 2022-02-02 01:12:04 2022-02-02 04:58:28 2022-02-02 05:30:16 0:31:48 0:19:59 0:11:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

"2022-02-02T05:27:06.438817+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6654258 2022-02-02 01:12:05 2022-02-02 04:58:38 2022-02-02 05:20:50 0:22:12 0:10:31 0:11:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} 3
pass 6654259 2022-02-02 01:12:06 2022-02-02 04:59:19 2022-02-02 05:27:43 0:28:24 0:16:42 0:11:42 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6654260 2022-02-02 01:12:07 2022-02-02 04:59:39 2022-02-02 05:22:49 0:23:10 0:10:30 0:12:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6654261 2022-02-02 01:12:08 2022-02-02 05:01:20 2022-02-02 05:28:32 0:27:12 0:15:38 0:11:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_map_unmap} 3
pass 6654262 2022-02-02 01:12:09 2022-02-02 05:02:50 2022-02-02 05:29:55 0:27:05 0:13:24 0:13:41 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6654263 2022-02-02 01:12:10 2022-02-02 05:03:41 2022-02-02 05:44:09 0:40:28 0:28:44 0:11:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} 3