Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 6566867 2021-12-17 03:21:03 2021-12-17 03:21:44 2021-12-17 03:43:44 0:22:00 0:10:46 0:11:14 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_deep_flatten} 1
pass 6566868 2021-12-17 03:21:03 2021-12-17 03:21:45 2021-12-17 04:01:51 0:40:06 0:20:12 0:19:54 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6566869 2021-12-17 03:21:04 2021-12-17 03:21:45 2021-12-17 04:07:10 0:45:25 0:28:14 0:17:11 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_fio} 3
pass 6566870 2021-12-17 03:21:05 2021-12-17 03:21:45 2021-12-17 04:07:22 0:45:37 0:24:18 0:21:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_data_pool} 3
fail 6566871 2021-12-17 03:21:06 2021-12-17 03:21:46 2021-12-17 08:18:54 4:57:08 4:42:58 0:14:10 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{prefer-crc} msgr-failures/few tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi194 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludep_ahkfn7 -r -- '-g auto -g blockdev -x clone'"

pass 6566872 2021-12-17 03:21:06 2021-12-17 03:21:46 2021-12-17 04:22:11 1:00:25 0:40:05 0:20:20 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/backoff thrashosds-health workloads/rbd_fio} 3
fail 6566873 2021-12-17 03:21:07 2021-12-17 03:21:46 2021-12-17 03:32:50 0:11:04 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major

pass 6566874 2021-12-17 03:21:08 2021-12-17 03:21:47 2021-12-17 03:56:25 0:34:38 0:13:34 0:21:04 smithi master krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} 1
fail 6566875 2021-12-17 03:21:08 2021-12-17 03:21:47 2021-12-17 04:08:05 0:46:18 0:24:28 0:21:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2021-12-17T04:07:03.011987+0000 mon.a (mon.0) 257 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6566876 2021-12-17 03:21:09 2021-12-17 03:21:47 2021-12-17 03:55:34 0:33:47 0:20:23 0:13:24 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6566877 2021-12-17 03:21:10 2021-12-17 03:21:48 2021-12-17 04:00:24 0:38:36 0:16:58 0:21:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} 3
pass 6566878 2021-12-17 03:21:11 2021-12-17 03:21:48 2021-12-17 04:00:04 0:38:16 0:17:28 0:20:48 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6566879 2021-12-17 03:21:12 2021-12-17 03:21:48 2021-12-17 04:14:38 0:52:50 0:38:24 0:14:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6566880 2021-12-17 03:21:12 2021-12-17 03:21:49 2021-12-17 04:02:19 0:40:30 0:18:36 0:21:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6566881 2021-12-17 03:21:13 2021-12-17 03:21:49 2021-12-17 03:41:50 0:20:01 0:10:32 0:09:29 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_discard} 1
pass 6566882 2021-12-17 03:21:14 2021-12-17 03:21:49 2021-12-17 03:59:37 0:37:48 0:18:20 0:19:28 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
fail 6566883 2021-12-17 03:21:14 2021-12-17 03:21:49 2021-12-17 04:08:27 0:46:38 0:24:36 0:22:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2021-12-17T04:06:10.594756+0000 mon.a (mon.0) 204 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6566884 2021-12-17 03:21:15 2021-12-17 03:21:50 2021-12-17 03:56:27 0:34:37 0:19:54 0:14:43 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2021-12-17T03:54:05.590486+0000 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6566885 2021-12-17 03:21:16 2021-12-17 03:21:50 2021-12-17 04:11:13 0:49:23 0:37:42 0:11:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6566886 2021-12-17 03:21:17 2021-12-17 03:21:50 2021-12-17 03:53:19 0:31:29 0:15:11 0:16:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6566887 2021-12-17 03:21:17 2021-12-17 03:21:51 2021-12-17 04:00:11 0:38:20 0:16:54 0:21:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6566888 2021-12-17 03:21:18 2021-12-17 03:21:51 2021-12-17 03:59:25 0:37:34 0:16:13 0:21:21 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} 3
pass 6566889 2021-12-17 03:21:19 2021-12-17 03:21:52 2021-12-17 03:47:24 0:25:32 0:09:10 0:16:22 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} 1
fail 6566890 2021-12-17 03:21:20 2021-12-17 03:21:52 2021-12-17 04:01:45 0:39:53 0:19:59 0:19:54 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2021-12-17T03:59:56.170915+0000 mon.a (mon.0) 270 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6566891 2021-12-17 03:21:20 2021-12-17 03:21:52 2021-12-17 04:08:43 0:46:51 0:24:19 0:22:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_symlinks} 3
Failure Reason:

"2021-12-17T03:56:00.350049+0000 mon.a (mon.0) 181 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6566892 2021-12-17 03:21:21 2021-12-17 03:21:52 2021-12-17 04:01:12 0:39:20 0:16:47 0:22:33 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6566893 2021-12-17 03:21:22 2021-12-17 03:21:53 2021-12-17 04:09:15 0:47:22 0:24:40 0:22:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_concurrent} 3
Failure Reason:

"2021-12-17T04:06:53.620798+0000 mon.a (mon.0) 204 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6566894 2021-12-17 03:21:22 2021-12-17 03:21:53 2021-12-17 04:08:46 0:46:53 0:24:16 0:22:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2021-12-17T04:06:43.387508+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6566895 2021-12-17 03:21:23 2021-12-17 03:21:53 2021-12-17 04:07:17 0:45:24 0:23:50 0:21:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6566896 2021-12-17 03:21:24 2021-12-17 03:21:54 2021-12-17 04:00:49 0:38:55 0:16:09 0:22:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_image_read} 3
pass 6566897 2021-12-17 03:21:25 2021-12-17 03:21:54 2021-12-17 03:45:12 0:23:18 0:09:49 0:13:29 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_modprobe} 1
pass 6566898 2021-12-17 03:21:25 2021-12-17 03:21:54 2021-12-17 04:04:18 0:42:24 0:20:33 0:21:51 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6566899 2021-12-17 03:21:26 2021-12-17 03:21:55 2021-12-17 05:05:31 1:43:36 1:22:20 0:21:16 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6566900 2021-12-17 03:21:27 2021-12-17 03:21:55 2021-12-17 03:59:01 0:37:06 0:15:21 0:21:45 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_kernel} 3
pass 6566901 2021-12-17 03:21:28 2021-12-17 03:21:56 2021-12-17 04:07:23 0:45:27 0:22:44 0:22:43 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6566902 2021-12-17 03:21:28 2021-12-17 03:21:56 2021-12-17 03:58:03 0:36:07 0:14:07 0:22:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} 3
fail 6566903 2021-12-17 03:21:29 2021-12-17 03:21:56 2021-12-17 03:56:33 0:34:37 0:20:06 0:14:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_unmap} 3
Failure Reason:

"2021-12-17T03:54:44.663930+0000 mon.a (mon.0) 323 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6566904 2021-12-17 03:21:30 2021-12-17 03:21:57 2021-12-17 04:00:24 0:38:27 0:17:18 0:21:09 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6566905 2021-12-17 03:21:31 2021-12-17 03:21:57 2021-12-17 04:01:44 0:39:47 0:26:25 0:13:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_simple_big} 3
pass 6566906 2021-12-17 03:21:32 2021-12-17 03:21:57 2021-12-17 03:57:35 0:35:38 0:13:48 0:21:50 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_msgr_segments} 1
pass 6566907 2021-12-17 03:21:32 2021-12-17 03:21:57 2021-12-17 04:09:56 0:47:59 0:27:50 0:20:09 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
fail 6566908 2021-12-17 03:21:33 2021-12-17 03:21:58 2021-12-17 03:35:19 0:13:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

Command failed on smithi122 with status 128: 'cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/master'

fail 6566909 2021-12-17 03:21:34 2021-12-17 03:21:58 2021-12-17 04:08:57 0:46:59 0:23:24 0:23:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2021-12-17T03:56:24.158765+0000 mon.a (mon.0) 180 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6566910 2021-12-17 03:21:34 2021-12-17 03:33:00 2021-12-17 04:01:18 0:28:18 0:14:15 0:14:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6566911 2021-12-17 03:21:35 2021-12-17 03:35:21 2021-12-17 04:11:49 0:36:28 0:18:14 0:18:14 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6566912 2021-12-17 03:21:36 2021-12-17 03:41:52 2021-12-17 04:09:48 0:27:56 0:11:46 0:16:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_fallocate} 3
pass 6566913 2021-12-17 03:21:37 2021-12-17 03:47:33 2021-12-17 04:15:53 0:28:20 0:11:43 0:16:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} 3
fail 6566914 2021-12-17 03:21:37 2021-12-17 03:53:24 2021-12-17 04:25:48 0:32:24 0:19:34 0:12:50 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
Failure Reason:

"2021-12-17T04:24:40.940324+0000 mon.a (mon.0) 260 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6566915 2021-12-17 03:21:38 2021-12-17 03:55:35 2021-12-17 04:22:36 0:27:01 0:14:49 0:12:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6566916 2021-12-17 03:21:39 2021-12-17 03:56:35 2021-12-17 04:16:35 0:20:00 0:09:23 0:10:37 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} 1
pass 6566917 2021-12-17 03:21:40 2021-12-17 03:56:36 2021-12-17 04:26:52 0:30:16 0:18:26 0:11:50 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6566918 2021-12-17 03:21:40 2021-12-17 03:56:36 2021-12-17 04:20:05 0:23:29 0:11:35 0:11:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_namespaces} 3
pass 6566919 2021-12-17 03:21:41 2021-12-17 03:58:06 2021-12-17 05:27:28 1:29:22 1:16:19 0:13:03 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6566920 2021-12-17 03:21:42 2021-12-17 03:59:07 2021-12-17 04:51:57 0:52:50 0:40:15 0:12:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6566921 2021-12-17 03:21:42 2021-12-17 03:59:27 2021-12-17 04:30:48 0:31:21 0:16:15 0:15:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6566922 2021-12-17 03:21:43 2021-12-17 03:59:48 2021-12-17 04:24:19 0:24:31 0:13:11 0:11:20 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6566923 2021-12-17 03:21:44 2021-12-17 04:00:08 2021-12-17 04:24:23 0:24:15 0:12:29 0:11:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_netns} 3
pass 6566924 2021-12-17 03:21:45 2021-12-17 04:00:18 2021-12-17 04:22:28 0:22:10 0:10:56 0:11:14 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_read_only} 1
pass 6566925 2021-12-17 03:21:45 2021-12-17 04:00:19 2021-12-17 04:30:40 0:30:21 0:18:59 0:11:22 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6566926 2021-12-17 03:21:46 2021-12-17 04:00:29 2021-12-17 04:50:49 0:50:20 0:39:00 0:11:20 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/pggrow thrashosds-health workloads/rbd_fio} 3
pass 6566927 2021-12-17 03:21:47 2021-12-17 04:00:29 2021-12-17 04:25:04 0:24:35 0:12:39 0:11:56 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6566928 2021-12-17 03:21:48 2021-12-17 04:00:50 2021-12-17 04:25:34 0:24:44 0:11:47 0:12:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6566929 2021-12-17 03:21:48 2021-12-17 04:01:20 2021-12-17 08:15:51 4:14:31 4:00:07 0:14:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

Command failed (workunit test rbd/concurrent.sh) on smithi172 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1db238840efd3668e5fb3b829cf206334b1dbe4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh'

pass 6566930 2021-12-17 03:21:49 2021-12-17 04:01:20 2021-12-17 04:25:49 0:24:29 0:12:11 0:12:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6566931 2021-12-17 03:21:50 2021-12-17 04:36:12 1350 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
pass 6566932 2021-12-17 03:21:51 2021-12-17 04:26:08 747 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} 3
pass 6566933 2021-12-17 03:21:52 2021-12-17 04:01:51 2021-12-17 04:24:31 0:22:40 0:10:56 0:11:44 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_whole_object_zeroout} 1
pass 6566934 2021-12-17 03:21:53 2021-12-17 04:02:22 2021-12-17 04:36:39 0:34:17 0:20:34 0:13:43 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6566935 2021-12-17 03:21:53 2021-12-17 04:31:26 652 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_kernel} 3
pass 6566936 2021-12-17 03:21:54 2021-12-17 04:38:46 990 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6566937 2021-12-17 03:21:55 2021-12-17 04:29:18 663 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6566938 2021-12-17 03:21:56 2021-12-17 04:07:24 2021-12-17 04:35:37 0:28:13 0:15:45 0:12:28 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} 3
fail 6566939 2021-12-17 03:21:57 2021-12-17 04:42:19 1202 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
Failure Reason:

"2021-12-17T04:29:53.708415+0000 mon.a (mon.0) 173 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6566940 2021-12-17 03:21:58 2021-12-17 04:08:15 2021-12-17 04:50:25 0:42:10 0:27:31 0:14:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_simple_big} 3
pass 6566941 2021-12-17 03:21:59 2021-12-17 04:08:35 2021-12-17 04:32:47 0:24:12 0:10:21 0:13:51 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} 1
pass 6566942 2021-12-17 03:21:59 2021-12-17 04:08:35 2021-12-17 04:44:47 0:36:12 0:21:36 0:14:36 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6566943 2021-12-17 03:22:00 2021-12-17 04:08:46 2021-12-17 04:52:45 0:43:59 0:29:03 0:14:56 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_fio} 3
pass 6566944 2021-12-17 03:22:01 2021-12-17 04:08:56 2021-12-17 04:43:09 0:34:13 0:19:18 0:14:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_data_pool} 3
pass 6566945 2021-12-17 03:22:02 2021-12-17 04:09:06 2021-12-17 04:33:34 0:24:28 0:09:51 0:14:37 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} 2
pass 6566946 2021-12-17 03:22:02 2021-12-17 04:09:17 2021-12-17 04:47:02 0:37:45 0:22:18 0:15:27 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} 3
pass 6566947 2021-12-17 03:22:03 2021-12-17 04:09:57 2021-12-17 04:36:24 0:26:27 0:12:36 0:13:51 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} 3
pass 6566948 2021-12-17 03:22:04 2021-12-17 04:09:57 2021-12-17 04:45:16 0:35:19 0:19:45 0:15:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6566949 2021-12-17 03:22:05 2021-12-17 04:11:18 2021-12-17 04:36:24 0:25:06 0:11:43 0:13:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_fallocate} 3
pass 6566950 2021-12-17 03:22:05 2021-12-17 04:11:58 2021-12-17 04:38:50 0:26:52 0:11:45 0:15:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6566951 2021-12-17 03:22:06 2021-12-17 04:14:39 2021-12-17 05:04:30 0:49:51 0:37:17 0:12:34 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6566952 2021-12-17 03:22:07 2021-12-17 04:16:00 2021-12-17 04:42:37 0:26:37 0:14:07 0:12:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6566953 2021-12-17 03:22:07 2021-12-17 04:16:40 2021-12-17 04:42:08 0:25:28 0:09:54 0:15:34 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_discard} 1
pass 6566954 2021-12-17 03:22:08 2021-12-17 04:20:11 2021-12-17 04:52:21 0:32:10 0:19:21 0:12:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6566955 2021-12-17 03:22:09 2021-12-17 04:22:11 2021-12-17 06:10:30 1:48:19 1:37:07 0:11:12 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6566956 2021-12-17 03:22:10 2021-12-17 04:22:32 2021-12-17 04:54:52 0:32:20 0:19:44 0:12:36 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_namespaces} 3
Failure Reason:

"2021-12-17T04:52:41.075418+0000 mon.a (mon.0) 294 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6566957 2021-12-17 03:22:10 2021-12-17 04:22:42 2021-12-17 05:58:30 1:35:48 1:23:27 0:12:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6566958 2021-12-17 03:22:11 2021-12-17 04:24:23 2021-12-17 05:14:45 0:50:22 0:37:40 0:12:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_enumerate} 3
fail 6566959 2021-12-17 03:22:12 2021-12-17 04:24:33 2021-12-17 04:57:19 0:32:46 0:19:44 0:13:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2021-12-17T04:45:08.446391+0000 mon.a (mon.0) 116 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6566960 2021-12-17 03:22:12 2021-12-17 04:25:14 2021-12-17 04:49:53 0:24:39 0:12:28 0:12:11 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
fail 6566961 2021-12-17 03:22:13 2021-12-17 04:25:44 2021-12-17 04:48:04 0:22:20 0:10:10 0:12:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_netns} 3
Failure Reason:

Command failed on smithi175 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1db238840efd3668e5fb3b829cf206334b1dbe4c'

pass 6566962 2021-12-17 03:22:14 2021-12-17 04:25:55 2021-12-17 04:47:44 0:21:49 0:09:34 0:12:15 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_huge_image} 1
pass 6566963 2021-12-17 03:22:15 2021-12-17 04:25:55 2021-12-17 05:00:10 0:34:15 0:21:42 0:12:33 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6566964 2021-12-17 03:22:15 2021-12-17 04:25:55 2021-12-17 04:50:27 0:24:32 0:12:49 0:11:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6566965 2021-12-17 03:22:16 2021-12-17 04:26:16 2021-12-17 04:49:22 0:23:06 0:11:20 0:11:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6566966 2021-12-17 03:22:17 2021-12-17 04:26:56 2021-12-17 04:55:13 0:28:17 0:15:42 0:12:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} 3
pass 6566967 2021-12-17 03:22:17 2021-12-17 04:29:27 2021-12-17 04:52:59 0:23:32 0:10:59 0:12:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6566968 2021-12-17 03:22:18 2021-12-17 04:30:47 2021-12-17 05:03:55 0:33:08 0:19:22 0:13:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6566969 2021-12-17 03:22:19 2021-12-17 04:30:58 2021-12-17 04:55:37 0:24:39 0:11:07 0:13:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_image_read} 3
pass 6566970 2021-12-17 03:22:20 2021-12-17 04:31:28 2021-12-17 04:52:59 0:21:31 0:09:03 0:12:28 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} 1
pass 6566971 2021-12-17 03:22:20 2021-12-17 04:32:48 2021-12-17 05:07:56 0:35:08 0:20:03 0:15:05 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
fail 6566972 2021-12-17 03:22:21 2021-12-17 04:35:39 2021-12-17 05:08:33 0:32:54 0:19:34 0:13:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} 3
Failure Reason:

"2021-12-17T05:05:58.247865+0000 mon.a (mon.0) 257 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6566973 2021-12-17 03:22:22 2021-12-17 04:36:20 2021-12-17 05:08:48 0:32:28 0:20:16 0:12:12 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2021-12-17T04:57:52.144585+0000 osd.3 (osd.3) 3 : cluster [WRN] slow request osd_op(client.4512.0:9726 2.2 2:5cc62631:::rbd_data.1183163d8235.0000000000000005:head [zero 3354624~4096] snapc 0=[] ondisk+write+known_if_redirected e24) initiated 2021-12-17T04:57:21.372768+0000 currently started" in cluster log

pass 6566974 2021-12-17 03:22:22 2021-12-17 04:36:30 2021-12-17 05:00:51 0:24:21 0:11:18 0:13:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6566975 2021-12-17 03:22:23 2021-12-17 04:36:31 2021-12-17 05:05:05 0:28:34 0:16:20 0:12:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_map_unmap} 3
pass 6566976 2021-12-17 03:22:24 2021-12-17 04:36:41 2021-12-17 05:01:03 0:24:22 0:11:37 0:12:45 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6566977 2021-12-17 03:22:24 2021-12-17 04:38:52 2021-12-17 05:18:15 0:39:23 0:27:44 0:11:39 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} 3
pass 6566978 2021-12-17 03:22:25 2021-12-17 04:38:52 2021-12-17 04:59:16 0:20:24 0:09:25 0:10:59 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_msgr_segments} 1
pass 6566979 2021-12-17 03:22:26 2021-12-17 04:38:52 2021-12-17 05:18:53 0:40:01 0:24:06 0:15:55 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
fail 6566980 2021-12-17 03:22:27 2021-12-17 04:42:23 2021-12-17 05:14:53 0:32:30 0:19:46 0:12:44 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

"2021-12-17T05:12:03.125655+0000 mon.a (mon.0) 228 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6566981 2021-12-17 03:22:27 2021-12-17 04:42:44 2021-12-17 05:17:23 0:34:39 0:22:46 0:11:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_data_pool} 3
fail 6566982 2021-12-17 03:22:28 2021-12-17 04:43:14 2021-12-17 05:17:53 0:34:39 0:19:40 0:14:59 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{crc} msgr-failures/many tasks/rbd_xfstests} 4
Failure Reason:

"2021-12-17T05:16:21.127345+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6566983 2021-12-17 03:22:29 2021-12-17 04:44:55 2021-12-17 06:31:40 1:46:45 1:34:55 0:11:50 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6566984 2021-12-17 03:22:29 2021-12-17 04:45:25 2021-12-17 05:15:53 0:30:28 0:15:31 0:14:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_exclusive_option} 3
dead 6566985 2021-12-17 03:22:30 2021-12-17 04:47:06 2021-12-17 05:04:09 0:17:03 0:04:54 0:12:09 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

{'Failure object was': {'smithi058.front.sepia.ceph.com': {'msg': 'Failed to update apt cache: ', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, '_ansible_no_log': False, 'attempts': 24, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"}

pass 6566986 2021-12-17 03:22:31 2021-12-17 04:48:06 2021-12-17 05:15:52 0:27:46 0:15:21 0:12:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} 3
pass 6566987 2021-12-17 03:22:32 2021-12-17 04:49:27 2021-12-17 05:16:06 0:26:39 0:14:35 0:12:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_huge_osdmap} 3
fail 6566988 2021-12-17 03:22:32 2021-12-17 04:49:57 2021-12-17 05:22:53 0:32:56 0:20:03 0:12:53 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
Failure Reason:

"2021-12-17T05:20:49.259288+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6566989 2021-12-17 03:22:33 2021-12-17 04:50:27 2021-12-17 05:23:06 0:32:39 0:20:39 0:12:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
Failure Reason:

"2021-12-17T05:21:51.820710+0000 mon.a (mon.0) 264 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6566990 2021-12-17 03:22:34 2021-12-17 04:50:38 2021-12-17 05:14:47 0:24:09 0:11:43 0:12:26 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_parent_overlap} 1
pass 6566991 2021-12-17 03:22:34 2021-12-17 04:50:38 2021-12-17 05:21:12 0:30:34 0:18:00 0:12:34 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6566992 2021-12-17 03:22:35 2021-12-17 04:50:58 2021-12-17 05:18:27 0:27:29 0:14:48 0:12:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} 3
pass 6566993 2021-12-17 03:22:36 2021-12-17 04:51:59 2021-12-17 06:00:29 1:08:30 0:55:23 0:13:07 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
fail 6566994 2021-12-17 03:22:36 2021-12-17 04:52:29 2021-12-17 05:26:28 0:33:59 0:21:24 0:12:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_enumerate} 3
Failure Reason:

"2021-12-17T05:13:23.924206+0000 mon.a (mon.0) 131 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6566995 2021-12-17 03:22:37 2021-12-17 04:52:30 2021-12-17 05:23:02 0:30:32 0:18:32 0:12:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6566996 2021-12-17 03:22:38 2021-12-17 04:52:50 2021-12-17 05:19:12 0:26:22 0:15:10 0:11:12 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6566997 2021-12-17 03:22:38 2021-12-17 04:53:01 2021-12-17 05:21:02 0:28:01 0:13:20 0:14:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} 3
pass 6566998 2021-12-17 03:22:39 2021-12-17 04:55:01 2021-12-17 05:22:19 0:27:18 0:11:09 0:16:09 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} 1
fail 6566999 2021-12-17 03:22:40 2021-12-17 04:55:01 2021-12-17 05:31:17 0:36:16 0:20:25 0:15:51 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2021-12-17T05:28:22.183456+0000 mon.a (mon.0) 342 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6567000 2021-12-17 03:22:41 2021-12-17 04:55:22 2021-12-17 05:23:52 0:28:30 0:12:56 0:15:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6567001 2021-12-17 03:22:41 2021-12-17 04:55:42 2021-12-17 05:23:24 0:27:42 0:11:35 0:16:07 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6567002 2021-12-17 03:22:42 2021-12-17 04:57:23 2021-12-17 08:48:29 3:51:06 3:33:15 0:17:51 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

Command failed (workunit test rbd/concurrent.sh) on smithi181 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1db238840efd3668e5fb3b829cf206334b1dbe4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh'

pass 6567003 2021-12-17 03:22:43 2021-12-17 05:00:14 2021-12-17 05:31:06 0:30:52 0:11:14 0:19:38 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6567004 2021-12-17 03:22:43 2021-12-17 05:00:54 2021-12-17 05:41:17 0:40:23 0:19:51 0:20:32 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2021-12-17T05:28:41.992041+0000 mon.a (mon.0) 118 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6567005 2021-12-17 03:22:44 2021-12-17 05:01:04 2021-12-17 05:37:02 0:35:58 0:14:09 0:21:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_image_read} 3
pass 6567006 2021-12-17 03:22:45 2021-12-17 05:04:05 2021-12-17 05:32:35 0:28:30 0:09:50 0:18:40 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_whole_object_zeroout} 1
fail 6567007 2021-12-17 03:22:45 2021-12-17 05:04:05 2021-12-17 05:38:16 0:34:11 0:15:30 0:18:41 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

"2021-12-17T05:29:39.048497+0000 mon.a (mon.0) 121 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6567008 2021-12-17 03:22:46 2021-12-17 05:04:16 2021-12-17 06:01:04 0:56:48 0:37:48 0:19:00 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} 3
pass 6567009 2021-12-17 03:22:47 2021-12-17 05:04:36 2021-12-17 05:35:54 0:31:18 0:13:20 0:17:58 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_kernel} 3
pass 6567010 2021-12-17 03:22:47 2021-12-17 05:05:07 2021-12-17 05:40:17 0:35:10 0:17:05 0:18:05 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6567011 2021-12-17 03:22:48 2021-12-17 05:05:37 2021-12-17 05:36:09 0:30:32 0:12:43 0:17:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6567012 2021-12-17 03:22:49 2021-12-17 05:07:58 2021-12-17 05:40:38 0:32:40 0:17:27 0:15:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_map_unmap} 3
pass 6567013 2021-12-17 03:22:49 2021-12-17 05:08:38 2021-12-17 05:36:52 0:28:14 0:13:04 0:15:10 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
dead 6567014 2021-12-17 03:22:50 2021-12-17 05:08:49 2021-12-17 05:30:50 0:22:01 0:04:44 0:17:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_simple_big} 3
Failure Reason:

{'Failure object was': {'smithi139.front.sepia.ceph.com': {'stdout': '', 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6150 (apt-get)\\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\\n', 'rc': 100, 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': True, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': ['ceph', 'ceph-common', 'libcephfs1', 'radosgw', 'python-ceph', 'python-rados', 'python-cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph-fs-common', 'openmpi-common'], 'autoclean': False, 'install_recommends': 'None', 'name': ['ceph', 'ceph-common', 'libcephfs1', 'radosgw', 'python-ceph', 'python-rados', 'python-cephfs', 'python-rbd', 'librbd1', 'librados2', 'ceph-fs-common-dbg', 'ceph-fs-common', 'openmpi-common'], 'purge': False, 'allow_unauthenticated': False, 'state': 'absent', 'upgrade': 'None', 'update_cache': 'None', 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, 'msg': "'apt-get remove 'librbd1' 'librados2'' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6150 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n", 'stdout_lines': [], 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6150 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'invocation')"}

pass 6567015 2021-12-17 03:22:51 2021-12-17 05:14:50 2021-12-17 05:36:58 0:22:08 0:11:12 0:10:56 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_deep_flatten} 1
pass 6567016 2021-12-17 03:22:52 2021-12-17 05:14:50 2021-12-17 05:54:58 0:40:08 0:28:00 0:12:08 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6567017 2021-12-17 03:22:52 2021-12-17 05:15:00 2021-12-17 05:59:05 0:44:05 0:29:33 0:14:32 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} 3
pass 6567018 2021-12-17 03:22:53 2021-12-17 05:16:01 2021-12-17 05:50:28 0:34:27 0:21:01 0:13:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} 3
pass 6567019 2021-12-17 03:22:54 2021-12-17 05:16:01 2021-12-17 05:40:00 0:23:59 0:12:21 0:11:38 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} 2
pass 6567020 2021-12-17 03:22:54 2021-12-17 05:16:02 2021-12-17 05:52:22 0:36:20 0:24:07 0:12:13 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} 3
fail 6567021 2021-12-17 03:22:55 2021-12-17 05:16:12 2021-12-17 05:49:35 0:33:23 0:20:15 0:13:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2021-12-17T05:48:08.052106+0000 mon.a (mon.0) 292 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6567022 2021-12-17 03:22:56 2021-12-17 05:17:33 2021-12-17 05:50:20 0:32:47 0:20:08 0:12:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2021-12-17T05:48:39.568805+0000 mon.a (mon.0) 320 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6567023 2021-12-17 03:22:56 2021-12-17 15:32:56 2021-12-17 15:57:30 0:24:34 0:11:14 0:13:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_fallocate} 3
fail 6567024 2021-12-17 03:22:57 2021-12-17 15:35:17 2021-12-17 16:07:42 0:32:25 0:19:36 0:12:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2021-12-17T16:06:11.477593+0000 mon.a (mon.0) 224 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6567025 2021-12-17 03:22:58 2021-12-17 15:37:28 2021-12-17 16:32:21 0:54:53 0:36:54 0:17:59 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6567026 2021-12-17 03:22:59 2021-12-17 15:42:59 2021-12-17 16:06:56 0:23:57 0:14:07 0:09:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6567027 2021-12-17 03:22:59 2021-12-17 15:42:59 2021-12-17 16:07:25 0:24:26 0:10:12 0:14:14 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} 1
pass 6567028 2021-12-17 03:23:00 2021-12-17 15:45:30 2021-12-17 16:14:32 0:29:02 0:16:30 0:12:32 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6567029 2021-12-17 03:23:01 2021-12-17 15:46:20 2021-12-17 16:14:41 0:28:21 0:12:41 0:15:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_namespaces} 3
pass 6567030 2021-12-17 03:23:01 2021-12-17 15:51:21 2021-12-17 17:33:47 1:42:26 1:24:32 0:17:54 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6567031 2021-12-17 03:23:02 2021-12-17 15:57:32 2021-12-17 16:47:29 0:49:57 0:37:16 0:12:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} 3
fail 6567032 2021-12-17 03:23:03 2021-12-17 15:59:03 2021-12-17 16:29:32 0:30:29 0:19:40 0:10:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2021-12-17T16:17:58.863047+0000 mon.a (mon.0) 141 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6567033 2021-12-17 03:23:03 2021-12-17 15:59:43 2021-12-17 16:24:38 0:24:55 0:12:13 0:12:42 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6567034 2021-12-17 03:23:04 2021-12-17 16:00:04 2021-12-17 16:28:46 0:28:42 0:11:38 0:17:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_netns} 3
pass 6567035 2021-12-17 03:23:05 2021-12-17 16:07:05 2021-12-17 16:27:10 0:20:05 0:09:34 0:10:31 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_huge_image} 1
fail 6567036 2021-12-17 03:23:06 2021-12-17 16:07:05 2021-12-17 16:41:41 0:34:36 0:22:08 0:12:28 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2021-12-17T16:26:01.876167+0000 mon.a (mon.0) 317 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6567037 2021-12-17 03:23:06 2021-12-17 16:07:46 2021-12-17 17:51:34 1:43:48 1:32:01 0:11:47 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{prefer-crc} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6567038 2021-12-17 03:23:07 2021-12-17 16:09:16 2021-12-17 16:33:53 0:24:37 0:12:35 0:12:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6567039 2021-12-17 03:23:08 2021-12-17 16:09:57 2021-12-17 16:34:14 0:24:17 0:11:21 0:12:56 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6567040 2021-12-17 03:23:08 2021-12-17 16:12:17 2021-12-17 16:39:39 0:27:22 0:15:45 0:11:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_concurrent} 3
pass 6567041 2021-12-17 03:23:09 2021-12-17 16:13:18 2021-12-17 16:38:53 0:25:35 0:11:10 0:14:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6567042 2021-12-17 03:23:10 2021-12-17 16:14:38 2021-12-17 16:44:59 0:30:21 0:18:45 0:11:36 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
fail 6567043 2021-12-17 03:23:11 2021-12-17 16:14:49 2021-12-17 16:52:10 0:37:21 0:19:23 0:17:58 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} 3
Failure Reason:

"2021-12-17T16:40:17.887188+0000 mon.a (mon.0) 113 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6567044 2021-12-17 03:23:11 2021-12-17 16:21:30 2021-12-17 16:41:40 0:20:10 0:09:32 0:10:38 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_modprobe} 1
pass 6567045 2021-12-17 03:23:12 2021-12-17 16:21:30 2021-12-17 16:53:00 0:31:30 0:19:27 0:12:03 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6567046 2021-12-17 03:23:13 2021-12-17 16:22:51 2021-12-17 16:47:36 0:24:45 0:11:13 0:13:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_kernel} 3
pass 6567047 2021-12-17 03:23:13 2021-12-17 16:24:41 2021-12-17 16:59:05 0:34:24 0:18:33 0:15:51 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6567048 2021-12-17 03:23:14 2021-12-17 16:28:52 2021-12-17 16:51:42 0:22:50 0:10:32 0:12:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6567049 2021-12-17 03:23:15 2021-12-17 16:29:43 2021-12-17 16:58:42 0:28:59 0:15:41 0:13:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} 3
pass 6567050 2021-12-17 03:23:15 2021-12-17 16:30:33 2021-12-17 16:57:19 0:26:46 0:11:52 0:14:54 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6567051 2021-12-17 03:23:16 2021-12-17 16:32:24 2021-12-17 17:12:03 0:39:39 0:26:25 0:13:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_simple_big} 3
pass 6567052 2021-12-17 03:23:17 2021-12-17 16:33:54 2021-12-17 16:56:36 0:22:42 0:09:41 0:13:01 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} 1
pass 6567053 2021-12-17 03:23:18 2021-12-17 16:34:24 2021-12-17 17:12:34 0:38:10 0:25:50 0:12:20 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6567054 2021-12-17 03:23:18 2021-12-17 16:34:25 2021-12-17 17:15:44 0:41:19 0:27:52 0:13:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_fio} 3
pass 6567055 2021-12-17 03:23:19 2021-12-17 16:34:25 2021-12-17 17:09:51 0:35:26 0:23:23 0:12:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_data_pool} 3
pass 6567056 2021-12-17 03:23:20 2021-12-17 16:34:36 2021-12-17 17:01:22 0:26:46 0:11:33 0:15:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6567057 2021-12-17 03:23:20 2021-12-17 16:38:57 2021-12-17 17:09:49 0:30:52 0:18:37 0:12:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6567058 2021-12-17 03:23:21 2021-12-17 16:39:47 2021-12-17 17:07:37 0:27:50 0:12:09 0:15:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_fallocate} 3
pass 6567059 2021-12-17 03:23:22 2021-12-17 16:41:48 2021-12-17 17:06:36 0:24:48 0:12:14 0:12:34 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_huge_osdmap} 3
fail 6567060 2021-12-17 03:23:22 2021-12-17 16:41:48 2021-12-17 17:14:01 0:32:13 0:19:40 0:12:33 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
Failure Reason:

"2021-12-17T17:11:44.574658+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6567061 2021-12-17 03:23:23 2021-12-17 16:41:48 2021-12-17 17:14:36 0:32:48 0:20:13 0:12:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
Failure Reason:

"2021-12-17T17:13:10.336346+0000 mon.a (mon.0) 321 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6567062 2021-12-17 03:23:24 2021-12-17 16:42:19 2021-12-17 17:04:40 0:22:21 0:09:56 0:12:25 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc tasks/krbd_parent_overlap} 1
pass 6567063 2021-12-17 03:23:25 2021-12-17 16:44:59 2021-12-17 17:13:19 0:28:20 0:16:07 0:12:13 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6567064 2021-12-17 03:23:25 2021-12-17 16:45:10 2021-12-17 17:42:45 0:57:35 0:43:24 0:14:11 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_fio} 3
pass 6567065 2021-12-17 03:23:26 2021-12-17 16:47:30 2021-12-17 17:12:23 0:24:53 0:12:03 0:12:50 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/krbd_namespaces} 3
pass 6567066 2021-12-17 03:23:27 2021-12-17 16:47:41 2021-12-17 18:21:20 1:33:39 1:20:21 0:13:18 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6567067 2021-12-17 03:23:27 2021-12-17 16:49:21 2021-12-17 17:44:01 0:54:40 0:41:04 0:13:36 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6567068 2021-12-17 03:23:28 2021-12-17 16:51:52 2021-12-17 17:21:05 0:29:13 0:15:22 0:13:51 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6567069 2021-12-17 03:23:29 2021-12-17 16:52:12 2021-12-17 17:17:12 0:25:00 0:13:07 0:11:53 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6567070 2021-12-17 03:23:29 2021-12-17 16:53:03 2021-12-17 17:20:20 0:27:17 0:11:47 0:15:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/krbd_udev_netns} 3
pass 6567071 2021-12-17 03:23:30 2021-12-17 16:57:24 2021-12-17 17:19:41 0:22:17 0:10:13 0:12:04 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy tasks/krbd_read_only} 1
fail 6567072 2021-12-17 03:23:31 2021-12-17 16:57:24 2021-12-17 17:32:43 0:35:19 0:23:13 0:12:06 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
Failure Reason:

"2021-12-17T17:17:01.696415+0000 mon.a (mon.0) 306 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6567073 2021-12-17 03:23:32 2021-12-17 16:58:25 2021-12-17 17:23:00 0:24:35 0:12:21 0:12:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6567074 2021-12-17 03:23:32 2021-12-17 16:58:45 2021-12-17 17:23:05 0:24:20 0:11:54 0:12:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
fail 6567075 2021-12-17 03:23:33 2021-12-17 16:58:55 2021-12-17 17:31:20 0:32:25 0:20:07 0:12:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

"2021-12-17T17:28:52.529415+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6567076 2021-12-17 03:23:34 2021-12-17 16:59:06 2021-12-17 17:22:56 0:23:50 0:10:39 0:13:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6567077 2021-12-17 03:23:34 2021-12-17 17:06:44 2021-12-17 17:39:47 0:33:03 0:19:47 0:13:16 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2021-12-17T17:37:27.579170+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6567078 2021-12-17 03:23:35 2021-12-17 17:07:45 2021-12-17 17:41:28 0:33:43 0:19:29 0:14:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/many tasks/rbd_image_read} 3
Failure Reason:

"2021-12-17T17:40:16.365689+0000 mon.a (mon.0) 317 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6567079 2021-12-17 03:23:36 2021-12-17 17:09:56 2021-12-17 17:32:12 0:22:16 0:10:12 0:12:04 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} 1
pass 6567080 2021-12-17 03:23:37 2021-12-17 17:09:56 2021-12-17 17:40:55 0:30:59 0:17:39 0:13:20 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{prefer-crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
fail 6567081 2021-12-17 03:23:37 2021-12-17 17:09:56 2021-12-17 17:40:34 0:30:38 0:19:16 0:11:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} 3
Failure Reason:

"2021-12-17T17:39:30.571766+0000 mon.a (mon.0) 284 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6567082 2021-12-17 03:23:38 2021-12-17 17:10:27 2021-12-17 17:40:15 0:29:48 0:16:41 0:13:07 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
pass 6567083 2021-12-17 03:23:39 2021-12-17 17:12:07 2021-12-17 17:35:23 0:23:16 0:10:54 0:12:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6567084 2021-12-17 03:23:39 2021-12-17 17:12:28 2021-12-17 17:40:49 0:28:21 0:15:46 0:12:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy msgr-failures/few tasks/rbd_map_unmap} 3
pass 6567085 2021-12-17 03:23:40 2021-12-17 17:12:38 2021-12-17 17:37:39 0:25:01 0:12:19 0:12:42 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
fail 6567086 2021-12-17 03:23:41 2021-12-17 17:13:28 2021-12-17 17:46:14 0:32:46 0:19:49 0:12:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} 3
Failure Reason:

"2021-12-17T17:43:21.853080+0000 mon.a (mon.0) 371 : cluster [WRN] Health check failed: 3 osds down (OSD_DOWN)" in cluster log