Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 6659703 2022-02-03 22:41:06 2022-02-03 22:45:30 2022-02-03 23:05:40 0:20:10 0:09:01 0:11:09 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_deep_flatten} 1
pass 6659704 2022-02-03 22:41:07 2022-02-03 22:46:21 2022-02-03 23:16:38 0:30:17 0:18:58 0:11:19 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6659705 2022-02-03 22:41:08 2022-02-03 22:46:41 2022-02-03 23:28:42 0:42:01 0:28:26 0:13:35 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_fio} 3
pass 6659706 2022-02-03 22:41:09 2022-02-03 22:48:42 2022-02-03 23:19:46 0:31:04 0:18:20 0:12:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_data_pool} 3
fail 6659707 2022-02-03 22:41:10 2022-02-03 22:50:22 2022-02-03 23:14:02 0:23:40 0:09:47 0:13:53 smithi master krbd/rxbounce/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/krbd_rxbounce} 3
Failure Reason:

Command failed (workunit test rbd/krbd_rxbounce.sh) on smithi085 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=239f3b058ecd10b05894b75aab82b58c444cda60 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_rxbounce.sh'

fail 6659708 2022-02-03 22:41:11 2022-02-03 22:51:13 2022-02-04 03:28:17 4:37:04 4:22:24 0:14:40 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy-rxbounce} msgr-failures/few tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi125 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludenh7cgfb7 -r -- '-g auto -g blockdev -x clone'"

pass 6659709 2022-02-03 22:41:12 2022-02-03 22:54:34 2022-02-03 23:46:26 0:51:52 0:40:01 0:11:51 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/backoff thrashosds-health workloads/rbd_fio} 3
fail 6659710 2022-02-03 22:41:13 2022-02-03 22:54:54 2022-02-03 23:03:05 0:08:11 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/pre-single-major tasks/unmap} 2
Failure Reason:

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major

pass 6659711 2022-02-03 22:41:14 2022-02-03 22:55:44 2022-02-03 23:15:12 0:19:28 0:08:55 0:10:33 smithi master krbd/wac/sysfs/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf tasks/stable_writes} 1
fail 6659712 2022-02-03 22:41:15 2022-02-03 22:55:55 2022-02-03 23:27:36 0:31:41 0:19:24 0:12:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_exclusive_option} 3
Failure Reason:

"2022-02-03T23:24:29.523141+0000 mon.a (mon.0) 235 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6659713 2022-02-03 22:41:16 2022-02-03 22:56:15 2022-02-03 23:28:59 0:32:44 0:20:02 0:12:42 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
Failure Reason:

"2022-02-03T23:26:56.474694+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6659714 2022-02-03 22:41:17 2022-02-03 22:56:46 2022-02-03 23:20:52 0:24:06 0:11:44 0:12:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_fallocate} 3
fail 6659715 2022-02-03 22:41:18 2022-02-03 22:57:16 2022-02-03 23:29:28 0:32:12 0:20:04 0:12:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_huge_osdmap} 3
Failure Reason:

"2022-02-03T23:26:49.117777+0000 mon.a (mon.0) 245 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659716 2022-02-03 22:41:19 2022-02-03 22:57:26 2022-02-03 23:45:26 0:48:00 0:37:17 0:10:43 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6659717 2022-02-03 22:41:20 2022-02-03 22:57:47 2022-02-03 23:23:24 0:25:37 0:13:17 0:12:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6659718 2022-02-03 22:41:21 2022-02-03 23:00:07 2022-02-03 23:18:49 0:18:42 0:09:50 0:08:52 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_discard} 1
pass 6659719 2022-02-03 22:41:22 2022-02-03 23:00:08 2022-02-03 23:27:58 0:27:50 0:17:37 0:10:13 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6659720 2022-02-03 22:41:23 2022-02-03 23:00:38 2022-02-03 23:22:48 0:22:10 0:11:29 0:10:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_namespaces} 3
fail 6659721 2022-02-03 22:41:24 2022-02-03 23:01:28 2022-02-04 00:36:39 1:35:11 1:21:16 0:13:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2022-02-04T00:30:01.789485+0000 osd.3 (osd.3) 3 : cluster [WRN] 10 slow requests (by type [ 'started' : 3 'waiting for sub ops' : 7 ] most affected pool [ 'rbd' : 10 ])" in cluster log

pass 6659722 2022-02-03 22:41:25 2022-02-03 23:02:49 2022-02-03 23:54:37 0:51:48 0:37:17 0:14:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_enumerate} 3
fail 6659723 2022-02-03 22:41:26 2022-02-03 23:04:50 2022-02-03 23:37:08 0:32:18 0:19:16 0:13:02 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
Failure Reason:

"2022-02-03T23:24:45.301995+0000 mon.a (mon.0) 184 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6659724 2022-02-03 22:41:27 2022-02-03 23:05:40 2022-02-03 23:29:28 0:23:48 0:11:50 0:11:58 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6659725 2022-02-03 22:41:29 2022-02-03 23:05:50 2022-02-03 23:29:47 0:23:57 0:10:51 0:13:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netns} 3
pass 6659726 2022-02-03 22:41:30 2022-02-03 23:08:31 2022-02-03 23:27:54 0:19:23 0:08:32 0:10:51 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_huge_image} 1
fail 6659727 2022-02-03 22:41:31 2022-02-03 23:08:31 2022-02-03 23:39:10 0:30:39 0:19:01 0:11:38 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
Failure Reason:

"2022-02-03T23:38:00.852016+0000 mon.a (mon.0) 240 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659728 2022-02-03 22:41:32 2022-02-03 23:09:02 2022-02-03 23:33:17 0:24:15 0:11:38 0:12:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks} 3
pass 6659729 2022-02-03 22:41:32 2022-02-03 23:09:53 2022-02-03 23:31:22 0:21:29 0:10:50 0:10:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6659730 2022-02-03 22:41:34 2022-02-03 23:10:13 2022-02-03 23:35:57 0:25:44 0:15:26 0:10:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_concurrent} 3
pass 6659731 2022-02-03 22:41:35 2022-02-03 23:10:33 2022-02-03 23:32:12 0:21:39 0:10:42 0:10:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6659732 2022-02-03 22:41:36 2022-02-03 23:10:44 2022-02-03 23:40:21 0:29:37 0:18:45 0:10:52 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6659733 2022-02-03 22:41:37 2022-02-03 23:10:54 2022-02-03 23:34:32 0:23:38 0:10:37 0:13:01 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_image_read} 3
pass 6659734 2022-02-03 22:41:38 2022-02-03 23:12:45 2022-02-03 23:32:06 0:19:21 0:08:49 0:10:32 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_modprobe} 1
pass 6659735 2022-02-03 22:41:39 2022-02-03 23:12:45 2022-02-03 23:39:08 0:26:23 0:15:10 0:11:13 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6659736 2022-02-03 22:41:40 2022-02-03 23:13:45 2022-02-04 00:42:47 1:29:02 1:16:40 0:12:22 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6659737 2022-02-03 22:41:41 2022-02-03 23:14:06 2022-02-03 23:34:12 0:20:06 0:09:49 0:10:17 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_kernel} 3
fail 6659738 2022-02-03 22:41:42 2022-02-03 23:14:56 2022-02-03 23:46:10 0:31:14 0:18:48 0:12:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2022-02-03T23:34:11.812921+0000 mon.a (mon.0) 156 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

fail 6659739 2022-02-03 22:41:43 2022-02-03 23:15:57 2022-02-03 23:48:53 0:32:56 0:19:29 0:13:27 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2022-02-03T23:45:33.252564+0000 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659740 2022-02-03 22:41:44 2022-02-03 23:16:47 2022-02-03 23:44:22 0:27:35 0:15:03 0:12:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_map_unmap} 3
pass 6659741 2022-02-03 22:41:45 2022-02-03 23:18:58 2022-02-03 23:41:15 0:22:17 0:11:15 0:11:02 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6659742 2022-02-03 22:41:46 2022-02-03 23:19:48 2022-02-03 23:55:09 0:35:21 0:22:35 0:12:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_simple_big} 3
pass 6659743 2022-02-03 22:41:47 2022-02-03 23:20:29 2022-02-03 23:39:35 0:19:06 0:09:02 0:10:04 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_msgr_segments} 1
pass 6659744 2022-02-03 22:41:48 2022-02-03 23:20:29 2022-02-03 23:52:38 0:32:09 0:21:47 0:10:22 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6659745 2022-02-03 22:41:49 2022-02-03 23:20:59 2022-02-04 00:01:03 0:40:04 0:28:29 0:11:35 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_fio} 3
pass 6659746 2022-02-03 22:41:50 2022-02-03 23:21:30 2022-02-03 23:56:20 0:34:50 0:23:20 0:11:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_data_pool} 3
pass 6659747 2022-02-03 22:41:51 2022-02-03 23:22:50 2022-02-03 23:45:24 0:22:34 0:10:50 0:11:44 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6659748 2022-02-03 22:41:52 2022-02-03 23:23:31 2022-02-03 23:51:28 0:27:57 0:17:41 0:10:16 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6659749 2022-02-03 22:41:53 2022-02-03 23:24:01 2022-02-03 23:48:53 0:24:52 0:11:12 0:13:40 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_fallocate} 3
pass 6659750 2022-02-03 22:41:54 2022-02-03 23:27:42 2022-02-03 23:49:20 0:21:38 0:10:57 0:10:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6659751 2022-02-03 22:41:55 2022-02-03 23:28:02 2022-02-04 00:18:53 0:50:51 0:38:32 0:12:19 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6659752 2022-02-03 22:41:56 2022-02-03 23:28:43 2022-02-03 23:52:29 0:23:46 0:13:36 0:10:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6659753 2022-02-03 22:41:57 2022-02-03 23:28:53 2022-02-03 23:48:15 0:19:22 0:08:44 0:10:38 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_parent_overlap} 1
pass 6659754 2022-02-03 22:41:58 2022-02-03 23:28:53 2022-02-03 23:59:02 0:30:09 0:17:13 0:12:56 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6659755 2022-02-03 22:41:59 2022-02-03 23:29:04 2022-02-03 23:53:29 0:24:25 0:11:07 0:13:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_namespaces} 3
pass 6659756 2022-02-03 22:42:00 2022-02-03 23:29:34 2022-02-04 00:53:23 1:23:49 1:12:43 0:11:06 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6659757 2022-02-03 22:42:01 2022-02-03 23:29:35 2022-02-04 00:21:16 0:51:41 0:39:42 0:11:59 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6659758 2022-02-03 22:42:02 2022-02-03 23:29:55 2022-02-03 23:56:50 0:26:55 0:14:43 0:12:12 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
fail 6659759 2022-02-03 22:42:03 2022-02-03 23:31:26 2022-02-04 00:02:44 0:31:18 0:19:30 0:11:48 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
Failure Reason:

"2022-02-04T00:00:59.049925+0000 mon.a (mon.0) 231 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6659760 2022-02-03 22:42:04 2022-02-03 23:32:16 2022-02-04 00:02:58 0:30:42 0:19:15 0:11:27 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-02-04T00:01:02.355345+0000 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659761 2022-02-03 22:42:05 2022-02-03 23:33:26 2022-02-03 23:52:49 0:19:23 0:08:56 0:10:27 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_read_only} 1
pass 6659762 2022-02-03 22:42:06 2022-02-03 23:33:27 2022-02-04 00:03:46 0:30:19 0:18:29 0:11:50 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{secure} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6659763 2022-02-03 22:42:07 2022-02-03 23:34:17 2022-02-04 00:22:18 0:48:01 0:37:17 0:10:44 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{secure} thrashers/pggrow thrashosds-health workloads/rbd_fio} 3
pass 6659764 2022-02-03 22:42:08 2022-02-03 23:34:38 2022-02-03 23:57:22 0:22:44 0:10:50 0:11:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6659765 2022-02-03 22:42:09 2022-02-03 23:35:58 2022-02-03 23:58:41 0:22:43 0:10:12 0:12:31 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
dead 6659766 2022-02-03 22:42:10 2022-02-03 23:37:19 2022-02-04 02:38:43 3:01:24 2:48:17 0:13:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

SSH connection to smithi173 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

pass 6659767 2022-02-03 22:42:11 2022-02-03 23:39:09 2022-02-04 00:01:07 0:21:58 0:10:25 0:11:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6659768 2022-02-03 22:42:12 2022-02-03 23:39:20 2022-02-04 00:08:57 0:29:37 0:19:18 0:10:19 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2022-02-04T00:07:31.935663+0000 mon.a (mon.0) 229 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6659769 2022-02-03 22:42:13 2022-02-03 23:39:40 2022-02-04 00:02:05 0:22:25 0:11:16 0:11:09 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_image_read} 3
pass 6659770 2022-02-03 22:42:14 2022-02-03 23:40:31 2022-02-04 00:00:45 0:20:14 0:09:17 0:10:57 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_whole_object_zeroout} 1
fail 6659771 2022-02-03 22:42:15 2022-02-03 23:41:21 2022-02-04 00:13:49 0:32:28 0:19:15 0:13:13 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
Failure Reason:

"2022-02-04T00:11:48.340412+0000 mon.a (mon.0) 203 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

pass 6659772 2022-02-03 22:42:16 2022-02-03 23:44:32 2022-02-04 00:07:07 0:22:35 0:10:09 0:12:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_kernel} 3
pass 6659773 2022-02-03 22:42:17 2022-02-03 23:45:32 2022-02-04 00:11:21 0:25:49 0:15:34 0:10:15 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
fail 6659774 2022-02-03 22:42:18 2022-02-03 23:45:33 2022-02-04 00:16:24 0:30:51 0:19:45 0:11:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2022-02-04T00:14:52.700733+0000 mon.a (mon.0) 285 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659775 2022-02-03 22:42:19 2022-02-03 23:46:13 2022-02-04 00:12:11 0:25:58 0:15:16 0:10:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_map_unmap} 3
pass 6659776 2022-02-03 22:42:20 2022-02-03 23:46:33 2022-02-04 00:11:53 0:25:20 0:12:36 0:12:44 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6659777 2022-02-03 22:42:21 2022-02-03 23:48:24 2022-02-04 00:24:53 0:36:29 0:25:00 0:11:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_simple_big} 3
pass 6659778 2022-02-03 22:42:22 2022-02-03 23:48:54 2022-02-04 00:08:13 0:19:19 0:08:57 0:10:22 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_deep_flatten} 1
pass 6659779 2022-02-03 22:42:23 2022-02-03 23:48:55 2022-02-04 00:20:41 0:31:46 0:20:54 0:10:52 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6659780 2022-02-03 22:42:24 2022-02-03 23:49:25 2022-02-04 00:31:18 0:41:53 0:27:59 0:13:54 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_fio} 3
fail 6659781 2022-02-03 22:42:25 2022-02-03 23:51:36 2022-02-04 00:22:05 0:30:29 0:19:10 0:11:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_data_pool} 3
Failure Reason:

"2022-02-04T00:09:45.687530+0000 mon.a (mon.0) 132 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6659782 2022-02-03 22:42:26 2022-02-03 23:52:36 2022-02-04 00:11:57 0:19:21 0:08:49 0:10:32 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-off tasks/unmap} 2
pass 6659783 2022-02-03 22:42:27 2022-02-03 23:52:37 2022-02-04 00:24:49 0:32:12 0:21:22 0:10:50 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/many-resets} 3
pass 6659784 2022-02-03 22:42:28 2022-02-03 23:52:47 2022-02-04 00:17:41 0:24:54 0:11:31 0:13:23 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_exclusive_option} 3
pass 6659785 2022-02-03 22:42:29 2022-02-03 23:53:37 2022-02-04 00:24:06 0:30:29 0:18:44 0:11:45 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6659786 2022-02-03 22:42:30 2022-02-03 23:54:18 2022-02-04 00:16:23 0:22:05 0:10:54 0:11:11 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_fallocate} 3
pass 6659787 2022-02-03 22:42:31 2022-02-03 23:54:38 2022-02-04 00:19:57 0:25:19 0:11:54 0:13:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6659788 2022-02-03 22:42:32 2022-02-03 23:55:19 2022-02-04 00:45:54 0:50:35 0:37:13 0:13:22 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6659789 2022-02-03 22:42:33 2022-02-03 23:56:29 2022-02-04 00:22:14 0:25:45 0:14:02 0:11:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6659790 2022-02-03 22:42:34 2022-02-03 23:57:00 2022-02-04 00:18:53 0:21:53 0:09:45 0:12:08 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_discard} 1
pass 6659791 2022-02-03 22:42:35 2022-02-03 23:57:30 2022-02-04 00:26:23 0:28:53 0:17:07 0:11:46 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6659792 2022-02-03 22:42:36 2022-02-03 23:58:50 2022-02-04 01:33:22 1:34:32 1:23:46 0:10:46 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6659793 2022-02-03 22:42:37 2022-02-03 23:59:11 2022-02-04 00:21:53 0:22:42 0:11:38 0:11:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_namespaces} 3
pass 6659794 2022-02-03 22:42:38 2022-02-04 00:00:11 2022-02-04 01:30:35 1:30:24 1:18:03 0:12:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
pass 6659795 2022-02-03 22:42:39 2022-02-04 00:00:12 2022-02-04 00:49:23 0:49:11 0:37:16 0:11:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_enumerate} 3
pass 6659796 2022-02-03 22:42:40 2022-02-04 00:01:12 2022-02-04 00:26:43 0:25:31 0:15:12 0:10:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6659797 2022-02-03 22:42:41 2022-02-04 00:01:12 2022-02-04 00:23:45 0:22:33 0:11:28 0:11:05 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
pass 6659798 2022-02-03 22:42:42 2022-02-04 00:02:13 2022-02-04 00:24:09 0:21:56 0:10:56 0:11:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_netns} 3
pass 6659799 2022-02-03 22:42:44 2022-02-04 00:02:53 2022-02-04 00:22:21 0:19:28 0:08:55 0:10:33 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_huge_image} 1
pass 6659800 2022-02-03 22:42:45 2022-02-04 00:02:54 2022-02-04 00:36:25 0:33:31 0:22:03 0:11:28 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
fail 6659801 2022-02-03 22:42:46 2022-02-04 00:03:04 2022-02-04 00:33:23 0:30:19 0:19:16 0:11:03 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_udev_symlinks} 3
Failure Reason:

"2022-02-04T00:31:48.944184+0000 mon.a (mon.0) 236 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659802 2022-02-03 22:42:47 2022-02-04 00:03:54 2022-02-04 00:28:53 0:24:59 0:10:38 0:14:21 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6659803 2022-02-03 22:42:48 2022-02-04 00:07:15 2022-02-04 00:33:41 0:26:26 0:15:00 0:11:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_concurrent} 3
pass 6659804 2022-02-03 22:42:49 2022-02-04 00:08:16 2022-02-04 00:30:14 0:21:58 0:10:23 0:11:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_huge_tickets} 3
pass 6659805 2022-02-03 22:42:50 2022-02-04 00:09:06 2022-02-04 00:41:20 0:32:14 0:18:39 0:13:35 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6659806 2022-02-03 22:42:51 2022-02-04 00:11:27 2022-02-04 00:33:20 0:21:53 0:11:07 0:10:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_image_read} 3
pass 6659807 2022-02-03 22:42:52 2022-02-04 00:11:57 2022-02-04 00:31:19 0:19:22 0:09:00 0:10:22 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_modprobe} 1
pass 6659808 2022-02-03 22:42:53 2022-02-04 00:11:58 2022-02-04 00:47:48 0:35:50 0:23:59 0:11:51 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6659809 2022-02-03 22:42:54 2022-02-04 00:12:18 2022-02-04 00:35:08 0:22:50 0:10:17 0:12:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_kernel} 3
fail 6659810 2022-02-03 22:42:55 2022-02-04 00:13:59 2022-02-04 00:46:40 0:32:41 0:19:00 0:13:41 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

"2022-02-04T00:45:10.405464+0000 mon.a (mon.0) 272 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659811 2022-02-03 22:42:56 2022-02-04 00:16:29 2022-02-04 00:36:10 0:19:41 0:09:55 0:09:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_map_snapshot_io} 3
pass 6659812 2022-02-03 22:42:57 2022-02-04 00:16:30 2022-02-04 00:43:48 0:27:18 0:14:58 0:12:20 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_map_unmap} 3
pass 6659813 2022-02-03 22:42:58 2022-02-04 00:17:51 2022-02-04 00:42:55 0:25:04 0:11:38 0:13:26 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6659814 2022-02-03 22:42:59 2022-02-04 00:19:01 2022-02-04 00:58:36 0:39:35 0:25:38 0:13:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_simple_big} 3
pass 6659815 2022-02-03 22:43:00 2022-02-04 00:20:02 2022-02-04 00:39:18 0:19:16 0:08:42 0:10:34 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_msgr_segments} 1
pass 6659816 2022-02-03 22:43:01 2022-02-04 00:20:02 2022-02-04 00:53:32 0:33:30 0:21:41 0:11:49 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
fail 6659817 2022-02-03 22:43:02 2022-02-04 00:20:02 2022-02-04 00:52:43 0:32:41 0:19:01 0:13:40 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_fio} 3
Failure Reason:

"2022-02-04T00:49:59.244177+0000 mon.a (mon.0) 293 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659818 2022-02-03 22:43:03 2022-02-04 00:21:23 2022-02-04 00:55:26 0:34:03 0:22:39 0:11:24 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_data_pool} 3
fail 6659819 2022-02-03 22:43:04 2022-02-04 00:21:53 2022-02-04 05:12:09 4:50:16 4:37:37 0:12:39 smithi master krbd/singleton/{bluestore-bitmap conf ms_mode$/{legacy} msgr-failures/many tasks/rbd_xfstests} 4
Failure Reason:

Command failed on smithi189 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludegbya1gu7 -r -- '-g auto -g blockdev -x clone'"

pass 6659820 2022-02-03 22:43:05 2022-02-04 00:22:13 2022-02-04 01:54:13 1:32:00 1:20:43 0:11:17 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/backoff thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
pass 6659821 2022-02-03 22:43:06 2022-02-04 00:22:24 2022-02-04 00:45:42 0:23:18 0:11:52 0:11:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6659822 2022-02-03 22:43:07 2022-02-04 00:22:24 2022-02-04 00:53:38 0:31:14 0:19:09 0:12:05 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
pass 6659823 2022-02-03 22:43:08 2022-02-04 00:23:55 2022-02-04 00:45:47 0:21:52 0:11:48 0:10:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_fallocate} 3
pass 6659824 2022-02-03 22:43:09 2022-02-04 00:24:15 2022-02-04 00:48:18 0:24:03 0:11:59 0:12:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6659825 2022-02-03 22:43:10 2022-02-04 00:24:15 2022-02-04 01:15:04 0:50:49 0:38:03 0:12:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6659826 2022-02-03 22:43:11 2022-02-04 00:24:56 2022-02-04 00:50:37 0:25:41 0:14:48 0:10:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6659827 2022-02-03 22:43:12 2022-02-04 00:24:56 2022-02-04 00:44:05 0:19:09 0:08:58 0:10:11 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_parent_overlap} 1
pass 6659828 2022-02-03 22:43:13 2022-02-04 00:24:56 2022-02-04 00:56:02 0:31:06 0:18:39 0:12:27 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6659829 2022-02-03 22:43:14 2022-02-04 00:26:27 2022-02-04 00:48:26 0:21:59 0:11:26 0:10:33 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_namespaces} 3
pass 6659830 2022-02-03 22:43:15 2022-02-04 00:26:47 2022-02-04 01:54:46 1:27:59 1:14:24 0:13:35 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6659831 2022-02-03 22:43:16 2022-02-04 00:28:58 2022-02-04 01:19:45 0:50:47 0:39:48 0:10:59 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6659832 2022-02-03 22:43:17 2022-02-04 00:30:19 2022-02-04 00:56:58 0:26:39 0:15:13 0:11:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6659833 2022-02-03 22:43:18 2022-02-04 00:31:19 2022-02-04 00:56:47 0:25:28 0:13:41 0:11:47 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
pass 6659834 2022-02-03 22:43:19 2022-02-04 00:33:30 2022-02-04 00:54:58 0:21:28 0:11:18 0:10:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_netns} 3
pass 6659835 2022-02-03 22:43:20 2022-02-04 00:33:30 2022-02-04 00:52:55 0:19:25 0:09:16 0:10:09 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_read_only} 1
pass 6659836 2022-02-03 22:43:21 2022-02-04 00:33:31 2022-02-04 01:09:30 0:35:59 0:25:17 0:10:42 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6659837 2022-02-03 22:43:22 2022-02-04 00:33:51 2022-02-04 00:56:34 0:22:43 0:11:01 0:11:42 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6659838 2022-02-03 22:43:23 2022-02-04 00:35:11 2022-02-04 00:58:49 0:23:38 0:10:35 0:13:03 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6659839 2022-02-03 22:43:24 2022-02-04 00:35:32 2022-02-04 01:05:53 0:30:21 0:15:13 0:15:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_concurrent} 3
pass 6659840 2022-02-03 22:43:25 2022-02-04 00:36:12 2022-02-04 01:02:04 0:25:52 0:11:37 0:14:15 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_huge_tickets} 3
pass 6659841 2022-02-03 22:43:26 2022-02-04 00:36:33 2022-02-04 01:10:23 0:33:50 0:20:36 0:13:14 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
pass 6659842 2022-02-03 22:43:28 2022-02-04 00:36:43 2022-02-04 01:05:19 0:28:36 0:11:31 0:17:05 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_image_read} 3
pass 6659843 2022-02-03 22:43:29 2022-02-04 00:41:24 2022-02-04 01:02:47 0:21:23 0:09:31 0:11:52 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_whole_object_zeroout} 1
pass 6659844 2022-02-03 22:43:30 2022-02-04 00:41:24 2022-02-04 01:09:44 0:28:20 0:14:15 0:14:05 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6659845 2022-02-03 22:43:31 2022-02-04 00:42:55 2022-02-04 01:33:03 0:50:08 0:37:32 0:12:36 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/mon-thrasher thrashosds-health workloads/rbd_fio} 3
pass 6659846 2022-02-03 22:43:32 2022-02-04 00:42:55 2022-02-04 01:05:56 0:23:01 0:10:36 0:12:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_kernel} 3
dead 6659847 2022-02-03 22:43:33 2022-02-04 00:43:56 2022-02-04 01:00:57 0:17:01 0:04:06 0:12:55 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
Failure Reason:

Failure object was: {'smithi040.front.sepia.ceph.com': {'cache_updated': False, 'stdout': '', 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6309 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'rc': 100, 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': True, 'force_apt_get': False, 'policy_rc_d': None, 'package': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'autoclean': False, 'install_recommends': None, 'name': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': None, 'update_cache': None, 'default_release': None, 'only_upgrade': False, 'deb': None, 'cache_valid_time': 0}}, 'cache_update_time': 1643936211, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" --force-yes install \'qemu-system-x86\' \'git-core\'\' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6309 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'stdout_lines': [], 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6309 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], '_ansible_no_log': False, 'changed': False}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)yaml.representer.RepresenterError: ('cannot represent an object', 'cache_update_time')

pass 6659848 2022-02-03 22:43:34 2022-02-04 00:45:46 2022-02-04 01:09:17 0:23:31 0:12:23 0:11:08 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_snapshot_io} 3
pass 6659849 2022-02-03 22:43:35 2022-02-04 00:45:57 2022-02-04 01:13:26 0:27:29 0:17:15 0:10:14 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_map_unmap} 3
pass 6659850 2022-02-03 22:43:36 2022-02-04 00:45:57 2022-02-04 01:12:59 0:27:02 0:13:05 0:13:57 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6659851 2022-02-03 22:43:37 2022-02-04 00:46:48 2022-02-04 01:25:30 0:38:42 0:24:56 0:13:46 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_simple_big} 3
pass 6659852 2022-02-03 22:43:38 2022-02-04 00:47:58 2022-02-04 01:11:03 0:23:05 0:11:07 0:11:58 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy} tasks/krbd_deep_flatten} 1
pass 6659853 2022-02-03 22:43:39 2022-02-04 00:47:58 2022-02-04 01:28:17 0:40:19 0:26:32 0:13:47 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6659854 2022-02-03 22:43:40 2022-02-04 00:48:19 2022-02-04 01:29:55 0:41:36 0:29:35 0:12:01 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_fio} 3
pass 6659855 2022-02-03 22:43:41 2022-02-04 00:48:29 2022-02-04 01:21:34 0:33:05 0:19:55 0:13:10 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_data_pool} 3
pass 6659856 2022-02-03 22:43:42 2022-02-04 00:49:30 2022-02-04 01:14:25 0:24:55 0:09:08 0:15:47 smithi master krbd/unmap/{ceph/ceph clusters/separate-client conf filestore-xfs kernels/single-major-on tasks/unmap} 2
pass 6659857 2022-02-03 22:43:43 2022-02-04 00:50:40 2022-02-04 01:27:59 0:37:19 0:20:38 0:16:41 smithi master krbd/wac/wac/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf tasks/wac verify/no-resets} 3
pass 6659858 2022-02-03 22:43:44 2022-02-04 00:52:51 2022-02-04 01:21:16 0:28:25 0:11:19 0:17:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_exclusive_option} 3
pass 6659859 2022-02-03 22:43:45 2022-02-04 00:53:31 2022-02-04 01:27:16 0:33:45 0:18:35 0:15:10 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/rbd_workunit_kernel_untar_build} 3
pass 6659860 2022-02-03 22:43:46 2022-02-04 00:53:42 2022-02-04 01:21:13 0:27:31 0:11:27 0:16:04 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_fallocate} 3
pass 6659861 2022-02-03 22:43:47 2022-02-04 00:53:42 2022-02-04 01:20:23 0:26:41 0:11:04 0:15:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_huge_osdmap} 3
pass 6659862 2022-02-03 22:43:48 2022-02-04 00:55:02 2022-02-04 01:49:05 0:54:03 0:36:53 0:17:10 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_dbench} 3
pass 6659863 2022-02-03 22:43:49 2022-02-04 00:55:33 2022-02-04 01:25:37 0:30:04 0:13:23 0:16:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_latest_osdmap_on_map} 3
pass 6659864 2022-02-03 22:43:50 2022-02-04 00:56:03 2022-02-04 01:21:26 0:25:23 0:09:41 0:15:42 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_discard} 1
pass 6659865 2022-02-03 22:43:51 2022-02-04 00:56:04 2022-02-04 01:28:01 0:31:57 0:16:17 0:15:40 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc-rxbounce} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
pass 6659866 2022-02-03 22:43:52 2022-02-04 00:56:44 2022-02-04 01:24:19 0:27:35 0:12:06 0:15:29 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/krbd_namespaces} 3
fail 6659867 2022-02-03 22:43:53 2022-02-04 00:56:54 2022-02-04 01:32:47 0:35:53 0:19:21 0:16:32 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_ffsb} 3
Failure Reason:

"2022-02-04T01:30:44.576878+0000 mon.a (mon.0) 287 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 6659868 2022-02-03 22:43:54 2022-02-04 00:57:05 2022-02-04 01:33:29 0:36:24 0:19:31 0:16:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_enumerate} 3
Failure Reason:

"2022-02-04T01:21:26.943908+0000 mon.a (mon.0) 187 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6659869 2022-02-03 22:43:55 2022-02-04 00:58:45 2022-02-04 01:30:24 0:31:39 0:15:21 0:16:18 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/krbd_udev_netlink_enobufs} 3
pass 6659870 2022-02-03 22:43:56 2022-02-04 00:58:56 2022-02-04 01:26:16 0:27:20 0:12:34 0:14:46 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress} 3
fail 6659871 2022-02-03 22:43:58 2022-02-04 01:01:06 2022-02-04 01:33:35 0:32:29 0:19:32 0:12:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_udev_netns} 3
Failure Reason:

"2022-02-04T01:32:02.594665+0000 mon.a (mon.0) 194 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659872 2022-02-03 22:43:59 2022-02-04 01:02:07 2022-02-04 01:23:10 0:21:03 0:09:31 0:11:32 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc} tasks/krbd_huge_image} 1
pass 6659873 2022-02-03 22:44:00 2022-02-04 01:02:07 2022-02-04 01:37:22 0:35:15 0:20:37 0:14:38 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-1-client} 3
pass 6659874 2022-02-03 22:44:01 2022-02-04 01:05:28 2022-02-04 02:36:00 1:30:32 1:19:05 0:11:27 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{crc-rxbounce} thrashers/pggrow thrashosds-health workloads/rbd_workunit_suites_ffsb} 3
fail 6659875 2022-02-03 22:44:02 2022-02-04 01:05:58 2022-02-04 01:35:47 0:29:49 0:19:12 0:10:37 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_udev_symlinks} 3
Failure Reason:

"2022-02-04T01:33:49.944424+0000 mon.a (mon.0) 205 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659876 2022-02-03 22:44:03 2022-02-04 01:05:59 2022-02-04 01:30:47 0:24:48 0:10:13 0:14:35 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsstress_ext4} 3
pass 6659877 2022-02-03 22:44:04 2022-02-04 01:09:19 2022-02-04 01:35:15 0:25:56 0:14:59 0:10:57 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_concurrent} 3
fail 6659878 2022-02-03 22:44:05 2022-02-04 01:09:40 2022-02-04 01:40:40 0:31:00 0:19:01 0:11:59 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_huge_tickets} 3
Failure Reason:

"2022-02-04T01:39:31.765902+0000 mon.a (mon.0) 274 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659879 2022-02-03 22:44:06 2022-02-04 01:09:50 2022-02-04 01:40:18 0:30:28 0:18:46 0:11:42 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_fsx} 3
pass 6659880 2022-02-03 22:44:07 2022-02-04 01:10:31 2022-02-04 01:35:36 0:25:05 0:10:11 0:14:54 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_image_read} 3
pass 6659881 2022-02-03 22:44:08 2022-02-04 01:13:01 2022-02-04 01:30:00 0:16:59 0:08:35 0:08:24 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_modprobe} 1
pass 6659882 2022-02-03 22:44:09 2022-02-04 01:13:02 2022-02-04 01:44:59 0:31:57 0:20:19 0:11:38 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-3-client} 3
pass 6659883 2022-02-03 22:44:10 2022-02-04 01:13:32 2022-02-04 01:36:10 0:22:38 0:10:19 0:12:19 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_kernel} 3
pass 6659884 2022-02-03 22:44:11 2022-02-04 01:14:33 2022-02-04 01:43:19 0:28:46 0:17:19 0:11:27 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_iozone} 3
pass 6659885 2022-02-03 22:44:12 2022-02-04 01:15:13 2022-02-04 01:41:10 0:25:57 0:10:01 0:15:56 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_snapshot_io} 3
dead 6659886 2022-02-03 22:44:14 2022-02-04 01:19:54 2022-02-04 01:35:46 0:15:52 0:04:03 0:11:49 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_map_unmap} 3
Failure Reason:

Failure object was: {'smithi033.front.sepia.ceph.com': {'cache_updated': False, 'stdout': '', 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6310 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'rc': 100, 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': True, 'force_apt_get': False, 'policy_rc_d': None, 'package': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'autoclean': False, 'install_recommends': None, 'name': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': None, 'update_cache': None, 'default_release': None, 'only_upgrade': False, 'deb': None, 'cache_valid_time': 0}}, 'cache_update_time': 1643938294, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" --force-yes install \'qemu-system-x86\' \'git-core\'\' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6310 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'stdout_lines': [], 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6310 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], '_ansible_no_log': False, 'changed': False}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)yaml.representer.RepresenterError: ('cannot represent an object', 'cache_update_time')

pass 6659887 2022-02-03 22:44:15 2022-02-04 01:20:24 2022-02-04 01:42:44 0:22:20 0:11:36 0:10:44 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_trivial_sync} 3
pass 6659888 2022-02-03 22:44:16 2022-02-04 01:21:15 2022-02-04 01:55:03 0:33:48 0:22:22 0:11:26 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_simple_big} 3
pass 6659889 2022-02-03 22:44:17 2022-02-04 01:21:25 2022-02-04 01:41:36 0:20:11 0:08:39 0:11:32 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_msgr_segments} 1
pass 6659890 2022-02-03 22:44:18 2022-02-04 01:21:36 2022-02-04 01:59:21 0:37:45 0:26:44 0:11:01 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{crc} objectstore/filestore-xfs striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-1-client} 3
pass 6659891 2022-02-03 22:44:19 2022-02-04 01:21:36 2022-02-04 02:01:56 0:40:20 0:27:35 0:12:45 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_fio} 3
fail 6659892 2022-02-03 22:44:20 2022-02-04 01:24:27 2022-02-04 01:57:19 0:32:52 0:19:20 0:13:32 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_data_pool} 3
Failure Reason:

"2022-02-04T01:54:18.389238+0000 mon.a (mon.0) 233 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659893 2022-02-03 22:44:21 2022-02-04 01:25:37 2022-02-04 01:47:06 0:21:29 0:11:22 0:10:07 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_exclusive_option} 3
pass 6659894 2022-02-03 22:44:22 2022-02-04 01:25:38 2022-02-04 01:55:32 0:29:54 0:19:15 0:10:39 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_kernel_untar_build} 3
dead 6659895 2022-02-03 22:44:23 2022-02-04 01:26:18 2022-02-04 01:42:53 0:16:35 0:04:04 0:12:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_fallocate} 3
Failure Reason:

Failure object was: {'smithi142.front.sepia.ceph.com': {'cache_updated': False, 'stdout': '', 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6402 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'rc': 100, 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': True, 'force_apt_get': False, 'policy_rc_d': None, 'package': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'autoclean': False, 'install_recommends': None, 'name': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': None, 'update_cache': None, 'default_release': None, 'only_upgrade': False, 'deb': None, 'cache_valid_time': 0}}, 'cache_update_time': 1643938731, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" --force-yes install \'qemu-system-x86\' \'git-core\'\' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6402 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'stdout_lines': [], 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6402 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], '_ansible_no_log': False, 'changed': False}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)yaml.representer.RepresenterError: ('cannot represent an object', 'cache_update_time')

pass 6659896 2022-02-03 22:44:24 2022-02-04 01:27:18 2022-02-04 01:49:29 0:22:11 0:11:16 0:10:55 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/krbd_huge_osdmap} 3
pass 6659897 2022-02-03 22:44:25 2022-02-04 01:28:09 2022-02-04 02:17:28 0:49:19 0:38:36 0:10:43 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_suites_dbench} 3
pass 6659898 2022-02-03 22:44:26 2022-02-04 01:28:09 2022-02-04 01:54:21 0:26:12 0:13:59 0:12:13 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/krbd_latest_osdmap_on_map} 3
pass 6659899 2022-02-03 22:44:27 2022-02-04 01:28:20 2022-02-04 01:47:28 0:19:08 0:09:28 0:09:40 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/crc$/{crc-rxbounce} tasks/krbd_parent_overlap} 1
pass 6659900 2022-02-03 22:44:28 2022-02-04 01:28:20 2022-02-04 01:57:30 0:29:10 0:15:53 0:13:17 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{legacy-rxbounce} objectstore/bluestore-bitmap striping/default/{msgr-failures/few randomized-striping-off} tasks/fsx-3-client} 3
pass 6659901 2022-02-03 22:44:29 2022-02-04 01:30:00 2022-02-04 02:20:03 0:50:03 0:38:27 0:11:36 smithi master krbd/thrash/{bluestore-bitmap ceph/ceph clusters/fixed-3 conf ms_mode$/{legacy} thrashers/upmap thrashosds-health workloads/rbd_fio} 3
pass 6659902 2022-02-03 22:44:30 2022-02-04 01:30:31 2022-02-04 01:52:52 0:22:21 0:10:56 0:11:25 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/krbd_namespaces} 3
pass 6659903 2022-02-03 22:44:31 2022-02-04 01:30:41 2022-02-04 02:54:19 1:23:38 1:11:28 0:12:10 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy} msgr-failures/few tasks/rbd_workunit_suites_ffsb} 3
pass 6659904 2022-02-03 22:44:32 2022-02-04 01:30:52 2022-02-04 02:26:17 0:55:25 0:43:19 0:12:06 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/krbd_udev_enumerate} 3
pass 6659905 2022-02-03 22:44:33 2022-02-04 01:32:52 2022-02-04 01:59:09 0:26:17 0:14:50 0:11:27 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/krbd_udev_netlink_enobufs} 3
pass 6659906 2022-02-03 22:44:34 2022-02-04 01:33:13 2022-02-04 01:59:34 0:26:21 0:12:31 0:13:50 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/many tasks/rbd_workunit_suites_fsstress} 3
fail 6659907 2022-02-03 22:44:35 2022-02-04 01:33:23 2022-02-04 02:06:08 0:32:45 0:19:02 0:13:43 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc-rxbounce} msgr-failures/many tasks/krbd_udev_netns} 3
Failure Reason:

"2022-02-04T02:04:30.617174+0000 mon.a (mon.0) 317 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659908 2022-02-03 22:44:36 2022-02-04 01:33:33 2022-02-04 01:52:24 0:18:51 0:08:59 0:09:52 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/legacy$/{legacy-rxbounce} tasks/krbd_read_only} 1
pass 6659909 2022-02-03 22:44:37 2022-02-04 01:33:34 2022-02-04 02:07:05 0:33:31 0:21:35 0:11:56 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/object-map ms_mode$/{secure} objectstore/filestore-xfs striping/fancy/{msgr-failures/few randomized-striping-on} tasks/fsx-1-client} 3
pass 6659910 2022-02-03 22:44:38 2022-02-04 01:33:44 2022-02-04 01:58:56 0:25:12 0:11:19 0:13:53 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/few tasks/krbd_udev_symlinks} 3
pass 6659911 2022-02-03 22:44:39 2022-02-04 01:35:24 2022-02-04 02:00:16 0:24:52 0:10:17 0:14:35 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc-rxbounce} msgr-failures/few tasks/rbd_workunit_suites_fsstress_ext4} 3
dead 6659912 2022-02-03 22:44:40 2022-02-04 01:35:45 2022-02-04 01:53:30 0:17:45 0:04:04 0:13:41 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_concurrent} 3
Failure Reason:

Failure object was: {'smithi159.front.sepia.ceph.com': {'cache_updated': False, 'stdout': '', 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6405 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'rc': 100, 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': True, 'force_apt_get': False, 'policy_rc_d': None, 'package': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'autoclean': False, 'install_recommends': None, 'name': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': None, 'update_cache': None, 'default_release': None, 'only_upgrade': False, 'deb': None, 'cache_valid_time': 0}}, 'cache_update_time': 1643939375, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" --force-yes install \'qemu-system-x86\' \'git-core\'\' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6405 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'stdout_lines': [], 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 6405 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], '_ansible_no_log': False, 'changed': False}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)yaml.representer.RepresenterError: ('cannot represent an object', 'cache_update_time')

pass 6659913 2022-02-03 22:44:41 2022-02-04 01:35:55 2022-02-04 01:59:07 0:23:12 0:10:42 0:12:30 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/few tasks/rbd_huge_tickets} 3
fail 6659914 2022-02-03 22:44:42 2022-02-04 01:35:56 2022-02-04 02:07:53 0:31:57 0:19:16 0:12:41 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/legacy$/{legacy-rxbounce} msgr-failures/many tasks/rbd_workunit_suites_fsx} 3
Failure Reason:

"2022-02-04T02:06:01.546014+0000 mon.a (mon.0) 310 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659915 2022-02-03 22:44:43 2022-02-04 01:36:16 2022-02-04 02:01:16 0:25:00 0:10:38 0:14:22 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy} msgr-failures/many tasks/rbd_image_read} 3
pass 6659916 2022-02-03 22:44:44 2022-02-04 01:37:27 2022-02-04 02:00:17 0:22:50 0:09:14 0:13:36 smithi master krbd/basic/{bluestore-bitmap ceph/ceph clusters/fixed-1 conf ms_mode/secure tasks/krbd_whole_object_zeroout} 1
fail 6659917 2022-02-03 22:44:46 2022-02-04 01:40:27 2022-02-04 02:13:22 0:32:55 0:19:17 0:13:38 smithi master krbd/fsx/{ceph/ceph clusters/3-node conf features/no-object-map ms_mode$/{crc} objectstore/bluestore-bitmap striping/default/{msgr-failures/many randomized-striping-off} tasks/fsx-3-client} 3
Failure Reason:

"2022-02-04T02:10:11.489370+0000 mon.a (mon.0) 230 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659918 2022-02-03 22:44:47 2022-02-04 02:02:41 601 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/few tasks/rbd_kernel} 3
pass 6659919 2022-02-03 22:44:48 2022-02-04 01:41:18 2022-02-04 02:09:38 0:28:20 0:15:40 0:12:40 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/secure msgr-failures/few tasks/rbd_workunit_suites_iozone} 3
fail 6659920 2022-02-03 22:44:49 2022-02-04 01:41:39 2022-02-04 02:14:18 0:32:39 0:19:04 0:13:35 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_map_snapshot_io} 3
Failure Reason:

"2022-02-04T02:11:11.924638+0000 mon.a (mon.0) 212 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 6659921 2022-02-03 22:44:50 2022-02-04 01:42:49 2022-02-04 02:08:23 0:25:34 0:15:03 0:10:31 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/legacy$/{legacy-rxbounce} msgr-failures/few tasks/rbd_map_unmap} 3
pass 6659922 2022-02-03 22:44:51 2022-02-04 01:42:59 2022-02-04 02:07:36 0:24:37 0:12:43 0:11:54 smithi master krbd/rbd/{bluestore-bitmap clusters/fixed-3 conf ms_mode/crc$/{crc} msgr-failures/many tasks/rbd_workunit_trivial_sync} 3
pass 6659923 2022-02-03 22:44:52 2022-02-04 01:43:30 2022-02-04 02:22:35 0:39:05 0:26:05 0:13:00 smithi master krbd/rbd-nomount/{bluestore-bitmap clusters/fixed-3 conf install/ceph ms_mode/secure msgr-failures/many tasks/rbd_simple_big} 3