User | Scheduled | Started | Updated | Runtime | Suite | Branch | Machine Type | Revision | Pass | Fail | Dead |
---|---|---|---|---|---|---|---|---|---|---|---|
teuthology | 2020-04-01 05:15:02 | 2020-04-04 06:45:59 | 2020-04-04 19:20:21 | 12:34:22 | krbd | mimic | smithi | 36ac2fb | 46 | 17 | 4 |
Status | Job ID | Links | Posted | Started | Updated | Runtime |
Duration |
In Waiting |
Machine | Teuthology Branch | OS Type | OS Version | Description | Nodes |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
pass | 4912183 | 2020-04-01 05:15:23 | 2020-04-04 06:45:59 | 2020-04-04 07:01:58 | 0:15:59 | 0:07:55 | 0:08:04 | smithi | master | krbd/basic/{ceph/ceph.yaml clusters/fixed-1.yaml conf.yaml tasks/krbd_huge_image.yaml} | 1 | |||
pass | 4912184 | 2020-04-01 05:15:24 | 2020-04-04 06:48:11 | 2020-04-04 08:32:12 | 1:44:01 | 0:28:53 | 1:15:08 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/few.yaml tasks/rbd_fio.yaml} | 3 | |||
fail | 4912185 | 2020-04-01 05:15:25 | 2020-04-04 06:48:11 | 2020-04-04 10:24:14 | 3:36:03 | 3:11:20 | 0:24:43 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/krbd_data_pool.yaml} | 3 | |||
Failure Reason:
Command failed (workunit test rbd/krbd_data_pool.sh) on smithi130 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36ac2fb654d2cf4adb279402ce44a37699a8d949 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_data_pool.sh' |
||||||||||||||
fail | 4912186 | 2020-04-01 05:15:26 | 2020-04-04 06:48:11 | 2020-04-04 13:14:18 | 6:26:07 | 2:14:17 | 4:11:50 | smithi | master | krbd/singleton/{conf.yaml msgr-failures/few.yaml tasks/rbd_xfstests.yaml} | 4 | |||
Failure Reason:
Command failed on smithi171 with status 2: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludef_yUZL -r -- '-g auto -g blockdev -x clone'" |
||||||||||||||
pass | 4912187 | 2020-04-01 05:15:27 | 2020-04-04 06:48:11 | 2020-04-04 07:48:11 | 1:00:00 | 0:42:48 | 0:17:12 | smithi | master | krbd/thrash/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml thrashers/backoff.yaml thrashosds-health.yaml workloads/rbd_fio.yaml} | 3 | |||
pass | 4912188 | 2020-04-01 05:15:27 | 2020-04-04 06:49:29 | 2020-04-04 07:09:29 | 0:20:00 | 0:08:15 | 0:11:45 | smithi | master | krbd/unmap/{ceph/ceph.yaml clusters/separate-client.yaml conf.yaml filestore-xfs.yaml kernels/pre-single-major.yaml tasks/unmap.yaml} | 2 | |||
pass | 4912189 | 2020-04-01 05:15:28 | 2020-04-04 06:49:29 | 2020-04-04 07:07:29 | 0:18:00 | 0:07:41 | 0:10:19 | smithi | master | krbd/wac/sysfs/{ceph/ceph.yaml clusters/fixed-1.yaml conf.yaml tasks/stable_pages_required.yaml} | 1 | |||
pass | 4912190 | 2020-04-01 05:15:29 | 2020-04-04 06:49:40 | 2020-04-04 07:15:39 | 0:25:59 | 0:11:54 | 0:14:05 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/krbd_exclusive_option.yaml} | 3 | |||
pass | 4912191 | 2020-04-01 05:15:30 | 2020-04-04 06:49:42 | 2020-04-04 07:29:42 | 0:40:00 | 0:14:07 | 0:25:53 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/many.yaml tasks/rbd_workunit_kernel_untar_build.yaml} | 3 | |||
pass | 4912192 | 2020-04-01 05:15:31 | 2020-04-04 06:49:53 | 2020-04-04 09:43:55 | 2:54:02 | 0:10:10 | 2:43:52 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/krbd_fallocate.yaml} | 3 | |||
pass | 4912193 | 2020-04-01 05:15:32 | 2020-04-04 06:51:54 | 2020-04-04 08:19:54 | 1:28:00 | 0:12:51 | 1:15:09 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/krbd_latest_osdmap_on_map.yaml} | 3 | |||
pass | 4912194 | 2020-04-01 05:15:33 | 2020-04-04 06:51:54 | 2020-04-04 09:15:56 | 2:24:02 | 0:40:21 | 1:43:41 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/few.yaml tasks/rbd_workunit_suites_dbench.yaml} | 3 | |||
fail | 4912195 | 2020-04-01 05:15:33 | 2020-04-04 06:51:54 | 2020-04-04 07:51:54 | 1:00:00 | 0:37:01 | 0:22:59 | smithi | master | krbd/thrash/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml thrashers/mon-thrasher.yaml thrashosds-health.yaml workloads/rbd_workunit_suites_ffsb.yaml} | 3 | |||
Failure Reason:
"2020-04-04 07:20:09.906320 mon.b mon.0 172.21.15.69:6789/0 100 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log |
||||||||||||||
pass | 4912196 | 2020-04-01 05:15:34 | 2020-04-04 06:51:54 | 2020-04-04 07:37:54 | 0:46:00 | 0:20:01 | 0:25:59 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/krbd_udev_enumerate.yaml} | 3 | |||
fail | 4912197 | 2020-04-01 05:15:35 | 2020-04-04 06:52:01 | 2020-04-04 07:58:02 | 1:06:01 | 0:52:22 | 0:13:39 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/many.yaml tasks/rbd_workunit_suites_ffsb.yaml} | 3 | |||
Failure Reason:
"2020-04-04 07:11:59.340666 mon.b mon.0 172.21.15.86:6789/0 124 : cluster [WRN] Health check failed: 14 slow ops, oldest one blocked for 33 sec, daemons [osd.3,osd.6] have slow ops. (SLOW_OPS)" in cluster log |
||||||||||||||
pass | 4912198 | 2020-04-01 05:15:36 | 2020-04-04 06:52:19 | 2020-04-04 07:52:19 | 1:00:00 | 0:14:40 | 0:45:20 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/krbd_udev_netlink_enobufs.yaml} | 3 | |||
pass | 4912199 | 2020-04-01 05:15:37 | 2020-04-04 06:53:50 | 2020-04-04 07:11:49 | 0:17:59 | 0:07:57 | 0:10:02 | smithi | master | krbd/basic/{ceph/ceph.yaml clusters/fixed-1.yaml conf.yaml tasks/krbd_msgr_segments.yaml} | 1 | |||
pass | 4912200 | 2020-04-01 05:15:37 | 2020-04-04 06:53:50 | 2020-04-04 07:31:50 | 0:38:00 | 0:14:28 | 0:23:32 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/rbd_concurrent.yaml} | 3 | |||
pass | 4912201 | 2020-04-01 05:15:38 | 2020-04-04 06:53:50 | 2020-04-04 07:25:50 | 0:32:00 | 0:09:11 | 0:22:49 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/few.yaml tasks/rbd_workunit_suites_fsstress.yaml} | 3 | |||
pass | 4912202 | 2020-04-01 05:15:39 | 2020-04-04 06:53:50 | 2020-04-04 07:19:50 | 0:26:00 | 0:09:31 | 0:16:29 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/rbd_huge_tickets.yaml} | 3 | |||
pass | 4912203 | 2020-04-01 05:15:40 | 2020-04-04 06:54:18 | 2020-04-04 08:26:19 | 1:32:01 | 0:47:23 | 0:44:38 | smithi | master | krbd/thrash/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/rbd_fio.yaml} | 3 | |||
pass | 4912204 | 2020-04-01 05:15:41 | 2020-04-02 23:26:04 | 2020-04-03 00:06:04 | 0:40:00 | 0:09:30 | 0:30:30 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/many.yaml tasks/rbd_workunit_suites_fsstress_ext4.yaml} | 3 | |||
fail | 4912205 | 2020-04-01 05:15:42 | 2020-04-04 06:55:55 | 2020-04-04 12:10:00 | 5:14:05 | 3:11:28 | 2:02:37 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/rbd_image_read.yaml} | 3 | |||
Failure Reason:
Command failed (workunit test rbd/image_read.sh) on smithi142 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36ac2fb654d2cf4adb279402ce44a37699a8d949 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/image_read.sh' |
||||||||||||||
pass | 4912206 | 2020-04-01 05:15:42 | 2020-04-04 06:56:02 | 2020-04-04 07:52:02 | 0:56:00 | 0:09:06 | 0:46:54 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/rbd_kernel.yaml} | 3 | |||
pass | 4912207 | 2020-04-01 05:15:43 | 2020-04-04 06:57:58 | 2020-04-04 07:23:57 | 0:25:59 | 0:14:56 | 0:11:03 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/few.yaml tasks/rbd_workunit_suites_fsx.yaml} | 3 | |||
pass | 4912208 | 2020-04-01 05:15:44 | 2020-04-04 06:59:37 | 2020-04-04 07:17:37 | 0:18:00 | 0:08:42 | 0:09:18 | smithi | master | krbd/unmap/{ceph/ceph.yaml clusters/separate-client.yaml conf.yaml filestore-xfs.yaml kernels/single-major-off.yaml tasks/unmap.yaml} | 2 | |||
pass | 4912209 | 2020-04-01 05:15:45 | 2020-04-04 06:59:37 | 2020-04-04 07:29:37 | 0:30:00 | 0:16:25 | 0:13:35 | smithi | master | krbd/wac/wac/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml tasks/wac.yaml verify/many-resets.yaml} | 3 | |||
dead | 4912210 | 2020-04-01 05:15:46 | 2020-04-04 06:59:43 | 2020-04-04 19:02:10 | 12:02:27 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/rbd_kfsx.yaml} | 3 | |||||
pass | 4912211 | 2020-04-01 05:15:46 | 2020-04-04 06:59:49 | 2020-04-04 08:13:49 | 1:14:00 | 0:43:04 | 0:30:56 | smithi | master | krbd/thrash/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml thrashers/upmap.yaml thrashosds-health.yaml workloads/rbd_workunit_suites_ffsb.yaml} | 3 | |||
fail | 4912212 | 2020-04-01 05:15:47 | 2020-04-04 06:59:50 | 2020-04-04 08:25:51 | 1:26:01 | 0:19:26 | 1:06:35 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/many.yaml tasks/rbd_workunit_suites_iozone.yaml} | 3 | |||
Failure Reason:
"2020-04-04 08:13:55.812854 mon.a mon.0 172.21.15.2:6789/0 164 : cluster [WRN] Health check failed: 34 slow ops, oldest one blocked for 30 sec, osd.1 has slow ops (SLOW_OPS)" in cluster log |
||||||||||||||
pass | 4912213 | 2020-04-01 05:15:48 | 2020-04-04 06:59:54 | 2020-04-04 07:57:54 | 0:58:00 | 0:09:25 | 0:48:35 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/rbd_map_snapshot_io.yaml} | 3 | |||
dead | 4912214 | 2020-04-01 05:15:49 | 2020-04-04 06:59:55 | 2020-04-04 19:02:22 | 12:02:27 | smithi | master | krbd/basic/{ceph/ceph.yaml clusters/fixed-1.yaml conf.yaml tasks/krbd_parent_overlap.yaml} | 1 | |||||
fail | 4912215 | 2020-04-01 05:15:50 | 2020-04-04 07:01:45 | 2020-04-04 08:01:46 | 1:00:01 | 0:13:58 | 0:46:03 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/rbd_map_unmap.yaml} | 3 | |||
Failure Reason:
'/home/ubuntu/cephtest/archive/syslog/kern.log:2020-04-04T07:56:41.152863+00:00 smithi098 kernel: [ 620.828385] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low! ' in syslog |
||||||||||||||
pass | 4912216 | 2020-04-01 05:15:50 | 2020-04-04 07:01:45 | 2020-04-04 08:17:46 | 1:16:01 | 0:09:01 | 1:07:00 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/few.yaml tasks/rbd_workunit_trivial_sync.yaml} | 3 | |||
fail | 4912217 | 2020-04-01 05:15:51 | 2020-04-04 07:01:55 | 2020-04-04 08:53:56 | 1:52:01 | 0:51:24 | 1:00:37 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/rbd_simple_big.yaml} | 3 | |||
Failure Reason:
"2020-04-04 08:10:25.389742 mon.b mon.0 172.21.15.50:6789/0 119 : cluster [WRN] Health check failed: 32 slow ops, oldest one blocked for 33 sec, daemons [osd.2,osd.6] have slow ops. (SLOW_OPS)" in cluster log |
||||||||||||||
pass | 4912218 | 2020-04-01 05:15:52 | 2020-04-04 07:01:59 | 2020-04-04 07:53:59 | 0:52:00 | 0:28:14 | 0:23:46 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/many.yaml tasks/rbd_fio.yaml} | 3 | |||
fail | 4912219 | 2020-04-01 05:15:53 | 2020-04-04 07:03:03 | 2020-04-04 11:41:08 | 4:38:05 | 3:11:31 | 1:26:34 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/krbd_data_pool.yaml} | 3 | |||
Failure Reason:
Command failed (workunit test rbd/krbd_data_pool.sh) on smithi116 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36ac2fb654d2cf4adb279402ce44a37699a8d949 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/krbd_data_pool.sh' |
||||||||||||||
fail | 4912220 | 2020-04-01 05:15:54 | 2020-04-04 07:03:38 | 2020-04-04 10:17:41 | 3:14:03 | 2:58:50 | 0:15:13 | smithi | master | krbd/singleton/{conf.yaml msgr-failures/many.yaml tasks/rbd_xfstests.yaml} | 4 | |||
Failure Reason:
Command failed on smithi125 with status 2: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludeGomDAi -r -- '-g auto -g blockdev -x clone'" |
||||||||||||||
pass | 4912221 | 2020-04-01 05:15:54 | 2020-04-02 23:26:04 | 2020-04-03 00:22:04 | 0:56:00 | 0:43:39 | 0:12:21 | smithi | master | krbd/thrash/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml thrashers/backoff.yaml thrashosds-health.yaml workloads/rbd_workunit_suites_ffsb.yaml} | 3 | |||
pass | 4912222 | 2020-04-01 05:15:55 | 2020-04-04 07:04:20 | 2020-04-04 07:46:19 | 0:41:59 | 0:09:57 | 0:32:02 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/krbd_exclusive_option.yaml} | 3 | |||
pass | 4912223 | 2020-04-01 05:15:56 | 2020-04-04 07:05:55 | 2020-04-04 08:43:56 | 1:38:01 | 0:13:39 | 1:24:22 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/few.yaml tasks/rbd_workunit_kernel_untar_build.yaml} | 3 | |||
pass | 4912224 | 2020-04-01 05:15:57 | 2020-04-04 07:05:55 | 2020-04-04 08:23:55 | 1:18:00 | 0:14:10 | 1:03:50 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/krbd_fallocate.yaml} | 3 | |||
dead | 4912225 | 2020-04-01 05:15:58 | 2020-04-04 07:05:55 | 2020-04-04 19:08:22 | 12:02:27 | smithi | master | krbd/basic/{ceph/ceph.yaml clusters/fixed-1.yaml conf.yaml tasks/krbd_read_only.yaml} | 1 | |||||
pass | 4912226 | 2020-04-01 05:15:58 | 2020-04-04 07:05:55 | 2020-04-04 07:59:55 | 0:54:00 | 0:12:16 | 0:41:44 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/krbd_latest_osdmap_on_map.yaml} | 3 | |||
pass | 4912227 | 2020-04-01 05:15:59 | 2020-04-04 07:05:55 | 2020-04-04 09:03:56 | 1:58:01 | 0:38:15 | 1:19:46 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/many.yaml tasks/rbd_workunit_suites_dbench.yaml} | 3 | |||
fail | 4912228 | 2020-04-01 05:16:00 | 2020-04-04 07:07:46 | 2020-04-04 08:49:47 | 1:42:01 | 0:47:19 | 0:54:42 | smithi | master | krbd/thrash/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml thrashers/mon-thrasher.yaml thrashosds-health.yaml workloads/rbd_fio.yaml} | 3 | |||
Failure Reason:
"2020-04-04 08:07:34.043699 mon.b mon.0 172.21.15.16:6789/0 179 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log |
||||||||||||||
pass | 4912229 | 2020-04-01 05:16:01 | 2020-04-04 07:07:46 | 2020-04-04 09:21:48 | 2:14:02 | 0:28:12 | 1:45:50 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/krbd_udev_enumerate.yaml} | 3 | |||
fail | 4912230 | 2020-04-01 05:16:02 | 2020-04-04 07:07:46 | 2020-04-04 09:21:47 | 2:14:01 | 0:36:51 | 1:37:10 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/few.yaml tasks/rbd_workunit_suites_ffsb.yaml} | 3 | |||
Failure Reason:
"2020-04-04 08:51:12.068585 mon.b mon.0 172.21.15.89:6789/0 137 : cluster [WRN] Health check failed: 33 slow ops, oldest one blocked for 30 sec, daemons [osd.2,osd.3,osd.6] have slow ops. (SLOW_OPS)" in cluster log |
||||||||||||||
pass | 4912231 | 2020-04-01 05:16:02 | 2020-04-04 07:07:56 | 2020-04-04 07:29:55 | 0:21:59 | 0:08:33 | 0:13:26 | smithi | master | krbd/unmap/{ceph/ceph.yaml clusters/separate-client.yaml conf.yaml filestore-xfs.yaml kernels/single-major-on.yaml tasks/unmap.yaml} | 2 | |||
pass | 4912232 | 2020-04-01 05:16:03 | 2020-04-04 07:07:56 | 2020-04-04 07:51:56 | 0:44:00 | 0:16:45 | 0:27:15 | smithi | master | krbd/wac/wac/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml tasks/wac.yaml verify/no-resets.yaml} | 3 | |||
pass | 4912233 | 2020-04-01 05:16:04 | 2020-04-04 07:08:09 | 2020-04-04 07:38:09 | 0:30:00 | 0:14:36 | 0:15:24 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/krbd_udev_netlink_enobufs.yaml} | 3 | |||
fail | 4912234 | 2020-04-01 05:16:05 | 2020-04-04 07:09:41 | 2020-04-04 12:09:46 | 5:00:05 | 4:00:05 | 1:00:00 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/rbd_concurrent.yaml} | 3 | |||
Failure Reason:
Command failed (workunit test rbd/concurrent.sh) on smithi203 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36ac2fb654d2cf4adb279402ce44a37699a8d949 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/concurrent.sh' |
||||||||||||||
pass | 4912235 | 2020-04-01 05:16:06 | 2020-04-04 07:09:41 | 2020-04-04 07:37:41 | 0:28:00 | 0:09:46 | 0:18:14 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/many.yaml tasks/rbd_workunit_suites_fsstress.yaml} | 3 | |||
pass | 4912236 | 2020-04-01 05:16:06 | 2020-04-04 07:11:53 | 2020-04-04 07:33:53 | 0:22:00 | 0:08:57 | 0:13:03 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/rbd_huge_tickets.yaml} | 3 | |||
pass | 4912237 | 2020-04-01 05:16:07 | 2020-04-04 07:11:53 | 2020-04-04 08:19:54 | 1:08:01 | 0:48:40 | 0:19:21 | smithi | master | krbd/thrash/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/rbd_workunit_suites_ffsb.yaml} | 3 | |||
pass | 4912238 | 2020-04-01 05:16:08 | 2020-04-04 07:14:05 | 2020-04-04 07:46:04 | 0:31:59 | 0:09:10 | 0:22:49 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/few.yaml tasks/rbd_workunit_suites_fsstress_ext4.yaml} | 3 | |||
fail | 4912239 | 2020-04-01 05:16:09 | 2020-04-04 07:14:05 | 2020-04-04 10:46:08 | 3:32:03 | 3:11:57 | 0:20:06 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/rbd_image_read.yaml} | 3 | |||
Failure Reason:
Command failed (workunit test rbd/image_read.sh) on smithi006 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36ac2fb654d2cf4adb279402ce44a37699a8d949 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/image_read.sh' |
||||||||||||||
dead | 4912240 | 2020-04-01 05:16:10 | 2020-04-04 07:15:56 | 2020-04-04 19:18:19 | 12:02:23 | smithi | master | krbd/basic/{ceph/ceph.yaml clusters/fixed-1.yaml conf.yaml tasks/krbd_whole_object_zeroout.yaml} | 1 | |||||
pass | 4912241 | 2020-04-01 05:16:10 | 2020-04-04 07:16:27 | 2020-04-04 07:56:27 | 0:40:00 | 0:09:02 | 0:30:58 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/rbd_kernel.yaml} | 3 | |||
pass | 4912242 | 2020-04-01 05:16:11 | 2020-04-04 07:17:54 | 2020-04-04 08:19:54 | 1:02:00 | 0:16:20 | 0:45:40 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/many.yaml tasks/rbd_workunit_suites_fsx.yaml} | 3 | |||
fail | 4912243 | 2020-04-01 05:16:12 | 2020-04-04 07:17:54 | 2020-04-04 19:20:21 | 12:02:27 | 11:22:39 | 0:39:48 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/rbd_kfsx.yaml} | 3 | |||
Failure Reason:
psutil.NoSuchProcess process no longer exists (pid=5148) |
||||||||||||||
pass | 4912244 | 2020-04-01 05:16:13 | 2020-04-04 07:17:56 | 2020-04-04 08:39:57 | 1:22:01 | 0:43:36 | 0:38:25 | smithi | master | krbd/thrash/{ceph/ceph.yaml clusters/fixed-3.yaml conf.yaml thrashers/upmap.yaml thrashosds-health.yaml workloads/rbd_fio.yaml} | 3 | |||
fail | 4912245 | 2020-04-01 05:16:14 | 2020-04-04 07:19:54 | 2020-04-04 07:55:54 | 0:36:00 | 0:17:00 | 0:19:00 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/few.yaml tasks/rbd_workunit_suites_iozone.yaml} | 3 | |||
Failure Reason:
"2020-04-04 07:45:58.362762 mon.a mon.0 172.21.15.2:6789/0 134 : cluster [WRN] Health check failed: 54 slow ops, oldest one blocked for 34 sec, daemons [osd.0,osd.6,osd.7] have slow ops. (SLOW_OPS)" in cluster log |
||||||||||||||
pass | 4912246 | 2020-04-01 05:16:15 | 2020-04-04 07:19:54 | 2020-04-04 07:51:54 | 0:32:00 | 0:08:45 | 0:23:15 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/rbd_map_snapshot_io.yaml} | 3 | |||
pass | 4912247 | 2020-04-01 05:16:15 | 2020-04-04 07:19:54 | 2020-04-04 08:09:54 | 0:50:00 | 0:14:00 | 0:36:00 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/many.yaml tasks/rbd_map_unmap.yaml} | 3 | |||
pass | 4912248 | 2020-04-01 05:16:16 | 2020-04-04 07:20:09 | 2020-04-04 07:42:09 | 0:22:00 | 0:09:13 | 0:12:47 | smithi | master | krbd/rbd/{clusters/fixed-3.yaml conf.yaml msgr-failures/many.yaml tasks/rbd_workunit_trivial_sync.yaml} | 3 | |||
fail | 4912249 | 2020-04-01 05:16:17 | 2020-04-04 07:22:06 | 2020-04-04 11:22:09 | 4:00:03 | 3:10:15 | 0:49:48 | smithi | master | krbd/rbd-nomount/{clusters/fixed-3.yaml conf.yaml install/ceph.yaml msgr-failures/few.yaml tasks/rbd_simple_big.yaml} | 3 | |||
Failure Reason:
Command failed (workunit test rbd/simple_big.sh) on smithi132 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36ac2fb654d2cf4adb279402ce44a37699a8d949 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/simple_big.sh' |