Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version
fail 4431561 2019-10-21 15:33:08 2019-10-21 15:59:19 2019-10-21 16:47:19 0:48:00 0:11:59 0:36:01 smithi master
Failure Reason:

Command failed on smithi069 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok perf dump'

fail 4431563 2019-10-21 15:33:09 2019-10-21 16:03:00 2019-10-21 17:10:59 1:07:59 0:19:46 0:48:13 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi005.front.sepia.ceph.com

fail 4431565 2019-10-21 15:33:10 2019-10-21 16:03:00 2019-10-21 18:27:01 2:24:01 1:34:55 0:49:06 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431567 2019-10-21 15:33:11 2019-10-21 16:03:00 2019-10-21 17:27:00 1:24:00 0:27:40 0:56:20 smithi master
Failure Reason:

Command failed (workunit test kernel_untar_build.sh) on smithi093 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'

dead 4431569 2019-10-21 15:33:12 2019-10-21 16:03:01 2019-10-22 04:05:27 12:02:26 smithi master
fail 4431571 2019-10-21 15:33:14 2019-10-21 16:03:21 2019-10-21 16:37:20 0:33:59 0:15:15 0:18:44 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi092.front.sepia.ceph.com

fail 4431573 2019-10-21 15:33:15 2019-10-21 16:05:49 2019-10-21 17:15:49 1:10:00 0:37:23 0:32:37 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431575 2019-10-21 15:33:16 2019-10-21 16:08:19 2019-10-21 17:52:20 1:44:01 1:29:41 0:14:20 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi181.front.sepia.ceph.com

fail 4431577 2019-10-21 15:33:17 2019-10-21 16:08:40 2019-10-21 16:40:40 0:32:00 0:18:53 0:13:07 smithi master
Failure Reason:

Command failed (workunit test suites/pjd.sh) on smithi139 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

fail 4431579 2019-10-21 15:33:18 2019-10-21 16:08:54 2019-10-21 18:38:55 2:30:01 2:14:59 0:15:02 smithi master centos
Failure Reason:

saw valgrind issues

fail 4431581 2019-10-21 15:33:19 2019-10-21 16:11:09 2019-10-21 17:23:09 1:12:00 0:58:55 0:13:05 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431583 2019-10-21 15:33:20 2019-10-21 16:11:09 2019-10-21 16:47:08 0:35:59 0:22:10 0:13:49 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi019.front.sepia.ceph.com

fail 4431585 2019-10-21 15:33:21 2019-10-21 16:11:20 2019-10-21 17:41:20 1:30:00 1:13:48 0:16:12 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431587 2019-10-21 15:33:22 2019-10-21 16:13:10 2019-10-21 16:53:10 0:40:00 0:12:31 0:27:29 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi151.front.sepia.ceph.com

fail 4431589 2019-10-21 15:33:23 2019-10-21 16:13:10 2019-10-21 17:11:10 0:58:00 0:13:19 0:44:41 smithi master
Failure Reason:

Command failed on smithi071 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump'

fail 4431591 2019-10-21 15:33:24 2019-10-21 16:13:14 2019-10-21 16:47:13 0:33:59 0:17:43 0:16:16 smithi master
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi057 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

dead 4431593 2019-10-21 15:33:25 2019-10-21 16:13:34 2019-10-22 04:16:01 12:02:27 smithi master
fail 4431595 2019-10-21 15:33:26 2019-10-21 16:15:27 2019-10-21 17:27:27 1:12:00 0:53:01 0:18:59 smithi master
Failure Reason:

Command failed (workunit test kernel_untar_build.sh) on smithi105 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'

fail 4431597 2019-10-21 15:33:27 2019-10-21 16:15:40 2019-10-21 20:03:43 3:48:03 3:34:31 0:13:32 smithi master
Failure Reason:

Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi029 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/misc/multiple_rsync.sh'

fail 4431599 2019-10-21 15:33:28 2019-10-21 16:17:15 2019-10-21 17:15:14 0:57:59 0:43:43 0:14:16 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431601 2019-10-21 15:33:29 2019-10-21 16:17:15 2019-10-21 17:03:14 0:45:59 0:22:27 0:23:32 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi001.front.sepia.ceph.com

fail 4431603 2019-10-21 15:33:30 2019-10-21 16:17:15 2019-10-21 17:47:15 1:30:00 1:16:15 0:13:45 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431605 2019-10-21 15:33:31 2019-10-21 16:17:38 2019-10-21 16:47:37 0:29:59 0:12:08 0:17:51 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi067.front.sepia.ceph.com

fail 4431607 2019-10-21 15:33:32 2019-10-21 16:18:34 2019-10-21 18:28:35 2:10:01 0:55:43 1:14:18 smithi master centos
Failure Reason:

saw valgrind issues

fail 4431609 2019-10-21 15:33:34 2019-10-21 16:18:34 2019-10-21 18:40:35 2:22:01 2:06:09 0:15:52 smithi master
Failure Reason:

"2019-10-21 16:41:43.731723 mon.b mon.0 172.21.15.47:6789/0 144 : cluster [WRN] Health check failed: 3598 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)" in cluster log

fail 4431611 2019-10-21 15:33:35 2019-10-21 16:19:01 2019-10-21 23:19:08 7:00:07 6:31:45 0:28:22 smithi master
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi133 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

fail 4431613 2019-10-21 15:33:36 2019-10-21 16:22:50 2019-10-21 17:58:51 1:36:01 1:22:03 0:13:58 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi059.front.sepia.ceph.com

fail 4431615 2019-10-21 15:33:37 2019-10-21 16:23:15 2019-10-21 17:21:15 0:58:00 0:28:00 0:30:00 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431617 2019-10-21 15:33:38 2019-10-21 16:23:22 2019-10-21 17:01:21 0:37:59 0:12:43 0:25:16 smithi master
Failure Reason:

Command failed on smithi002 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump'

fail 4431619 2019-10-21 15:33:39 2019-10-21 16:25:13 2019-10-21 17:03:12 0:37:59 0:18:04 0:19:55 smithi master
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi205 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

dead 4431621 2019-10-21 15:33:40 2019-10-21 16:25:27 2019-10-22 04:27:53 12:02:26 smithi master
fail 4431623 2019-10-21 15:33:41 2019-10-21 16:27:15 2019-10-21 18:17:16 1:50:01 0:49:18 1:00:43 smithi master
Failure Reason:

Command failed (workunit test kernel_untar_build.sh) on smithi080 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

fail 4431625 2019-10-21 15:33:42 2019-10-21 16:27:15 2019-10-21 20:57:19 4:30:04 3:40:45 0:49:19 smithi master
Failure Reason:

Command failed (workunit test fs/misc/multiple_rsync.sh) on smithi180 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/multiple_rsync.sh'

fail 4431627 2019-10-21 15:33:43 2019-10-21 16:29:34 2019-10-21 17:29:34 1:00:00 0:34:31 0:25:29 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431629 2019-10-21 15:33:45 2019-10-21 16:29:34 2019-10-21 17:21:34 0:52:00 0:36:05 0:15:55 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi164.front.sepia.ceph.com

fail 4431631 2019-10-21 15:33:46 2019-10-21 16:33:12 2019-10-21 18:37:13 2:04:01 1:01:00 1:03:01 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431633 2019-10-21 15:33:47 2019-10-21 16:35:25 2019-10-21 17:11:25 0:36:00 0:11:50 0:24:10 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi067.front.sepia.ceph.com

fail 4431635 2019-10-21 15:33:48 2019-10-21 16:35:25 2019-10-21 17:55:26 1:20:01 0:55:18 0:24:43 smithi master centos
Failure Reason:

saw valgrind issues

fail 4431637 2019-10-21 15:33:49 2019-10-21 16:37:13 2019-10-21 17:51:13 1:14:00 0:34:44 0:39:16 smithi master
Failure Reason:

"2019-10-21 17:24:33.258726 mon.a mon.0 172.21.15.33:6789/0 248 : cluster [WRN] Health check failed: 23777 slow requests are blocked > 32 sec. Implicated osds 0,1,3,4,5 (REQUEST_SLOW)" in cluster log

fail 4431639 2019-10-21 15:33:50 2019-10-21 16:37:13 2019-10-21 17:31:13 0:54:00 0:36:11 0:17:49 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431641 2019-10-21 15:33:51 2019-10-21 16:37:21 2019-10-21 17:47:21 1:10:00 0:35:09 0:34:51 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi151.front.sepia.ceph.com

fail 4431643 2019-10-21 15:33:52 2019-10-21 16:41:00 2019-10-21 17:22:59 0:41:59 0:27:52 0:14:07 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431645 2019-10-21 15:33:53 2019-10-21 16:41:00 2019-10-21 17:06:59 0:25:59 0:12:52 0:13:07 smithi master
Failure Reason:

Command failed on smithi200 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump'

fail 4431647 2019-10-21 15:33:54 2019-10-21 16:43:17 2019-10-21 17:31:17 0:48:00 0:18:15 0:29:45 smithi master
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi158 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

fail 4431649 2019-10-21 15:33:55 2019-10-21 16:43:17 2019-10-21 17:49:17 1:06:00 0:33:03 0:32:57 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431650 2019-10-21 15:33:57 2019-10-21 16:44:32 2019-10-21 18:14:33 1:30:01 0:40:55 0:49:06 smithi master
Failure Reason:

Command failed (workunit test kernel_untar_build.sh) on smithi140 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3b61c26d51b53dfe6eb53faa2db820aba29b6a78 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

fail 4431651 2019-10-21 15:33:58 2019-10-21 16:44:55 2019-10-21 18:22:56 1:38:01 1:06:31 0:31:30 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi088.front.sepia.ceph.com

fail 4431652 2019-10-21 15:33:59 2019-10-21 16:47:25 2019-10-21 17:37:25 0:50:00 0:36:52 0:13:08 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431653 2019-10-21 15:34:00 2019-10-21 16:47:25 2019-10-21 18:21:26 1:34:01 0:26:33 1:07:28 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi097.front.sepia.ceph.com

fail 4431654 2019-10-21 15:34:01 2019-10-21 16:47:25 2019-10-21 18:43:26 1:56:01 1:27:47 0:28:14 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431655 2019-10-21 15:34:02 2019-10-21 16:47:39 2019-10-21 18:05:44 1:18:05 0:12:40 1:05:25 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi018.front.sepia.ceph.com

fail 4431656 2019-10-21 15:34:03 2019-10-21 16:49:31 2019-10-21 17:55:31 1:06:00 0:50:46 0:15:14 smithi master centos
Failure Reason:

saw valgrind issues

fail 4431657 2019-10-21 15:34:04 2019-10-21 17:01:45 2019-10-21 17:43:45 0:42:00 0:29:04 0:12:56 smithi master
Failure Reason:

"2019-10-21 17:22:14.355858 mon.b mon.0 172.21.15.1:6789/0 140 : cluster [WRN] Health check failed: 162 slow requests are blocked > 32 sec. Implicated osds 0,1,4,5 (REQUEST_SLOW)" in cluster log

fail 4431658 2019-10-21 15:34:05 2019-10-21 17:03:29 2019-10-21 18:03:29 1:00:00 0:34:25 0:25:35 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4431659 2019-10-21 15:34:06 2019-10-21 17:03:30 2019-10-21 18:51:31 1:48:01 1:15:49 0:32:12 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi179.front.sepia.ceph.com

fail 4431660 2019-10-21 15:34:07 2019-10-21 17:05:15 2019-10-21 18:01:14 0:55:59 0:29:01 0:26:58 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds