Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version
fail 4369519 2019-10-08 23:55:54 2019-10-09 02:53:23 2019-10-09 03:29:22 0:35:59 0:12:56 0:23:03 smithi master
Failure Reason:

Command failed on smithi157 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok perf dump'

fail 4369520 2019-10-08 23:55:55 2019-10-09 02:53:30 2019-10-09 04:13:31 1:20:01 0:19:55 1:00:06 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi161.front.sepia.ceph.com

fail 4369521 2019-10-08 23:55:55 2019-10-09 02:54:07 2019-10-09 04:56:08 2:02:01 1:36:39 0:25:22 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369522 2019-10-08 23:55:56 2019-10-09 02:57:49 2019-10-09 03:43:48 0:45:59 0:23:13 0:22:46 smithi master
Failure Reason:

Command failed (workunit test kernel_untar_build.sh) on smithi042 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

dead 4369523 2019-10-08 23:55:57 2019-10-09 03:04:30 2019-10-09 15:06:57 12:02:27 smithi master
fail 4369524 2019-10-08 23:55:58 2019-10-09 03:06:48 2019-10-09 03:36:47 0:29:59 0:14:53 0:15:06 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi101.front.sepia.ceph.com

fail 4369525 2019-10-08 23:55:58 2019-10-09 03:08:05 2019-10-09 04:06:05 0:58:00 0:36:33 0:21:27 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369526 2019-10-08 23:55:59 2019-10-09 03:08:17 2019-10-09 04:58:17 1:50:00 1:31:27 0:18:33 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi096.front.sepia.ceph.com

fail 4369527 2019-10-08 23:56:00 2019-10-09 03:09:41 2019-10-09 05:17:43 2:08:02 0:20:11 1:47:51 smithi master
Failure Reason:

Command failed (workunit test suites/pjd.sh) on smithi091 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

fail 4369528 2019-10-08 23:56:01 2019-10-09 03:09:41 2019-10-09 05:59:43 2:50:02 2:27:15 0:22:47 smithi master centos
Failure Reason:

saw valgrind issues

fail 4369529 2019-10-08 23:56:01 2019-10-09 03:09:41 2019-10-09 04:27:42 1:18:01 0:53:31 0:24:30 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369530 2019-10-08 23:56:02 2019-10-09 03:11:32 2019-10-09 04:29:32 1:18:00 0:22:21 0:55:39 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi199.front.sepia.ceph.com

fail 4369531 2019-10-08 23:56:03 2019-10-09 03:16:28 2019-10-09 04:28:28 1:12:00 0:41:37 0:30:23 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369532 2019-10-08 23:56:04 2019-10-09 03:16:28 2019-10-09 05:04:28 1:48:00 0:13:37 1:34:23 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi181.front.sepia.ceph.com

fail 4369533 2019-10-08 23:56:04 2019-10-09 03:16:59 2019-10-09 05:43:01 2:26:02 0:12:47 2:13:15 smithi master
Failure Reason:

Command failed on smithi050 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump'

fail 4369534 2019-10-08 23:56:05 2019-10-09 03:17:04 2019-10-09 04:33:04 1:16:00 0:18:10 0:57:50 smithi master
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi070 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

fail 4369535 2019-10-08 23:56:06 2019-10-09 03:19:15 2019-10-09 10:39:23 7:20:08 6:31:05 0:49:03 smithi master
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi201 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh'

fail 4369536 2019-10-08 23:56:06 2019-10-09 03:19:20 2019-10-09 04:47:20 1:28:00 0:55:48 0:32:12 smithi master
Failure Reason:

Command failed (workunit test kernel_untar_build.sh) on smithi174 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

fail 4369537 2019-10-08 23:56:07 2019-10-09 03:20:34 2019-10-09 05:18:35 1:58:01 1:01:55 0:56:06 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi022.front.sepia.ceph.com

fail 4369538 2019-10-08 23:56:08 2019-10-09 03:22:53 2019-10-09 05:40:54 2:18:01 0:34:52 1:43:09 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369539 2019-10-08 23:56:09 2019-10-09 03:23:47 2019-10-09 06:53:50 3:30:03 3:15:55 0:14:08 smithi master
Failure Reason:

Command failed (workunit test suites/blogbench.sh) on smithi139 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/blogbench.sh'

fail 4369540 2019-10-08 23:56:09 2019-10-09 03:25:19 2019-10-09 05:01:19 1:36:00 1:10:46 0:25:14 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369541 2019-10-08 23:56:10 2019-10-09 03:28:51 2019-10-09 04:40:51 1:12:00 0:44:46 0:27:14 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi181.front.sepia.ceph.com

fail 4369542 2019-10-08 23:56:11 2019-10-09 03:29:24 2019-10-09 04:43:24 1:14:00 0:56:39 0:17:21 smithi master centos
Failure Reason:

saw valgrind issues

fail 4369543 2019-10-08 23:56:12 2019-10-09 03:30:25 2019-10-09 05:48:26 2:18:01 2:01:07 0:16:54 smithi master
Failure Reason:

"2019-10-09 03:55:12.541739 mon.b mon.0 172.21.15.44:6789/0 145 : cluster [WRN] Health check failed: 4253 slow requests are blocked > 32 sec. Implicated osds 3 (REQUEST_SLOW)" in cluster log

fail 4369544 2019-10-08 23:56:12 2019-10-09 03:32:36 2019-10-09 04:48:37 1:16:01 0:34:09 0:41:52 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369545 2019-10-08 23:56:13 2019-10-09 03:36:36 2019-10-09 06:10:38 2:34:02 1:16:31 1:17:31 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi129.front.sepia.ceph.com

fail 4369546 2019-10-08 23:56:14 2019-10-09 03:36:48 2019-10-09 04:52:49 1:16:01 0:27:42 0:48:19 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369547 2019-10-08 23:56:14 2019-10-09 03:37:01 2019-10-09 06:07:03 2:30:02 0:12:38 2:17:24 smithi master
Failure Reason:

Command failed on smithi016 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok perf dump'

fail 4369548 2019-10-08 23:56:15 2019-10-09 03:38:00 2019-10-09 04:25:59 0:47:59 0:18:17 0:29:42 smithi master
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi065 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

fail 4369549 2019-10-08 23:56:16 2019-10-09 03:39:16 2019-10-09 05:57:18 2:18:02 0:34:10 1:43:52 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369550 2019-10-08 23:56:17 2019-10-09 03:44:07 2019-10-09 05:54:07 2:10:00 0:49:24 1:20:36 smithi master
Failure Reason:

Command failed (workunit test kernel_untar_build.sh) on smithi159 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'

dead 4369551 2019-10-08 23:56:17 2019-10-09 03:44:55 2019-10-09 15:47:18 12:02:23 smithi master
fail 4369552 2019-10-08 23:56:18 2019-10-09 03:48:49 2019-10-09 05:18:49 1:30:00 0:33:26 0:56:34 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369553 2019-10-08 23:56:19 2019-10-09 03:55:50 2019-10-09 05:09:50 1:14:00 0:29:58 0:44:02 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi016.front.sepia.ceph.com

fail 4369554 2019-10-08 23:56:20 2019-10-09 03:56:46 2019-10-09 06:12:48 2:16:02 1:00:49 1:15:13 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369555 2019-10-08 23:56:20 2019-10-09 03:57:03 2019-10-09 04:43:02 0:45:59 0:12:15 0:33:44 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi161.front.sepia.ceph.com

fail 4369556 2019-10-08 23:56:21 2019-10-09 04:02:50 2019-10-09 05:18:50 1:16:00 0:55:27 0:20:33 smithi master centos
Failure Reason:

saw valgrind issues

fail 4369557 2019-10-08 23:56:22 2019-10-09 04:04:36 2019-10-09 05:10:36 1:06:00 0:31:05 0:34:55 smithi master
Failure Reason:

"2019-10-09 04:46:14.815727 mon.b mon.0 172.21.15.53:6789/0 143 : cluster [WRN] Health check failed: 1466 slow requests are blocked > 32 sec. Implicated osds 6 (REQUEST_SLOW)" in cluster log

fail 4369558 2019-10-08 23:56:23 2019-10-09 04:06:20 2019-10-09 05:10:20 1:04:00 0:36:26 0:27:34 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369560 2019-10-08 23:56:23 2019-10-09 04:12:26 2019-10-09 05:08:26 0:56:00 0:35:15 0:20:45 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi071.front.sepia.ceph.com

fail 4369562 2019-10-08 23:56:24 2019-10-09 04:20:33 2019-10-09 05:06:37 0:46:04 0:28:12 0:17:52 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369564 2019-10-08 23:56:25 2019-10-09 04:26:15 2019-10-09 04:40:14 0:13:59 smithi master
Failure Reason:

Stale jobs detected, aborting.

dead 4369566 2019-10-08 23:56:26 2019-10-09 04:28:30 2019-10-09 16:30:56 12:02:26 smithi master
fail 4369568 2019-10-08 23:56:26 2019-10-09 04:33:19 2019-10-09 11:29:30 6:56:11 6:27:50 0:28:21 smithi master
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi007 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

fail 4369570 2019-10-08 23:56:27 2019-10-09 04:40:34 2019-10-09 05:38:34 0:58:00 0:40:32 0:17:28 smithi master
Failure Reason:

Command failed (workunit test kernel_untar_build.sh) on smithi174 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh'

dead 4369572 2019-10-08 23:56:28 2019-10-09 04:43:20 2019-10-09 16:49:49 12:06:29 11:34:45 0:31:44 smithi master
Failure Reason:

Command failed (workunit test fs/misc/dirfrag.sh) on smithi160 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=608764ad142da76029fc524994146b973593bf1d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/misc/dirfrag.sh'

fail 4369574 2019-10-08 23:56:29 2019-10-09 04:45:18 2019-10-09 06:59:19 2:14:01 0:36:09 1:37:52 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369576 2019-10-08 23:56:29 2019-10-09 04:47:38 2019-10-09 06:01:38 1:14:00 0:25:14 0:48:46 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi038.front.sepia.ceph.com

fail 4369578 2019-10-08 23:56:30 2019-10-09 04:52:03 2019-10-09 06:38:03 1:46:00 1:31:48 0:14:12 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369580 2019-10-08 23:56:31 2019-10-09 04:52:50 2019-10-09 05:28:50 0:36:00 0:11:59 0:24:01 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi049.front.sepia.ceph.com

fail 4369582 2019-10-08 23:56:32 2019-10-09 04:55:35 2019-10-09 06:23:36 1:28:01 0:55:35 0:32:26 smithi master centos
Failure Reason:

saw valgrind issues

fail 4369584 2019-10-08 23:56:33 2019-10-09 04:58:33 2019-10-09 06:00:32 1:01:59 0:30:48 0:31:11 smithi master
Failure Reason:

"2019-10-09 05:36:03.284552 mon.b mon.0 172.21.15.100:6789/0 231 : cluster [WRN] Health check failed: 221 slow requests are blocked > 32 sec. Implicated osds 0,1,3,5 (REQUEST_SLOW)" in cluster log

fail 4369586 2019-10-08 23:56:33 2019-10-09 05:00:06 2019-10-09 05:56:06 0:56:00 0:33:27 0:22:33 smithi master
Failure Reason:

reached maximum tries (150) after waiting for 900 seconds

fail 4369588 2019-10-08 23:56:34 2019-10-09 05:37:41 2019-10-09 07:49:42 2:12:01 1:13:16 0:58:45 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi203.front.sepia.ceph.com

fail 4369590 2019-10-08 23:56:35 2019-10-09 05:38:31 2019-10-09 06:44:31 1:06:00 0:13:23 0:52:37 smithi master
Failure Reason:

Found coredumps on ubuntu@smithi046.front.sepia.ceph.com