Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
pass 4545356 2019-11-26 10:51:00 2019-11-26 11:45:01 2019-11-26 12:11:00 0:25:59 0:15:25 0:10:34 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync.yaml workloads/pool-create-delete.yaml} 2
fail 4545357 2019-11-26 10:51:01 2019-11-26 12:06:35 2019-11-26 12:58:35 0:52:00 0:42:52 0:09:08 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/force-sync-many.yaml workloads/rados_5925.yaml} 2
Failure Reason:

"2019-11-26T12:37:17.390170+0000 mon.a (mon.1) 100 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4545358 2019-11-26 10:51:02 2019-11-26 12:11:17 2019-11-26 12:47:17 0:36:00 0:25:20 0:10:40 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/many.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on mira118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4545359 2019-11-26 10:51:03 2019-11-26 12:24:36 2019-11-26 13:42:36 1:18:00 1:07:36 0:10:24 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/one.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

fail 4545360 2019-11-26 10:51:04 2019-11-26 12:43:01 2019-11-26 13:41:00 0:57:59 0:41:03 0:16:56 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync-many.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

"2019-11-26T13:15:46.136991+0000 mon.c (mon.2) 55 : cluster [WRN] Health check failed: 1 pool(s) have no replicas configured (POOL_NO_REDUNDANCY)" in cluster log

fail 4545361 2019-11-26 10:51:05 2019-11-26 12:47:19 2019-11-26 13:49:19 1:02:00 0:46:48 0:15:12 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

pass 4545362 2019-11-26 10:51:05 2019-11-26 12:58:43 2019-11-26 13:30:42 0:31:59 0:20:44 0:11:15 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/force-sync-many.yaml workloads/pool-create-delete.yaml} 2
fail 4545363 2019-11-26 10:51:06 2019-11-26 13:31:04 2019-11-26 14:16:52 0:45:48 0:28:18 0:17:30 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/many.yaml workloads/rados_5925.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

fail 4545364 2019-11-26 10:51:07 2019-11-26 13:41:19 2019-11-26 14:19:18 0:37:59 0:27:42 0:10:17 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/one.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on mira105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4545365 2019-11-26 10:51:09 2019-11-26 13:42:38 2019-11-26 14:32:38 0:50:00 0:40:51 0:09:09 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/sync-many.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

"2019-11-26T14:13:33.918548+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4545366 2019-11-26 10:51:09 2019-11-26 13:49:21 2019-11-26 14:43:21 0:54:00 0:37:58 0:16:02 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

"2019-11-26T14:19:03.883266+0000 mon.a (mon.0) 121 : cluster [WRN] Health check failed: 1 pool(s) have no replicas configured (POOL_NO_REDUNDANCY)" in cluster log

fail 4545367 2019-11-26 10:51:10 2019-11-26 14:12:37 2019-11-26 14:56:37 0:44:00 0:32:38 0:11:22 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/force-sync-many.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

"2019-11-26T14:36:59.394777+0000 mon.b (mon.3) 121 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

pass 4545368 2019-11-26 10:51:11 2019-11-26 14:16:59 2019-11-26 14:52:58 0:35:59 0:29:21 0:06:38 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/many.yaml workloads/pool-create-delete.yaml} 2
fail 4545369 2019-11-26 10:51:12 2019-11-26 14:19:20 2019-11-26 15:09:20 0:50:00 0:41:49 0:08:11 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/one.yaml workloads/rados_5925.yaml} 2
Failure Reason:

"2019-11-26T14:48:43.739976+0000 mon.f (mon.0) 199 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4545370 2019-11-26 10:51:13 2019-11-26 14:32:40 2019-11-26 15:06:40 0:34:00 0:23:02 0:10:58 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync-many.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on mira046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4545371 2019-11-26 10:51:14 2019-11-26 14:43:40 2019-11-26 16:05:40 1:22:00 1:14:57 0:07:03 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/sync.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

fail 4545372 2019-11-26 10:51:15 2019-11-26 14:53:01 2019-11-26 15:43:01 0:50:00 0:37:39 0:12:21 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/force-sync-many.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

"2019-11-26T15:19:01.636990+0000 mon.b (mon.0) 168 : cluster [WRN] Health check failed: 1 pool(s) have no replicas configured (POOL_NO_REDUNDANCY)" in cluster log

fail 4545373 2019-11-26 10:51:16 2019-11-26 14:56:56 2019-11-26 15:48:56 0:52:00 0:38:17 0:13:43 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/many.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

"2019-11-26T15:30:06.941475+0000 mon.f (mon.0) 126 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

pass 4545374 2019-11-26 10:51:17 2019-11-26 15:06:42 2019-11-26 15:52:42 0:46:00 0:24:14 0:21:46 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/one.yaml workloads/pool-create-delete.yaml} 2
pass 4545375 2019-11-26 10:51:18 2019-11-26 15:09:42 2019-11-26 15:29:40 0:19:58 0:10:21 0:09:37 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync-many.yaml workloads/rados_5925.yaml} 2
fail 4545376 2019-11-26 10:51:19 2019-11-26 15:29:44 2019-11-26 16:19:43 0:49:59 0:29:24 0:20:35 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on mira105 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4545377 2019-11-26 10:51:20 2019-11-26 15:43:20 2019-11-26 16:21:19 0:37:59 0:27:49 0:10:10 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/force-sync-many.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

"2019-11-26T16:07:02.861707+0000 mon.f (mon.0) 89 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4545378 2019-11-26 10:51:21 2019-11-26 15:49:14 2019-11-26 16:31:13 0:41:59 0:30:43 0:11:16 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/many.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

"2019-11-26T16:07:07.803572+0000 mon.b (mon.0) 222 : cluster [WRN] Health check failed: 1 pool(s) have no replicas configured (POOL_NO_REDUNDANCY)" in cluster log

fail 4545379 2019-11-26 10:51:22 2019-11-26 15:52:59 2019-11-26 17:04:59 1:12:00 0:48:51 0:23:09 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/one.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

pass 4545380 2019-11-26 10:51:23 2019-11-26 16:05:42 2019-11-26 16:31:41 0:25:59 0:16:07 0:09:52 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync-many.yaml workloads/pool-create-delete.yaml} 2
fail 4545381 2019-11-26 10:51:24 2019-11-26 16:20:01 2019-11-26 17:02:00 0:41:59 0:33:47 0:08:12 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/sync.yaml workloads/rados_5925.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

fail 4545382 2019-11-26 10:51:25 2019-11-26 16:21:39 2019-11-26 17:11:38 0:49:59 0:36:52 0:13:07 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/force-sync-many.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on mira072 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4545383 2019-11-26 10:51:26 2019-11-26 16:31:15 2019-11-26 18:03:16 1:32:01 1:08:33 0:23:28 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/many.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

fail 4545384 2019-11-26 10:51:27 2019-11-26 16:31:59 2019-11-26 17:11:58 0:39:59 0:24:31 0:15:28 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/one.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

Command failed (workunit test mon/crush_ops.sh) on mira059 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/crush_ops.sh'

fail 4545385 2019-11-26 10:51:28 2019-11-26 17:02:19 2019-11-26 17:52:19 0:50:00 0:36:01 0:13:59 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync-many.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

"2019-11-26T17:34:54.415745+0000 mon.f (mon.0) 86 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

pass 4545386 2019-11-26 10:51:29 2019-11-26 17:05:20 2019-11-26 17:31:20 0:26:00 0:15:56 0:10:04 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync.yaml workloads/pool-create-delete.yaml} 2
pass 4545387 2019-11-26 10:51:30 2019-11-26 17:12:00 2019-11-26 19:56:01 2:44:01 2:25:17 0:18:44 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/force-sync-many.yaml workloads/rados_5925.yaml} 2
fail 4545388 2019-11-26 10:51:31 2019-11-26 17:12:00 2019-11-26 17:57:59 0:45:59 0:38:21 0:07:38 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/many.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on mira072 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4545389 2019-11-26 10:51:32 2019-11-26 17:31:24 2019-11-26 18:33:24 1:02:00 0:37:57 0:24:03 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/one.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

"2019-11-26T18:11:01.528984+0000 mon.f (mon.1) 91 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4545390 2019-11-26 10:51:33 2019-11-26 17:52:21 2019-11-26 18:40:21 0:48:00 0:40:31 0:07:29 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/sync-many.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

"2019-11-26T18:18:37.255348+0000 mon.b (mon.0) 164 : cluster [WRN] Health check failed: 1 pool(s) have no replicas configured (POOL_NO_REDUNDANCY)" in cluster log

fail 4545391 2019-11-26 10:51:34 2019-11-26 17:58:17 2019-11-26 18:52:17 0:54:00 0:42:58 0:11:02 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

pass 4545392 2019-11-26 10:51:35 2019-11-26 18:03:19 2019-11-26 18:29:19 0:26:00 0:16:30 0:09:30 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/force-sync-many.yaml workloads/pool-create-delete.yaml} 2
fail 4545393 2019-11-26 10:51:36 2019-11-26 18:29:21 2019-11-26 21:31:24 3:02:03 2:43:24 0:18:39 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/many.yaml workloads/rados_5925.yaml} 2
Failure Reason:

"2019-11-26T21:14:09.243153+0000 mon.f (mon.0) 240 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4545394 2019-11-26 10:51:37 2019-11-26 18:33:46 2019-11-26 19:19:46 0:46:00 0:36:02 0:09:58 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/one.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on mira082 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4545395 2019-11-26 10:51:38 2019-11-26 18:40:23 2019-11-26 19:30:35 0:50:12 0:35:47 0:14:25 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync-many.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

"2019-11-26T19:11:30.863520+0000 mon.f (mon.0) 96 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4545396 2019-11-26 10:51:39 2019-11-26 18:52:36 2019-11-26 19:30:40 0:38:04 0:27:12 0:10:52 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

"2019-11-26T19:10:22.269012+0000 mon.b (mon.1) 97 : cluster [WRN] Health check failed: 1 pool(s) have no replicas configured (POOL_NO_REDUNDANCY)" in cluster log

fail 4545397 2019-11-26 10:51:40 2019-11-26 19:19:49 2019-11-26 20:11:49 0:52:00 0:37:57 0:14:03 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/force-sync-many.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

"2019-11-26T19:51:00.115167+0000 mon.a (mon.0) 93 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

pass 4545398 2019-11-26 10:51:40 2019-11-26 19:31:02 2019-11-26 20:07:01 0:35:59 0:21:59 0:14:00 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/many.yaml workloads/pool-create-delete.yaml} 2
fail 4545399 2019-11-26 10:51:41 2019-11-26 19:31:01 2019-11-26 20:11:01 0:40:00 0:26:47 0:13:13 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/one.yaml workloads/rados_5925.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

fail 4545400 2019-11-26 10:51:42 2019-11-26 19:56:22 2019-11-26 20:40:21 0:43:59 0:34:08 0:09:51 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/sync-many.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on mira083 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4545401 2019-11-26 10:51:43 2019-11-26 20:07:24 2019-11-26 21:23:25 1:16:01 1:08:25 0:07:36 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/sync.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

fail 4545402 2019-11-26 10:51:44 2019-11-26 20:11:23 2019-11-26 20:59:23 0:48:00 0:40:58 0:07:02 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/force-sync-many.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

"2019-11-26T20:39:15.473180+0000 mon.b (mon.0) 161 : cluster [WRN] Health check failed: 1 pool(s) have no replicas configured (POOL_NO_REDUNDANCY)" in cluster log

dead 4545403 2019-11-26 10:51:45 2019-11-26 20:11:55 2019-11-27 08:14:20 12:02:25 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/many.yaml workloads/snaps-few-objects.yaml} 2
pass 4545404 2019-11-26 10:51:46 2019-11-26 20:40:24 2019-11-26 21:08:27 0:28:03 0:17:42 0:10:21 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/one.yaml workloads/pool-create-delete.yaml} 2
fail 4545405 2019-11-26 10:51:47 2019-11-26 20:59:26 2019-11-26 21:55:47 0:56:21 0:43:07 0:13:14 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync-many.yaml workloads/rados_5925.yaml} 2
Failure Reason:

"2019-11-26T21:34:00.004123+0000 mon.f (mon.0) 106 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4545406 2019-11-26 10:51:48 2019-11-26 21:08:48 2019-11-26 21:45:02 0:36:14 0:24:24 0:11:50 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on mira083 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35b0b52153206fd6178afe584a2cd9c0ef2bcc05 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4545407 2019-11-26 10:51:49 2019-11-26 21:23:45 2019-11-26 22:47:45 1:24:00 1:09:44 0:14:16 mira master centos 7.6 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/force-sync-many.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

failed to reach quorum size 9 before timeout expired

fail 4545408 2019-11-26 10:51:50 2019-11-26 21:31:26 2019-11-27 00:33:28 3:02:02 2:43:49 0:18:13 mira master rhel 7.7 rados:monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/many.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

"2019-11-27T00:11:09.384129+0000 mon.c (mon.2) 67 : cluster [WRN] Health check failed: 1 pool(s) have no replicas configured (POOL_NO_REDUNDANCY)" in cluster log

fail 4545409 2019-11-26 10:51:51 2019-11-26 21:45:04 2019-11-26 22:25:04 0:40:00 0:28:06 0:11:54 mira master ubuntu 18.04 rados:monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-avl.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/one.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

"2019-11-26T22:09:20.382659+0000 mon.f (mon.1) 117 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log