Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 4404860 2019-10-12 00:26:38 2019-10-12 00:26:41 2019-10-12 03:36:43 3:10:02 3:01:58 0:08:04 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/cache-agent-big.yaml} 2
Failure Reason:

"2019-10-12T00:52:36.899497+0000 mon.a (mon.1) 71 : cluster [WRN] Health check update: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

dead 4404861 2019-10-12 00:26:39 2019-10-12 00:26:41 2019-10-12 12:29:02 12:02:21 mira master ubuntu 18.04 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
dead 4404862 2019-10-12 00:26:40 2019-10-12 00:26:41 2019-10-12 00:48:40 0:21:59 mira master rhel 7.7 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/rados_workunit_loadgen_big.yaml}
Failure Reason:

reached maximum tries (100) after waiting for 600 seconds

fail 4404863 2019-10-12 00:26:41 2019-10-12 00:26:42 2019-10-12 00:52:41 0:25:59 0:15:20 0:10:39 mira master ubuntu 18.04 rados/perf/{ceph.yaml objectstore/bluestore-basic-min-osd-mem-target.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{ubuntu_latest.yaml} workloads/cosbench_64K_read_write.yaml} 1
Failure Reason:

Command failed on mira035 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'

fail 4404864 2019-10-12 00:26:42 2019-10-12 00:26:43 2019-10-12 01:04:42 0:37:59 0:27:45 0:10:14 mira master rhel 7.7 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
Failure Reason:

"2019-10-12T00:56:20.574468+0000 mon.b (mon.1) 16 : cluster [WRN] Health check update: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

pass 4404865 2019-10-12 00:26:42 2019-10-12 00:26:44 2019-10-12 03:06:45 2:40:01 2:20:45 0:19:16 mira master rhel 7.7 rados/singleton-nomsgr/{all/version-number-sanity.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404866 2019-10-12 00:26:43 2019-10-12 00:26:45 2019-10-12 00:50:44 0:23:59 0:12:28 0:11:31 mira master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml} 2
Failure Reason:

"2019-10-12T00:42:33.770047+0000 mon.b (mon.0) 19 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404867 2019-10-12 00:26:44 2019-10-12 00:26:46 2019-10-12 01:02:45 0:35:59 0:10:31 0:25:28 mira master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/nautilus.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/few.yaml rados.yaml thrashers/none.yaml thrashosds-health.yaml workloads/test_rbd_api.yaml} 4
Failure Reason:

Command failed on mira065 with status 1: 'sudo yum -y install ceph-daemon'

fail 4404868 2019-10-12 00:26:45 2019-10-12 00:26:46 2019-10-12 00:52:46 0:26:00 0:15:43 0:10:17 mira master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_workunit_loadgen_mix.yaml} 2
Failure Reason:

"2019-10-12T00:43:56.749799+0000 mon.a (mon.1) 184 : cluster [WRN] Health check update: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 4404869 2019-10-12 00:26:46 2019-10-12 00:26:47 2019-10-12 03:04:49 2:38:02 2:19:43 0:18:19 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml} 2
Failure Reason:

"2019-10-12T00:57:31.271485+0000 mon.b (mon.1) 43 : cluster [WRN] Health check update: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404870 2019-10-12 00:26:47 2019-10-12 00:48:42 2019-10-12 02:02:42 1:14:00 1:03:16 0:10:44 mira master ubuntu 18.04 rados/singleton/{all/watch-notify-same-primary.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

"2019-10-12T01:04:57.951627+0000 mon.b (mon.1) 44 : cluster [WRN] Health check update: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 4404871 2019-10-12 00:26:48 2019-10-12 00:50:46 2019-10-12 01:28:45 0:37:59 0:31:57 0:06:02 mira master rhel 7.7 rados/perf/{ceph.yaml objectstore/bluestore-bitmap.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/cosbench_64K_write.yaml} 1
fail 4404872 2019-10-12 00:26:49 2019-10-12 00:52:43 2019-10-12 02:12:43 1:20:00 1:09:01 0:10:59 mira master ubuntu 18.04 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/fastclose.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=3-m=1.yaml} 2
Failure Reason:

"2019-10-12T01:09:16.238942+0000 mon.b (mon.1) 34 : cluster [WRN] Health check update: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 4404873 2019-10-12 00:26:49 2019-10-12 00:52:47 2019-10-12 01:36:47 0:44:00 0:35:23 0:08:37 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-snaps.yaml} 2
fail 4404874 2019-10-12 00:26:50 2019-10-12 01:02:54 2019-10-12 04:58:57 3:56:03 3:34:55 0:21:08 mira master centos 7.6 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

Command failed (workunit test cephtool/test.sh) on mira032 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0dca79308d83afd02b99aa1aa8839d156215298c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'

fail 4404875 2019-10-12 00:26:51 2019-10-12 01:04:44 2019-10-12 01:38:44 0:34:00 0:26:53 0:07:07 mira master rhel 7.7 rados/upgrade/mimic-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/mimic.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-nautilus.yaml 8-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml supported-random-distro$/{rhel_7.yaml} thrashosds-health.yaml} 4
Failure Reason:

Command failed on mira117 with status 124: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_health_to_clog=false"

fail 4404876 2019-10-12 00:26:52 2019-10-12 01:28:47 2019-10-12 03:02:47 1:34:00 1:26:26 0:07:34 mira master rhel 7.7 rados/multimon/{clusters/3.yaml msgr-failures/few.yaml msgr/async-v2only.yaml no_pools.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/mon_clock_no_skews.yaml} 2
Failure Reason:

"2019-10-12T01:52:00.727973+0000 mon.b (mon.1) 27 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404877 2019-10-12 00:26:53 2019-10-12 01:36:57 2019-10-12 02:08:56 0:31:59 0:21:27 0:10:32 mira master ubuntu 18.04 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-comp.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/dashboard.yaml} 2
Failure Reason:

Test failure: test_perf_counters_mds_get (tasks.mgr.dashboard.test_perf_counters.PerfCountersControllerTest)

fail 4404878 2019-10-12 00:26:54 2019-10-12 01:38:59 2019-10-12 05:25:01 3:46:02 3:35:46 0:10:16 mira master rhel 7.7 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml} 3
Failure Reason:

"2019-10-12T02:06:33.245578+0000 mon.c (mon.0) 23 : cluster [WRN] Health check failed: 1/3 mons down, quorum c,b (MON_DOWN)" in cluster log

fail 4404879 2019-10-12 00:26:55 2019-10-12 02:02:57 2019-10-12 02:42:56 0:39:59 0:29:38 0:10:21 mira master rhel 7.7 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/fastclose.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/minsize_recovery.yaml thrashosds-health.yaml workloads/ec-small-objects-overwrites.yaml} 2
Failure Reason:

"2019-10-12T02:29:27.068305+0000 mon.b (mon.1) 58 : cluster [WRN] Health check update: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404880 2019-10-12 00:26:55 2019-10-12 02:09:11 2019-10-12 05:37:13 3:28:02 3:09:27 0:18:35 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/cache.yaml} 2
Failure Reason:

Command failed on mira064 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4404881 2019-10-12 00:26:56 2019-10-12 02:12:45 2019-10-12 05:18:47 3:06:02 2:51:38 0:14:24 mira master centos 7.6 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
Failure Reason:

"2019-10-12T02:44:27.862228+0000 mon.a (mon.0) 208 : cluster [WRN] Health check update: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 4404882 2019-10-12 00:26:57 2019-10-12 02:42:58 2019-10-12 03:10:57 0:27:59 0:20:19 0:07:40 mira master rhel 7.7 rados/singleton/{all/deduptool.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4404883 2019-10-12 00:26:58 2019-10-12 03:03:02 2019-10-12 03:31:01 0:27:59 0:21:20 0:06:39 mira master rhel 7.7 rados/perf/{ceph.yaml objectstore/bluestore-comp.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/fio_4K_rand_read.yaml} 1
fail 4404884 2019-10-12 00:26:59 2019-10-12 03:05:04 2019-10-12 06:03:06 2:58:02 2:39:32 0:18:30 mira master rhel 7.7 rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/sync.yaml workloads/rados_5925.yaml} 2
Failure Reason:

Command failed on mira027 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

dead 4404885 2019-10-12 00:27:00 2019-10-12 03:06:51 2019-10-12 15:09:14 12:02:23 11:52:38 0:09:45 mira master rhel 7.7 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/rados_workunit_loadgen_mostlyread.yaml} 2
Failure Reason:

SSH connection to mira046 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

pass 4404886 2019-10-12 00:27:00 2019-10-12 03:11:12 2019-10-12 03:49:11 0:37:59 0:20:31 0:17:28 mira master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/hammer.yaml backoff/normal.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/careful.yaml thrashosds-health.yaml workloads/test_rbd_api.yaml} 4
fail 4404887 2019-10-12 00:27:01 2019-10-12 03:31:16 2019-10-12 05:27:16 1:56:00 1:46:12 0:09:48 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-partial-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/dedup_tier.yaml} 2
Failure Reason:

"2019-10-12T03:57:48.279015+0000 mon.a (mon.1) 28 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 4404888 2019-10-12 00:27:02 2019-10-12 03:36:56 2019-10-12 04:02:56 0:26:00 0:20:07 0:05:53 mira master rhel 7.7 rados/singleton-nomsgr/{all/cache-fs-trunc.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4404889 2019-10-12 00:27:03 2019-10-12 03:49:25 2019-10-12 04:15:24 0:25:59 0:20:20 0:05:39 mira master rhel 7.7 rados/singleton/{all/divergent_priors.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4404890 2019-10-12 00:27:04 2019-10-12 04:03:10 2019-10-12 05:03:10 1:00:00 0:50:17 0:09:43 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/pool-snaps-few-objects.yaml} 2
fail 4404891 2019-10-12 00:27:05 2019-10-12 04:15:26 2019-10-12 05:19:26 1:04:00 0:44:20 0:19:40 mira master centos 7.6 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
Failure Reason:

"2019-10-12T04:48:40.148941+0000 mon.a (mon.0) 43 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4404892 2019-10-12 00:27:05 2019-10-12 04:59:12 2019-10-12 08:11:14 3:12:02 2:52:42 0:19:20 mira master rhel 7.7 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-radosbench.yaml} 2
Failure Reason:

Command failed on mira063 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it'

fail 4404893 2019-10-12 00:27:06 2019-10-12 05:03:24 2019-10-12 06:11:24 1:08:00 0:56:59 0:11:01 mira master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed on mira118 with status 1: 'sudo ceph --cluster ceph osd crush tunables jewel'

fail 4404894 2019-10-12 00:27:07 2019-10-12 05:18:54 2019-10-12 07:22:55 2:04:01 1:50:13 0:13:48 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/radosbench.yaml} 2
Failure Reason:

"2019-10-12T05:43:27.685832+0000 mon.a (mon.0) 23 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

pass 4404895 2019-10-12 00:27:08 2019-10-12 05:19:27 2019-10-12 05:49:27 0:30:00 0:20:49 0:09:11 mira master rhel 7.7 rados/perf/{ceph.yaml objectstore/bluestore-stupid.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/fio_4M_rand_read.yaml} 1
fail 4404896 2019-10-12 00:27:09 2019-10-12 05:25:16 2019-10-12 07:31:17 2:06:01 1:57:19 0:08:42 mira master rhel 7.7 rados/singleton/{all/ec-lost-unfound.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

"2019-10-12T05:50:03.372897+0000 mon.a (mon.0) 23 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

dead 4404897 2019-10-12 00:27:10 2019-10-12 05:27:18 2019-10-12 17:29:40 12:02:22 mira master rhel 7.7 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=4-m=2.yaml} 3
fail 4404898 2019-10-12 00:27:11 2019-10-12 05:37:15 2019-10-12 08:15:16 2:38:01 2:30:11 0:07:50 mira master rhel 7.7 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/few.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-snaps-few-objects-overwrites.yaml} 2
Failure Reason:

"2019-10-12T06:07:09.809342+0000 mon.a (mon.1) 13 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 4404899 2019-10-12 00:27:12 2019-10-12 05:49:41 2019-10-12 07:07:42 1:18:01 1:08:25 0:09:36 mira master rhel 7.7 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
Failure Reason:

Command failed on mira088 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4404900 2019-10-12 00:27:12 2019-10-12 06:03:20 2019-10-12 07:21:20 1:18:00 1:02:17 0:15:43 mira master centos 7.6 rados/multimon/{clusters/6.yaml msgr-failures/many.yaml msgr/async.yaml no_pools.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/mon_clock_with_skews.yaml} 2
Failure Reason:

Command failed on mira027 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

dead 4404901 2019-10-12 00:27:13 2019-10-12 06:11:38 2019-10-12 18:14:04 12:02:26 11:50:44 0:11:42 mira master rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/none.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml tasks/rados_api_tests.yaml validater/lockdep.yaml} 2
Failure Reason:

SSH connection to mira117 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

fail 4404902 2019-10-12 00:27:14 2019-10-12 07:07:57 2019-10-12 08:13:57 1:06:00 0:56:30 0:09:30 mira master ubuntu 18.04 rados/monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/force-sync-many.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed on mira088 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

pass 4404903 2019-10-12 00:27:15 2019-10-12 07:21:34 2019-10-12 07:45:33 0:23:59 0:18:15 0:05:44 mira master rhel 7.7 rados/singleton-nomsgr/{all/ceph-post-file.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404904 2019-10-12 00:27:16 2019-10-12 07:23:09 2019-10-12 07:47:09 0:24:00 0:13:33 0:10:27 mira master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-async-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/redirect_promote_tests.yaml} 2
Failure Reason:

Command failed on mira069 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode upmap'"

dead 4404905 2019-10-12 00:27:17 2019-10-12 07:31:27 2019-10-12 19:34:03 12:02:36 mira master rhel 7.7 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/repair_test.yaml} 2
pass 4404906 2019-10-12 00:27:17 2019-10-12 07:45:48 2019-10-12 08:13:47 0:27:59 0:21:44 0:06:15 mira master rhel 7.7 rados/perf/{ceph.yaml objectstore/bluestore-basic-min-osd-mem-target.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/fio_4M_rand_rw.yaml} 1
fail 4404907 2019-10-12 00:27:18 2019-10-12 07:47:23 2019-10-12 09:37:24 1:50:01 1:40:49 0:09:12 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/redirect_set_object.yaml} 2
Failure Reason:

"2019-10-12T08:15:02.337513+0000 mon.a (mon.1) 37 : cluster [WRN] Health check update: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 4404908 2019-10-12 00:27:19 2019-10-12 08:11:30 2019-10-12 09:37:30 1:26:00 1:00:53 0:25:07 mira master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/jewel.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/few.yaml rados.yaml thrashers/mapgap.yaml thrashosds-health.yaml workloads/radosbench.yaml} 4
dead 4404909 2019-10-12 00:27:20 2019-10-12 08:13:49 2019-10-12 20:16:16 12:02:27 mira master ubuntu 18.04 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/osd-delay.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/fastread.yaml thrashosds-health.yaml workloads/ec-small-objects-fast-read.yaml} 2
fail 4404910 2019-10-12 00:27:21 2019-10-12 08:14:13 2019-10-12 09:28:13 1:14:00 1:08:05 0:05:55 mira master rhel 7.7 rados/singleton/{all/lost-unfound.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

Command failed on mira057 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4404911 2019-10-12 00:27:22 2019-10-12 08:15:31 2019-10-12 10:55:33 2:40:02 2:21:10 0:18:52 mira master rhel 7.7 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/filestore-xfs.yaml supported-random-distro$/{rhel_7.yaml} tasks/module_selftest.yaml} 2
Failure Reason:

Test failure: test_devicehealth (tasks.mgr.test_module_selftest.TestModuleSelftest)

fail 4404912 2019-10-12 00:27:22 2019-10-12 09:28:19 2019-10-12 10:42:19 1:14:00 1:07:44 0:06:16 mira master rhel 7.7 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

Command failed on mira057 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4404913 2019-10-12 00:27:23 2019-10-12 09:37:38 2019-10-12 13:11:41 3:34:03 3:15:05 0:18:58 mira master rhel 7.7 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
Failure Reason:

"2019-10-12T12:16:45.468912+0000 mon.b (mon.0) 13 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

fail 4404914 2019-10-12 00:27:24 2019-10-12 09:37:38 2019-10-12 11:05:39 1:28:01 1:17:52 0:10:09 mira master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rgw_snaps.yaml} 2
Failure Reason:

"2019-10-12T09:54:16.129307+0000 mon.a (mon.0) 73 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4404915 2019-10-12 00:27:25 2019-10-12 10:42:35 2019-10-12 11:22:37 0:40:02 0:31:03 0:08:59 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-async-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/set-chunks-read.yaml} 2
Failure Reason:

"2019-10-12T11:08:22.038328+0000 mon.b (mon.1) 33 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404916 2019-10-12 00:27:26 2019-10-12 10:55:48 2019-10-12 12:13:48 1:18:00 1:05:17 0:12:43 mira master ubuntu 18.04 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml} 3
Failure Reason:

"2019-10-12T11:13:07.022577+0000 mon.a (mon.0) 15 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

dead 4404917 2019-10-12 00:27:26 2019-10-12 11:05:41 2019-10-12 23:08:06 12:02:25 mira master ubuntu 18.04 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/osd-delay.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-pool-snaps-few-objects-overwrites.yaml} 2
fail 4404918 2019-10-12 00:27:27 2019-10-12 11:22:52 2019-10-12 12:34:53 1:12:01 1:00:50 0:11:11 mira master ubuntu 18.04 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
Failure Reason:

"2019-10-12T11:38:45.166358+0000 mon.b (mon.1) 29 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404919 2019-10-12 00:27:28 2019-10-12 12:14:03 2019-10-12 15:04:05 2:50:02 2:30:44 0:19:18 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-async-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/small-objects.yaml} 2
Failure Reason:

"2019-10-12T14:49:21.278337+0000 mon.a (mon.1) 29 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 4404920 2019-10-12 00:27:29 2019-10-12 12:29:04 2019-10-12 12:59:03 0:29:59 0:23:36 0:06:23 mira master rhel 7.7 rados/perf/{ceph.yaml objectstore/bluestore-comp.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/radosbench_4K_rand_read.yaml} 1
pass 4404921 2019-10-12 00:27:30 2019-10-12 12:34:54 2019-10-12 13:24:54 0:50:00 0:24:39 0:25:21 mira master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/luminous-v1only.yaml backoff/normal.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/rbd_cls.yaml} 4
pass 4404922 2019-10-12 00:27:31 2019-10-12 12:59:06 2019-10-12 13:29:05 0:29:59 0:24:22 0:05:37 mira master rhel 7.7 rados/singleton/{all/max-pg-per-osd.from-replica.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404923 2019-10-12 00:27:31 2019-10-12 13:11:43 2019-10-12 15:47:44 2:36:01 2:21:15 0:14:46 mira master centos rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/default/{default.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml tasks/rados_cls_all.yaml validater/valgrind.yaml} 2
Failure Reason:

Command failed on mira069 with status 6: "sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog"

fail 4404924 2019-10-12 00:27:32 2019-10-12 13:24:56 2019-10-12 15:12:56 1:48:00 1:33:48 0:14:12 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/write_fadvise_dontneed.yaml} 2
Failure Reason:

"2019-10-12T13:51:22.590351+0000 mon.a (mon.0) 20 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4404925 2019-10-12 00:27:33 2019-10-12 13:29:07 2019-10-12 14:37:07 1:08:00 0:56:53 0:11:07 mira master ubuntu 18.04 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/fastclose.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/minsize_recovery.yaml thrashosds-health.yaml workloads/ec-small-objects-many-deletes.yaml} 2
Failure Reason:

Command failed on mira066 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

pass 4404926 2019-10-12 00:27:34 2019-10-12 14:37:09 2019-10-12 15:07:09 0:30:00 0:23:25 0:06:35 mira master rhel 7.7 rados/singleton/{all/mon-config-key-caps.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404927 2019-10-12 00:27:35 2019-10-12 15:04:08 2019-10-12 17:22:09 2:18:01 2:06:33 0:11:28 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/admin_socket_objecter_requests.yaml} 2
Failure Reason:

"2019-10-12T15:28:51.816112+0000 mon.b (mon.1) 13 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 4404928 2019-10-12 00:27:35 2019-10-12 15:07:25 2019-10-12 16:17:25 1:10:00 0:39:22 0:30:38 mira master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/luminous.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/none.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} 4
fail 4404929 2019-10-12 00:27:36 2019-10-12 15:09:29 2019-10-12 16:23:30 1:14:01 1:02:43 0:11:18 mira master centos 7.6 rados/singleton/{all/mon-config-keys.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

Command failed on mira057 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4404930 2019-10-12 00:27:37 2019-10-12 15:13:12 2019-10-12 16:39:12 1:26:00 1:15:46 0:10:14 mira master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_api_tests.yaml} 2
Failure Reason:

"2019-10-12T15:28:48.325106+0000 mon.b (mon.1) 10 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404931 2019-10-12 00:27:38 2019-10-12 15:47:47 2019-10-12 18:31:49 2:44:02 2:32:58 0:11:04 mira master ubuntu 18.04 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
Failure Reason:

"2019-10-12T16:03:32.828851+0000 mon.a (mon.1) 34 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

dead 4404932 2019-10-12 00:27:39 2019-10-12 16:17:40 2019-10-13 04:20:06 12:02:26 mira master ubuntu 18.04 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml} 3
fail 4404933 2019-10-12 00:27:40 2019-10-12 16:23:45 2019-10-12 20:27:48 4:04:03 3:50:33 0:13:30 mira master centos 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/fastclose.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-small-objects-fast-read-overwrites.yaml} 2
Failure Reason:

"2019-10-12T16:49:14.651789+0000 mon.b (mon.0) 84 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

fail 4404934 2019-10-12 00:27:40 2019-10-12 16:39:14 2019-10-12 17:23:13 0:43:59 0:34:25 0:09:34 mira master rhel 7.7 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
Failure Reason:

"2019-10-12T17:08:23.071016+0000 mon.a (mon.0) 23 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4404935 2019-10-12 00:27:41 2019-10-12 17:22:11 2019-10-12 21:28:14 4:06:03 3:48:05 0:17:58 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-partial-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/cache-agent-small.yaml} 2
Failure Reason:

"2019-10-12T19:57:59.101522+0000 mon.b (mon.1) 30 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404936 2019-10-12 00:27:42 2019-10-12 17:23:22 2019-10-13 05:25:44 12:02:22 mira master centos 7.6 rados/singleton/{all/mon-config.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

psutil.NoSuchProcess process no longer exists (pid=5742)

pass 4404937 2019-10-12 00:27:43 2019-10-12 17:29:42 2019-10-12 18:09:41 0:39:59 0:32:14 0:07:45 mira master rhel 7.7 rados/objectstore/{backends/filestore-idempotent.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404938 2019-10-12 00:27:44 2019-10-12 18:09:53 2019-10-12 22:43:56 4:34:03 4:21:39 0:12:24 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml} 2
Failure Reason:

Command failed on mira083 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0'

fail 4404939 2019-10-12 00:27:45 2019-10-12 18:14:06 2019-10-12 19:32:06 1:18:00 1:03:26 0:14:34 mira master centos 7.6 rados/monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/one.yaml workloads/rados_mon_workunits.yaml} 2
Failure Reason:

Command failed on mira118 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

pass 4404940 2019-10-12 00:27:45 2019-10-12 18:31:51 2019-10-12 19:17:50 0:45:59 0:21:24 0:24:35 mira master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/mimic-v1only.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/few.yaml rados.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/test_rbd_api.yaml} 4
fail 4404941 2019-10-12 00:27:46 2019-10-12 19:18:06 2019-10-12 21:24:07 2:06:01 1:51:28 0:14:33 mira master centos 7.6 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/few.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-small-objects.yaml} 2
Failure Reason:

"2019-10-12T19:44:30.510169+0000 mon.b (mon.0) 88 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 4404942 2019-10-12 00:27:47 2019-10-12 19:32:21 2019-10-12 20:18:21 0:46:00 0:39:48 0:06:12 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-partial-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml} 2
pass 4404943 2019-10-12 00:27:48 2019-10-12 19:34:19 2019-10-12 22:16:21 2:42:02 2:21:57 0:20:05 mira master rhel 7.7 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-low-osd-mem-target.yaml supported-random-distro$/{rhel_7.yaml} tasks/prometheus.yaml} 2
fail 4404944 2019-10-12 00:27:49 2019-10-12 20:16:33 2019-10-12 21:14:32 0:57:59 0:46:59 0:11:00 mira master centos 7.6 rados/singleton/{all/osd-recovery-incomplete.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

Command failed on mira056 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

dead 4404945 2019-10-12 00:27:50 2019-10-12 20:18:37 2019-10-13 08:20:59 12:02:22 mira master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/rados_cls_all.yaml} 2
dead 4404946 2019-10-12 00:27:50 2019-10-12 20:28:03 2019-10-13 08:30:29 12:02:26 mira master rhel 7.7 rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/nautilus.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-octopus.yaml 8-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml supported-random-distro$/{rhel_7.yaml} thrashosds-health.yaml} 4
fail 4404947 2019-10-12 00:27:51 2019-10-12 21:14:49 2019-10-12 23:56:50 2:42:01 2:23:45 0:18:16 mira master rhel 7.7 rados/multimon/{clusters/21.yaml msgr-failures/many.yaml msgr/async-v2only.yaml no_pools.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/mon_clock_no_skews.yaml} 3
Failure Reason:

Command failed on mira057 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4404948 2019-10-12 00:27:52 2019-10-12 21:24:08 2019-10-13 00:10:10 2:46:02 2:31:52 0:14:10 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/cache-snaps.yaml} 2
Failure Reason:

"2019-10-12T21:52:20.779481+0000 mon.b (mon.0) 26 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

dead 4404949 2019-10-12 00:27:53 2019-10-12 21:28:29 2019-10-13 09:30:51 12:02:22 mira master rhel 7.7 rados/singleton/{all/osd-recovery.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404950 2019-10-12 00:27:54 2019-10-12 22:16:24 2019-10-13 04:30:29 6:14:05 5:55:08 0:18:57 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/cache.yaml} 2
Failure Reason:

"2019-10-13T00:53:08.864932+0000 mon.b (mon.1) 12 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404951 2019-10-12 00:27:55 2019-10-12 22:43:58 2019-10-13 00:11:59 1:28:01 1:04:55 0:23:06 mira master centos 7.6 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
Failure Reason:

Command failed on mira083 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

pass 4404952 2019-10-12 00:27:55 2019-10-12 23:08:21 2019-10-12 23:36:20 0:27:59 0:21:04 0:06:55 mira master rhel 7.7 rados/singleton-nomsgr/{all/librados_hello_world.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404953 2019-10-12 00:27:56 2019-10-12 23:36:35 2019-10-13 00:12:34 0:35:59 0:21:11 0:14:48 mira master centos 7.6 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=4-m=2.yaml} 3
Failure Reason:

"2019-10-13T00:05:11.006068+0000 mon.c (mon.0) 61 : cluster [WRN] Health check failed: 1/3 mons down, quorum c,a (MON_DOWN)" in cluster log

fail 4404954 2019-10-12 00:27:57 2019-10-12 23:56:56 2019-10-13 02:42:58 2:46:02 2:23:04 0:22:58 mira master centos 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-small-objects-overwrites.yaml} 2
Failure Reason:

"2019-10-13T00:32:41.728592+0000 mon.a (mon.1) 13 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 4404955 2019-10-12 00:27:58 2019-10-13 00:10:12 2019-10-13 00:34:11 0:23:59 0:14:10 0:09:49 mira master ubuntu 18.04 rados/singleton/{all/pg-autoscaler.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 2
Failure Reason:

"2019-10-13T00:25:55.638606+0000 mon.a (mon.1) 15 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 4404956 2019-10-12 00:27:59 2019-10-13 00:12:00 2019-10-13 01:26:00 1:14:00 1:04:18 0:09:42 mira master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/pool-snaps-few-objects.yaml} 2
Failure Reason:

"2019-10-13T00:27:34.781523+0000 mon.b (mon.0) 18 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

fail 4404957 2019-10-12 00:28:00 2019-10-13 00:12:49 2019-10-13 02:44:50 2:32:01 2:14:21 0:17:40 mira master centos 7.6 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
Failure Reason:

"2019-10-13T00:44:36.418005+0000 mon.b (mon.1) 33 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4404958 2019-10-12 00:28:01 2019-10-13 00:34:13 2019-10-13 01:42:13 1:08:00 0:56:54 0:11:06 mira master ubuntu 18.04 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/osd-delay.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=clay-k=4-m=2.yaml} 2
Failure Reason:

Command failed on mira110 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

pass 4404959 2019-10-12 00:28:01 2019-10-13 01:26:15 2019-10-13 01:52:14 0:25:59 0:19:14 0:06:45 mira master rhel 7.7 rados/singleton/{all/pg-removal-interruption.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404960 2019-10-12 00:28:02 2019-10-13 01:42:28 2019-10-13 09:42:39 8:00:11 7:45:10 0:15:01 mira master centos 7.6 rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/sync-many.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

Command failed on mira035 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

fail 4404961 2019-10-12 00:28:03 2019-10-13 01:52:16 2019-10-13 02:40:16 0:48:00 0:34:35 0:13:25 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

"2019-10-13T02:17:02.094975+0000 mon.b (mon.0) 22 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

dead 4404962 2019-10-12 00:28:04 2019-10-13 02:40:31 2019-10-13 14:42:53 12:02:22 mira master rhel 7.7 rados/singleton-nomsgr/{all/msgr.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4404963 2019-10-12 00:28:05 2019-10-13 02:43:13 2019-10-13 03:25:13 0:42:00 0:35:12 0:06:48 mira master rhel 7.7 rados/singleton/{all/radostool.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404964 2019-10-12 00:28:06 2019-10-13 02:45:05 2019-10-13 03:13:05 0:28:00 0:10:05 0:17:55 mira master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/nautilus-v1only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/default.yaml thrashosds-health.yaml workloads/radosbench.yaml} 4
Failure Reason:

Command failed on mira117 with status 1: 'sudo yum -y install ceph-daemon'

dead 4404965 2019-10-12 00:28:07 2019-10-13 03:13:21 2019-10-13 15:15:43 12:02:22 mira master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/rados_stress_watch.yaml} 2
pass 4404966 2019-10-12 00:28:08 2019-10-13 03:25:29 2019-10-13 04:05:28 0:39:59 0:30:40 0:09:19 mira master rhel 7.7 rados/singleton/{all/random-eio.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 2
pass 4404967 2019-10-12 00:28:08 2019-10-13 04:05:31 2019-10-13 04:29:30 0:23:59 0:17:51 0:06:08 mira master rhel 7.7 rados/objectstore/{backends/keyvaluedb.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4404968 2019-10-12 00:28:09 2019-10-13 04:20:08 2019-10-13 06:22:09 2:02:01 1:55:48 0:06:13 mira master rhel 7.7 rados/standalone/{supported-random-distro$/{rhel_7.yaml} workloads/osd.yaml} 1
Failure Reason:

Command failed (workunit test osd/osd-recovery-prio.sh) on mira066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0dca79308d83afd02b99aa1aa8839d156215298c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh'

pass 4404969 2019-10-12 00:28:10 2019-10-13 04:29:45 2019-10-13 04:57:45 0:28:00 0:19:57 0:08:03 mira master rhel 7.7 rados/multimon/{clusters/3.yaml msgr-failures/few.yaml msgr/async.yaml no_pools.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/mon_clock_with_skews.yaml} 2
fail 4404970 2019-10-12 00:28:11 2019-10-13 04:30:31 2019-10-13 06:22:32 1:52:01 1:20:47 0:31:14 mira master centos rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/default/{default.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml tasks/rados_api_tests.yaml validater/valgrind.yaml} 2
Failure Reason:

saw valgrind issues

fail 4404971 2019-10-12 00:28:12 2019-10-13 04:57:47 2019-10-13 05:27:46 0:29:59 0:18:55 0:11:04 mira master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-partial-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/redirect.yaml} 2
Failure Reason:

"2019-10-13T05:13:30.112184+0000 mon.b (mon.1) 14 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 4404972 2019-10-12 00:28:13 2019-10-13 05:25:46 2019-10-13 05:55:46 0:30:00 0:23:03 0:06:57 mira master rhel 7.7 rados/perf/{ceph.yaml objectstore/bluestore-stupid.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/sample_radosbench.yaml} 1
fail 4404973 2019-10-12 00:28:13 2019-10-13 05:27:48 2019-10-13 06:39:48 1:12:00 1:03:07 0:08:53 mira master centos 7.6 rados/singleton/{all/rebuild-mondb.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

Command failed on mira056 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4404974 2019-10-12 00:28:14 2019-10-13 05:55:59 2019-10-13 07:07:59 1:12:00 1:02:11 0:09:49 mira master ubuntu 18.04 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
Failure Reason:

"2019-10-13T06:12:04.522496+0000 mon.a (mon.0) 23 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4404975 2019-10-12 00:28:15 2019-10-13 06:22:11 2019-10-13 06:58:10 0:35:59 0:10:08 0:25:51 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-async-partial-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/redirect_promote_tests.yaml} 2
Failure Reason:

Command failed on mira062 with status 1: 'sudo yum -y install ceph-radosgw'

fail 4404976 2019-10-12 00:28:16 2019-10-13 06:22:33 2019-10-13 09:50:36 3:28:03 3:09:05 0:18:58 mira master rhel 7.7 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/fastread.yaml thrashosds-health.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml} 3
Failure Reason:

Command failed on mira107 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4404977 2019-10-12 00:28:17 2019-10-13 06:40:04 2019-10-13 07:58:04 1:18:00 1:03:43 0:14:17 mira master centos 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/osd-delay.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{centos_7.yaml} thrashers/fastread.yaml thrashosds-health.yaml workloads/ec-snaps-few-objects-overwrites.yaml} 2
Failure Reason:

Command failed on mira057 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4404978 2019-10-12 00:28:18 2019-10-13 06:58:12 2019-10-13 09:38:14 2:40:02 2:11:17 0:28:45 mira master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/rados_striper.yaml} 2
Failure Reason:

"2019-10-13T07:40:16.054770+0000 mon.a (mon.0) 26 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4404979 2019-10-12 00:28:19 2019-10-13 07:08:15 2019-10-13 07:36:14 0:27:59 0:11:02 0:16:57 mira master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/nautilus-v2only.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/few.yaml rados.yaml thrashers/mapgap.yaml thrashosds-health.yaml workloads/rbd_cls.yaml} 4
Failure Reason:

Command failed on mira101 with status 1: 'sudo yum -y install ceph-daemon'

fail 4404980 2019-10-12 00:28:19 2019-10-13 07:36:29 2019-10-13 09:30:30 1:54:01 1:39:44 0:14:17 mira master centos 7.6 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/fastclose.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml} 2
Failure Reason:

"2019-10-13T08:03:17.348681+0000 mon.a (mon.0) 14 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4404981 2019-10-12 00:28:20 2019-10-13 07:58:18 2019-10-13 08:24:17 0:25:59 0:16:20 0:09:39 mira master ubuntu 18.04 rados/perf/{ceph.yaml objectstore/bluestore-stupid.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{ubuntu_latest.yaml} workloads/cosbench_64K_read_write.yaml} 1
Failure Reason:

Command failed on mira056 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'

fail 4404982 2019-10-12 00:28:21 2019-10-13 08:21:14 2019-10-13 10:33:16 2:12:02 2:04:15 0:07:47 mira master rhel 7.7 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

Command failed (workunit test cephtool/test.sh) on mira118 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0dca79308d83afd02b99aa1aa8839d156215298c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'

fail 4404983 2019-10-12 00:28:22 2019-10-13 08:24:18 2019-10-13 08:58:18 0:34:00 0:27:22 0:06:38 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-partial-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/set-chunk-promote-flush.yaml} 2
Failure Reason:

"2019-10-13T08:49:21.746470+0000 mon.b (mon.1) 29 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

dead 4404984 2019-10-12 00:28:23 2019-10-13 08:30:43 2019-10-13 20:33:05 12:02:22 mira master ubuntu 18.04 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
pass 4404985 2019-10-12 00:28:24 2019-10-13 08:58:21 2019-10-13 09:26:20 0:27:59 0:21:17 0:06:42 mira master rhel 7.7 rados/singleton/{all/test-crash.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4404986 2019-10-12 00:28:25 2019-10-13 09:26:35 2019-10-13 10:04:34 0:37:59 0:31:59 0:06:00 mira master rhel 7.7 rados/perf/{ceph.yaml objectstore/bluestore-basic-min-osd-mem-target.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/cosbench_64K_write.yaml} 1
fail 4404987 2019-10-12 00:28:25 2019-10-13 09:30:32 2019-10-13 11:20:33 1:50:01 1:41:11 0:08:50 mira master rhel 7.7 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-partial-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/set-chunks-read.yaml} 2
Failure Reason:

"2019-10-13T09:56:35.879103+0000 mon.a (mon.1) 12 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 4404988 2019-10-12 00:28:26 2019-10-13 09:30:52 2019-10-13 10:20:52 0:50:00 0:42:56 0:07:04 mira master rhel 7.7 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/rados_workunit_loadgen_big.yaml} 2
fail 4404989 2019-10-12 00:28:27 2019-10-13 09:38:16 2019-10-13 10:10:15 0:31:59 0:10:07 0:21:52 mira master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/nautilus.yaml backoff/normal.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} 4
Failure Reason:

Command failed on mira062 with status 1: 'sudo yum -y install ceph-daemon'

fail 4404990 2019-10-12 00:28:28 2019-10-13 09:42:41 2019-10-13 10:14:40 0:31:59 0:21:54 0:10:05 mira master ubuntu 18.04 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-bitmap.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/dashboard.yaml} 2
Failure Reason:

Test failure: test_perf_counters_mds_get (tasks.mgr.dashboard.test_perf_counters.PerfCountersControllerTest)

fail 4404991 2019-10-12 00:28:29 2019-10-13 09:50:51 2019-10-13 11:32:51 1:42:00 1:32:52 0:09:08 mira master rhel 7.7 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-async-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
Failure Reason:

"2019-10-13T10:17:06.848549+0000 mon.a (mon.1) 32 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 4404992 2019-10-12 00:28:30 2019-10-13 10:04:36 2019-10-13 11:26:36 1:22:00 1:12:19 0:09:41 mira master ubuntu 18.04 rados/standalone/{supported-random-distro$/{ubuntu_latest.yaml} workloads/scrub.yaml} 1
pass 4404993 2019-10-12 00:28:30 2019-10-13 10:10:30 2019-10-13 10:36:29 0:25:59 0:20:38 0:05:21 mira master rhel 7.7 rados/perf/{ceph.yaml objectstore/bluestore-bitmap.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/fio_4K_rand_read.yaml} 1
fail 4404994 2019-10-12 00:28:31 2019-10-13 10:14:43 2019-10-13 11:28:43 1:14:00 1:02:12 0:11:48 mira master rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/none.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml tasks/rados_cls_all.yaml validater/lockdep.yaml} 2
Failure Reason:

"2019-10-13T10:32:59.407835+0000 mon.b (mon.1) 19 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

dead 4404995 2019-10-12 00:28:32 2019-10-13 10:20:54 2019-10-13 22:23:15 12:02:21 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} 2
fail 4404996 2019-10-12 00:28:33 2019-10-13 10:33:30 2019-10-13 11:19:30 0:46:00 0:30:51 0:15:09 mira master centos 7.6 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{centos_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=4-m=2.yaml} 3
Failure Reason:

"2019-10-13T11:07:24.064471+0000 mon.c (mon.1) 17 : cluster [WRN] Health check update: 1/3 mons down, quorum c,a (MON_DOWN)" in cluster log

dead 4404997 2019-10-12 00:28:34 2019-10-13 10:36:31 2019-10-13 22:38:52 12:02:21 mira master ubuntu 18.04 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/fastclose.yaml rados.yaml recovery-overrides/{more-partial-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/minsize_recovery.yaml thrashosds-health.yaml workloads/ec-pool-snaps-few-objects-overwrites.yaml} 2
fail 4404998 2019-10-12 00:28:35 2019-10-13 11:19:45 2019-10-13 13:29:47 2:10:02 1:56:05 0:13:57 mira master centos 7.6 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-async-partial-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=3-m=1.yaml} 2
Failure Reason:

"2019-10-13T11:45:47.845174+0000 mon.a (mon.1) 25 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 4404999 2019-10-12 00:28:36 2019-10-13 11:20:34 2019-10-13 11:52:34 0:32:00 0:16:35 0:15:25 mira master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-async-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/write_fadvise_dontneed.yaml} 2
Failure Reason:

Command failed on mira101 with status 6: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0"

fail 4405000 2019-10-12 00:28:36 2019-10-13 11:26:51 2019-10-13 13:56:52 2:30:01 2:15:00 0:15:01 mira master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/rados_workunit_loadgen_mix.yaml} 2
Failure Reason:

"2019-10-13T11:53:48.556063+0000 mon.a (mon.0) 23 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

pass 4405001 2019-10-12 00:28:37 2019-10-13 11:28:57 2019-10-13 12:26:57 0:58:00 0:33:54 0:24:06 mira master centos 7.6 rados/singleton-nomsgr/{all/recovery-unfound-found.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
pass 4405002 2019-10-12 00:28:38 2019-10-13 11:32:54 2019-10-13 12:14:53 0:41:59 0:32:26 0:09:33 mira master rhel 7.7 rados/singleton/{all/thrash-rados/{thrash-rados.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 2