Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 4414521 2019-10-15 08:36:01 2019-10-15 08:38:56 2019-10-15 09:40:56 1:02:00 0:18:00 0:44:00 smithi master rhel 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/rados_python.yaml} 2
Failure Reason:

"2019-10-15T09:34:08.829846+0000 mon.b (mon.1) 15 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 4414522 2019-10-15 08:36:02 2019-10-15 08:44:27 2019-10-15 13:44:31 5:00:04 0:50:37 4:09:27 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/luminous.yaml backoff/normal.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/default.yaml thrashosds-health.yaml workloads/radosbench.yaml} 4
pass 4414523 2019-10-15 08:36:02 2019-10-15 08:44:27 2019-10-15 09:12:26 0:27:59 0:19:36 0:08:23 smithi master centos 7.6 rados/singleton-nomsgr/{all/msgr.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
fail 4414524 2019-10-15 08:36:03 2019-10-15 08:49:14 2019-10-15 12:33:16 3:44:02 3:27:08 0:16:54 smithi master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_stress_watch.yaml} 2
Failure Reason:

"2019-10-15T09:14:27.131870+0000 mon.b (mon.0) 12 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4414525 2019-10-15 08:36:04 2019-10-15 08:49:16 2019-10-15 10:57:17 2:08:01 1:59:15 0:08:46 smithi master centos 7.6 rados/singleton/{all/mon-config.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

"2019-10-15T09:05:28.316311+0000 mon.a (mon.0) 16 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4414526 2019-10-15 08:36:05 2019-10-15 08:49:25 2019-10-15 10:15:25 1:26:00 1:08:33 0:17:27 smithi master rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/default/{default.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml tasks/rados_cls_all.yaml validater/lockdep.yaml} 2
Failure Reason:

Command failed on smithi036 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

fail 4414527 2019-10-15 08:36:05 2019-10-15 08:53:50 2019-10-15 09:23:50 0:30:00 0:19:15 0:10:45 smithi master ubuntu 18.04 rados/singleton/{all/osd-backfill.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

Command failed on smithi146 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 4414528 2019-10-15 08:36:06 2019-10-15 08:53:56 2019-10-15 10:37:56 1:44:00 1:33:15 0:10:45 smithi master centos 7.6 rados/singleton/{all/osd-recovery.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

"2019-10-15T09:12:45.463647+0000 mon.a (mon.0) 25 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4414529 2019-10-15 08:36:07 2019-10-15 08:57:31 2019-10-15 12:39:34 3:42:03 3:18:51 0:23:12 smithi master rhel 7.6 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

Command failed (workunit test cephtool/test.sh) on smithi014 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41a92c66bd9507a384765ff15408d26dab705bdc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'

fail 4414530 2019-10-15 08:36:08 2019-10-15 08:57:39 2019-10-15 11:07:40 2:10:01 1:33:38 0:36:23 smithi master centos 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/fastclose.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-small-objects-overwrites.yaml} 2
Failure Reason:

"2019-10-15T09:46:17.653017+0000 mon.a (mon.0) 19 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4414531 2019-10-15 08:36:08 2019-10-15 09:03:32 2019-10-15 10:27:32 1:24:00 0:56:33 0:27:27 smithi master ubuntu 18.04 rados/singleton/{all/peer.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

Command failed on smithi146 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4414532 2019-10-15 08:36:09 2019-10-15 09:03:37 2019-10-15 09:37:37 0:34:00 0:21:26 0:12:34 smithi master centos 7.6 rados/perf/{ceph.yaml objectstore/bluestore-basic-min-osd-mem-target.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{centos_7.yaml} workloads/cosbench_64K_read_write.yaml} 1
Failure Reason:

Command failed on smithi005 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'

fail 4414533 2019-10-15 08:36:10 2019-10-15 09:11:14 2019-10-15 12:11:16 3:00:02 2:45:23 0:14:39 smithi master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/rados_workunit_loadgen_big.yaml} 2
Failure Reason:

"2019-10-15T09:33:35.831922+0000 mon.b (mon.0) 28 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4414534 2019-10-15 08:36:11 2019-10-15 09:12:42 2019-10-15 10:32:43 1:20:01 0:36:56 0:43:05 smithi master centos 7.6 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{centos_7.yaml} thrashers/fastread.yaml thrashosds-health.yaml workloads/ec-small-objects.yaml} 2
Failure Reason:

"2019-10-15T10:03:56.424454+0000 mon.a (mon.1) 14 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 4414535 2019-10-15 08:36:11 2019-10-15 09:15:37 2019-10-15 09:59:36 0:43:59 0:13:42 0:30:17 smithi master centos 7.6 rados/singleton/{all/pg-autoscaler.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 2
fail 4414536 2019-10-15 08:36:12 2019-10-15 09:15:37 2019-10-15 10:03:37 0:48:00 0:16:17 0:31:43 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/cache-agent-big.yaml} 2
Failure Reason:

"2019-10-15T09:55:57.874082+0000 mon.a (mon.0) 13 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 4414537 2019-10-15 08:36:13 2019-10-15 09:17:39 2019-10-15 10:15:39 0:58:00 0:19:19 0:38:41 smithi master ubuntu 18.04 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-comp.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/dashboard.yaml} 2
Failure Reason:

Test failure: test_perf_counters_mds_get (tasks.mgr.dashboard.test_perf_counters.PerfCountersControllerTest)

fail 4414538 2019-10-15 08:36:14 2019-10-15 09:17:52 2019-10-15 10:45:53 1:28:01 1:06:41 0:21:20 smithi master rhel 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/rados_workunit_loadgen_mix.yaml} 2
Failure Reason:

"2019-10-15T09:49:28.883594+0000 mon.a (mon.1) 29 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 4414539 2019-10-15 08:36:14 2019-10-15 09:22:03 2019-10-15 10:50:03 1:28:00 0:56:46 0:31:14 smithi master ubuntu 18.04 rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/one.yaml workloads/snaps-few-objects.yaml} 2
Failure Reason:

Command failed on smithi042 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

dead 4414540 2019-10-15 08:36:15 2019-10-15 09:24:05 2019-10-15 21:26:31 12:02:26 smithi master centos 7.6 rados/objectstore/{backends/objectstore.yaml supported-random-distro$/{centos_7.yaml}} 1
fail 4414541 2019-10-15 08:36:16 2019-10-15 09:24:05 2019-10-15 10:50:06 1:26:01 1:04:05 0:21:56 smithi master ubuntu 18.04 rados/singleton/{all/rebuild-mondb.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

Command failed on smithi005 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

dead 4414542 2019-10-15 08:36:17 2019-10-15 09:33:17 2019-10-15 21:35:39 12:02:22 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml} 2
fail 4414543 2019-10-15 08:36:18 2019-10-15 09:37:07 2019-10-15 10:43:07 1:06:00 0:51:34 0:14:26 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/nautilus-v2only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/cache-snaps.yaml} 4
Failure Reason:

Command failed on smithi037 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json'

fail 4414544 2019-10-15 08:36:18 2019-10-15 09:37:38 2019-10-15 10:47:38 1:10:00 0:56:41 0:13:19 smithi master ubuntu 18.04 rados/singleton/{all/recovery-preemption.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

Command failed on smithi071 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4414545 2019-10-15 08:36:19 2019-10-15 09:38:43 2019-10-15 10:52:43 1:14:00 1:01:55 0:12:05 smithi master centos 7.6 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

"2019-10-15T09:57:22.916269+0000 mon.b (mon.1) 10 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4414546 2019-10-15 08:36:20 2019-10-15 09:40:09 2019-10-15 14:24:12 4:44:03 4:30:24 0:13:39 smithi master rhel 7.6 rados/upgrade/mimic-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/mimic.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-nautilus.yaml 8-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml supported-random-distro$/{rhel_7.yaml} thrashosds-health.yaml} 4
Failure Reason:

failed to recover before timeout expired

fail 4414547 2019-10-15 08:36:21 2019-10-15 09:40:58 2019-10-15 10:10:57 0:29:59 0:22:05 0:07:54 smithi master rhel 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/readwrite.yaml} 2
Failure Reason:

"2019-10-15T10:01:12.701540+0000 mon.b (mon.0) 99 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4414548 2019-10-15 08:36:21 2019-10-15 09:42:21 2019-10-15 11:10:21 1:28:00 0:34:22 0:53:38 smithi master rhel 7.6 rados/multimon/{clusters/21.yaml msgr-failures/few.yaml msgr/async-v2only.yaml no_pools.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/mon_clock_with_skews.yaml} 3
Failure Reason:

Command failed on smithi146 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status'

fail 4414549 2019-10-15 08:36:22 2019-10-15 09:43:19 2019-10-15 11:21:20 1:38:01 1:23:46 0:14:15 smithi master rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/default/{default.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml tasks/rados_api_tests.yaml validater/lockdep.yaml} 2
Failure Reason:

"2019-10-15T10:01:06.661179+0000 mon.a (mon.0) 20 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4414550 2019-10-15 08:36:23 2019-10-15 09:47:44 2019-10-15 10:45:43 0:57:59 0:19:11 0:38:48 smithi master ubuntu 18.04 rados/monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync-many.yaml workloads/pool-create-delete.yaml} 2
Failure Reason:

Command failed on smithi167 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

dead 4414551 2019-10-15 08:36:24 2019-10-15 09:47:44 2019-10-15 21:50:10 12:02:26 smithi master ubuntu 18.04 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=4-m=2.yaml} 3
dead 4414552 2019-10-15 08:36:25 2019-10-15 09:48:02 2019-10-15 21:50:28 12:02:26 smithi master ubuntu 18.04 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/osd-delay.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/minsize_recovery.yaml thrashosds-health.yaml workloads/ec-pool-snaps-few-objects-overwrites.yaml} 2
fail 4414553 2019-10-15 08:36:25 2019-10-15 09:51:48 2019-10-15 12:11:49 2:20:01 2:01:49 0:18:12 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

"2019-10-15T10:15:24.258452+0000 mon.b (mon.1) 25 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4414554 2019-10-15 08:36:26 2019-10-15 09:54:13 2019-10-15 11:04:13 1:10:00 0:40:35 0:29:25 smithi master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/repair_test.yaml} 2
Failure Reason:

"2019-10-15T10:32:18.068133+0000 mon.b (mon.1) 43 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

pass 4414555 2019-10-15 08:36:27 2019-10-15 09:59:54 2019-10-15 11:19:54 1:20:00 0:38:32 0:41:28 smithi master rhel 7.6 rados/singleton/{all/thrash-eio.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 2
fail 4414556 2019-10-15 08:36:28 2019-10-15 10:03:39 2019-10-15 11:19:39 1:16:00 1:03:03 0:12:57 smithi master rhel 7.6 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
Failure Reason:

Command failed on smithi117 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4414557 2019-10-15 08:36:28 2019-10-15 10:10:58 2019-10-15 10:40:58 0:30:00 0:21:33 0:08:27 smithi master rhel 7.6 rados/singleton/{all/watch-notify-same-primary.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

"2019-10-15T10:30:31.401065+0000 mon.a (mon.0) 29 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 4414558 2019-10-15 08:36:29 2019-10-15 10:15:42 2019-10-15 11:25:42 1:10:00 1:02:37 0:07:23 smithi master rhel 7.6 rados/multimon/{clusters/3.yaml msgr-failures/many.yaml msgr/async.yaml no_pools.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/mon_recovery.yaml} 2
Failure Reason:

Command failed on smithi023 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4414559 2019-10-15 08:36:30 2019-10-15 10:15:42 2019-10-15 12:29:43 2:14:01 1:37:02 0:36:59 smithi master centos rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/none.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml tasks/rados_cls_all.yaml validater/valgrind.yaml} 2
Failure Reason:

"2019-10-15T11:16:18.542936+0000 mon.a (mon.1) 37 : cluster [WRN] Health check update: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

fail 4414560 2019-10-15 08:36:31 2019-10-15 10:15:42 2019-10-15 11:33:42 1:18:00 0:40:07 0:37:53 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/set-chunks-read.yaml} 2
Failure Reason:

"2019-10-15T10:58:32.173997+0000 mon.b (mon.0) 37 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

dead 4414561 2019-10-15 08:36:32 2019-10-15 10:16:00 2019-10-15 22:18:21 12:02:21 smithi master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/scrub_test.yaml} 2
fail 4414562 2019-10-15 08:36:32 2019-10-15 10:16:10 2019-10-15 12:02:11 1:46:01 1:10:46 0:35:15 smithi master rhel 7.6 rados/monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/force-sync-many.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed on smithi167 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

fail 4414563 2019-10-15 08:36:33 2019-10-15 10:27:49 2019-10-15 12:03:50 1:36:01 1:16:19 0:19:42 smithi master centos 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{centos_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-small-objects-overwrites.yaml} 2
Failure Reason:

"2019-10-15T10:58:36.408293+0000 mon.b (mon.1) 48 : cluster [WRN] Health check update: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4414564 2019-10-15 08:36:34 2019-10-15 10:29:42 2019-10-15 13:39:45 3:10:03 2:56:49 0:13:14 smithi master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_api_tests.yaml} 2
Failure Reason:

"2019-10-15T10:46:39.463061+0000 mon.b (mon.0) 64 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" in cluster log

fail 4414565 2019-10-15 08:36:35 2019-10-15 10:31:50 2019-10-15 11:51:50 1:20:00 0:59:37 0:20:23 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/admin_socket_objecter_requests.yaml} 2
Failure Reason:

Command failed on smithi037 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4414566 2019-10-15 08:36:36 2019-10-15 10:32:44 2019-10-15 11:40:44 1:08:00 0:59:34 0:08:26 smithi master ubuntu 18.04 rados/singleton/{all/ec-lost-unfound.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

"2019-10-15T10:46:14.367359+0000 mon.a (mon.0) 27 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,b (MON_DOWN)" in cluster log

fail 4414567 2019-10-15 08:36:36 2019-10-15 10:37:15 2019-10-15 12:39:16 2:02:01 0:56:42 1:05:19 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml} 2
Failure Reason:

Command failed on smithi183 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4414568 2019-10-15 08:36:37 2019-10-15 10:37:58 2019-10-15 12:09:58 1:32:00 1:22:56 0:09:04 smithi master rhel 7.6 rados/singleton/{all/lost-unfound.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

Command failed on smithi036 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1'

fail 4414569 2019-10-15 08:36:38 2019-10-15 10:41:15 2019-10-15 12:15:15 1:34:00 1:02:54 0:31:06 smithi master rhel 7.6 rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/many.yaml workloads/rados_mon_osdmap_prune.yaml} 2
Failure Reason:

Command failed on smithi040 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4414570 2019-10-15 08:36:39 2019-10-15 10:43:25 2019-10-15 12:13:25 1:30:00 0:57:22 0:32:38 smithi master ubuntu 18.04 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml} 3
Failure Reason:

Command failed on smithi139 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

fail 4414571 2019-10-15 08:36:39 2019-10-15 10:46:00 2019-10-15 13:52:03 3:06:03 2:54:11 0:11:52 smithi master centos 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/few.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{centos_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-pool-snaps-few-objects-overwrites.yaml} 2
Failure Reason:

Command failed on smithi101 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it'

fail 4414572 2019-10-15 08:36:40 2019-10-15 10:46:01 2019-10-15 12:52:02 2:06:01 1:52:18 0:13:43 smithi master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/rados_python.yaml} 2
Failure Reason:

Command failed (workunit test rados/test_python.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41a92c66bd9507a384765ff15408d26dab705bdc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh'

dead 4414573 2019-10-15 08:36:41 2019-10-15 10:47:54 2019-10-15 22:50:16 12:02:22 smithi master centos 7.6 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
fail 4414574 2019-10-15 08:36:42 2019-10-15 10:49:45 2019-10-15 12:11:45 1:22:00 0:57:35 0:24:25 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/cache.yaml} 2
Failure Reason:

Command failed on smithi093 with status 1: 'sudo ceph --cluster ceph osd crush tunables default'

dead 4414575 2019-10-15 08:36:42 2019-10-15 10:50:05 2019-10-15 22:52:26 12:02:21 11:53:49 0:08:32 smithi master rhel 7.6 rados/multimon/{clusters/9.yaml msgr-failures/many.yaml msgr/async-v2only.yaml no_pools.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/mon_clock_with_skews.yaml} 3
Failure Reason:

SSH connection to smithi016 was lost: "sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --"

fail 4414576 2019-10-15 08:36:43 2019-10-15 10:50:07 2019-10-15 18:18:14 7:28:07 6:38:21 0:49:46 smithi master centos rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/none.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml tasks/rados_api_tests.yaml validater/valgrind.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on smithi072 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=41a92c66bd9507a384765ff15408d26dab705bdc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'