Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
dead 3251331 2018-11-13 15:06:43 2018-11-13 15:08:34 2018-11-13 16:36:35 1:28:01 smithi master rhel 7.5 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml supported-random-distro$/{rhel_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/write_fadvise_dontneed.yaml} 2
dead 3251332 2018-11-13 15:06:44 2018-11-13 15:14:33 2018-11-13 16:36:33 1:22:00 smithi master centos 7.5 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/admin_socket_objecter_requests.yaml} 2
pass 3251333 2018-11-13 15:06:44 2018-11-13 15:16:34 2018-11-13 15:38:33 0:21:59 0:10:07 0:11:52 smithi master rhel 7.5 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore.yaml supported-random-distro$/{rhel_latest.yaml} tasks/workunits.yaml} 2
dead 3251334 2018-11-13 15:06:45 2018-11-13 15:20:29 2018-11-13 16:36:29 1:16:00 smithi master centos 7.5 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/simple.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-agent-big.yaml} 2
pass 3251335 2018-11-13 15:06:46 2018-11-13 15:22:28 2018-11-13 15:50:28 0:28:00 0:15:21 0:12:39 smithi master centos 7.5 rados/singleton/{all/max-pg-per-osd.from-replica.yaml msgr-failures/many.yaml msgr/simple.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_latest.yaml}} 1
fail 3251336 2018-11-13 15:06:47 2018-11-13 15:24:11 2018-11-13 15:54:10 0:29:59 0:13:24 0:16:35 smithi master rhel 7.5 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml rocksdb.yaml supported-random-distro$/{rhel_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/cache-agent-small.yaml} 2
Failure Reason:

Command failed on smithi197 with status 11: u'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 3.1'

dead 3251337 2018-11-13 15:06:48 2018-11-13 15:26:39 2018-11-13 16:34:39 1:08:00 smithi master rhel 7.5 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml supported-random-distro$/{rhel_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml} 2
dead 3251338 2018-11-13 15:06:48 2018-11-13 15:28:30 2018-11-13 16:36:30 1:08:00 smithi master rhel 7.5 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/simple.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml supported-random-distro$/{rhel_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml} 2
dead 3251339 2018-11-13 15:06:49 2018-11-13 15:30:17 2018-11-13 16:36:17 1:06:00 0:55:18 0:10:42 smithi master rhel 7.5 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml supported-random-distro$/{rhel_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/cache-snaps.yaml} 2
Failure Reason:

SSH connection to smithi199 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op snap_remove 50 --op snap_create 50 --op rollback 50 --op read 100 --op copy_from 50 --op write 50 --op write_excl 50 --op cache_try_flush 50 --op cache_flush 50 --op cache_evict 50 --op delete 50 --pool base'

fail 3251340 2018-11-13 15:06:50 2018-11-13 15:30:17 2018-11-13 15:54:16 0:23:59 0:11:21 0:12:38 smithi master centos 7.5 rados/singleton/{all/mon-auth-caps.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore.yaml rados.yaml supported-random-distro$/{centos_latest.yaml}} 1
Failure Reason:

"2018-11-13 15:50:14.107112 mon.a (mon.0) 315 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

dead 3251341 2018-11-13 15:06:50 2018-11-13 15:34:18 2018-11-13 16:36:18 1:02:00 smithi master centos 7.5 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/filestore-xfs.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/cache.yaml} 2
dead 3251342 2018-11-13 15:06:51 2018-11-13 15:34:22 2018-11-13 16:36:23 1:02:01 smithi master ubuntu 18.04 rados/objectstore/{backends/filestore-idempotent.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
pass 3251343 2018-11-13 15:06:52 2018-11-13 15:36:39 2018-11-13 15:54:39 0:18:00 0:11:34 0:06:26 smithi master rhel 7.5 rados/perf/{ceph.yaml objectstore/filestore-xfs.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_latest.yaml} workloads/sample_radosbench.yaml} 1
fail 3251344 2018-11-13 15:06:53 2018-11-13 15:38:31 2018-11-13 16:06:31 0:28:00 0:02:52 0:25:08 smithi master ubuntu 16.04 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/hammer.yaml backoff/normal.yaml ceph.yaml clusters/{openstack.yaml two-plus-three.yaml} d-balancer/off.yaml distro$/{ubuntu_16.04.yaml} msgr-failures/few.yaml msgr/async.yaml rados.yaml rocksdb.yaml thrashers/none.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} 3
Failure Reason:

Command failed on smithi073 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-mds=0.94.10-85-ga8e54ce-1xenial rbd-fuse=0.94.10-85-ga8e54ce-1xenial librbd1=0.94.10-85-ga8e54ce-1xenial ceph-fuse=0.94.10-85-ga8e54ce-1xenial python-ceph=0.94.10-85-ga8e54ce-1xenial ceph-common=0.94.10-85-ga8e54ce-1xenial libcephfs-java=0.94.10-85-ga8e54ce-1xenial ceph=0.94.10-85-ga8e54ce-1xenial libcephfs-jni=0.94.10-85-ga8e54ce-1xenial ceph-test=0.94.10-85-ga8e54ce-1xenial radosgw=0.94.10-85-ga8e54ce-1xenial'

dead 3251345 2018-11-13 15:06:53 2018-11-13 15:38:35 2018-11-13 16:36:35 0:58:00 smithi master ubuntu 16.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/simple.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml supported-random-distro$/{ubuntu_16.04.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/pool-snaps-few-objects.yaml} 2
dead 3251346 2018-11-13 15:06:54 2018-11-13 15:48:36 2018-11-13 16:36:36 0:48:00 smithi master centos 7.5 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/rados_api_tests.yaml} 2
fail 3251347 2018-11-13 15:06:55 2018-11-13 15:50:34 2018-11-13 16:18:33 0:27:59 0:13:41 0:14:18 smithi master ubuntu 16.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} mon_kv_backend/rocksdb.yaml msgr-failures/few.yaml msgr/simple.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_16.04.yaml} tasks/rados_stress_watch.yaml} 2
Failure Reason:

"2018-11-13 16:12:24.921991 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

dead 3251348 2018-11-13 15:06:56 2018-11-13 15:50:34 2018-11-13 16:36:34 0:46:00 smithi master centos 7.5 rados/monthrash/{ceph.yaml clusters/3-mons.yaml mon_kv_backend/rocksdb.yaml msgr-failures/mon-delay.yaml msgr/random.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_latest.yaml} thrashers/force-sync-many.yaml workloads/rados_mon_osdmap_prune.yaml} 2
dead 3251349 2018-11-13 15:06:57 2018-11-13 15:50:34 2018-11-13 16:36:34 0:46:00 smithi master centos 7.5 rados/multimon/{clusters/6.yaml mon_kv_backend/rocksdb.yaml msgr-failures/many.yaml msgr/simple.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_latest.yaml} tasks/mon_clock_with_skews.yaml} 2
dead 3251350 2018-11-13 15:06:57 2018-11-13 15:50:34 2018-11-13 16:36:34 0:46:00 0:19:29 0:26:31 smithi master centos rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/none.yaml mon_kv_backend/rocksdb.yaml msgr-failures/few.yaml msgr/simple.yaml objectstore/bluestore-bitmap.yaml rados.yaml tasks/rados_api_tests.yaml validater/valgrind.yaml} 2
Failure Reason:

SSH connection to smithi086 was lost: 'sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'

fail 3251351 2018-11-13 15:06:58 2018-11-13 15:52:36 2018-11-13 16:14:35 0:21:59 0:09:01 0:12:58 smithi master ubuntu 18.04 rados/singleton-nomsgr/{all/health-warnings.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

Command failed on smithi097 with status 11: u'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.7'

dead 3251352 2018-11-13 15:06:59 2018-11-13 15:54:21 2018-11-13 16:36:21 0:42:00 smithi master centos 7.5 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/radosbench.yaml} 2
dead 3251353 2018-11-13 15:07:00 2018-11-13 15:54:21 2018-11-13 16:36:21 0:42:00 smithi master rhel 7.5 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/simple.yaml objectstore/filestore-xfs.yaml rados.yaml rocksdb.yaml supported-random-distro$/{rhel_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/redirect.yaml} 2
fail 3251354 2018-11-13 15:07:00 2018-11-13 15:54:40 2018-11-13 16:20:39 0:25:59 0:15:34 0:10:25 smithi master centos 7.5 rados/singleton/{all/mon-config-keys.yaml msgr-failures/many.yaml msgr/random.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_latest.yaml}} 1
Failure Reason:

"2018-11-13 16:15:49.767201 mon.a (mon.0) 782 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

dead 3251355 2018-11-13 15:07:01 2018-11-13 15:56:26 2018-11-13 16:36:26 0:40:00 smithi master ubuntu 16.04 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml leveldb.yaml msgr-failures/osd-delay.yaml objectstore/bluestore.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_16.04.yaml} thrashers/fastread.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml} 2
dead 3251356 2018-11-13 15:07:02 2018-11-13 15:56:26 2018-11-13 16:36:26 0:40:00 smithi master rhel 7.5 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml supported-random-distro$/{rhel_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/redirect_promote_tests.yaml} 2
dead 3251357 2018-11-13 15:07:03 2018-11-13 16:00:24 2018-11-13 16:36:24 0:36:00 smithi master rhel 7.5 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} leveldb.yaml msgr-failures/osd-delay.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{rhel_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
dead 3251358 2018-11-13 15:07:04 2018-11-13 16:02:31 2018-11-13 16:36:30 0:33:59 smithi master rhel 7.5 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml supported-random-distro$/{rhel_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/redirect_set_object.yaml} 2
dead 3251359 2018-11-13 15:07:05 2018-11-13 16:04:21 2018-11-13 16:36:20 0:31:59 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/simple.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/set-chunks-read.yaml} 2
dead 3251360 2018-11-13 15:07:05 2018-11-13 16:06:20 2018-11-13 16:36:20 0:30:00 smithi master centos 7.5 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} leveldb.yaml msgr-failures/fastclose.yaml objectstore/bluestore.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml} 3
dead 3251361 2018-11-13 15:07:06 2018-11-13 16:06:21 2018-11-13 16:36:21 0:30:00 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml rocksdb.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/small-objects.yaml} 2
pass 3251362 2018-11-13 15:07:07 2018-11-13 16:06:24 2018-11-13 16:26:24 0:20:00 0:07:36 0:12:24 smithi master ubuntu 16.04 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore.yaml supported-random-distro$/{ubuntu_16.04.yaml} tasks/crash.yaml} 2
pass 3251363 2018-11-13 15:07:08 2018-11-13 16:06:32 2018-11-13 16:34:32 0:28:00 0:18:57 0:09:03 smithi master centos 7.5 rados/perf/{ceph.yaml objectstore/filestore-xfs.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{centos_latest.yaml} workloads/cosbench_64K_read_write.yaml} 1
dead 3251364 2018-11-13 15:07:08 2018-11-13 16:08:20 2018-11-13 16:36:20 0:28:00 smithi master centos 7.5 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_latest.yaml}} 1
dead 3251365 2018-11-13 15:07:09 2018-11-13 16:08:28 2018-11-13 16:36:28 0:28:00 smithi master rhel 7.5 rados/standalone/{supported-random-distro$/{rhel_latest.yaml} workloads/mon.yaml} 1
dead 3251366 2018-11-13 15:07:10 2018-11-13 16:08:29 2018-11-13 16:36:28 0:27:59 smithi master centos 7.5 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} leveldb.yaml msgr-failures/fastclose.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml}
fail 3251367 2018-11-13 15:07:11 2018-11-13 16:10:34 2018-11-13 16:30:33 0:19:59 0:08:18 0:11:41 smithi master ubuntu 16.04 rados/singleton/{all/mon-config.yaml msgr-failures/few.yaml msgr/simple.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_16.04.yaml}} 1
Failure Reason:

"2018-11-13 16:24:58.310331 mon.a (mon.0) 177 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

dead 3251368 2018-11-13 15:07:11 2018-11-13 16:12:38 2018-11-13 16:36:38 0:24:00 smithi master centos 7.5 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} 2
dead 3251369 2018-11-13 15:07:12 2018-11-13 16:14:25 2018-11-13 16:36:24 0:21:59 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/simple.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/write_fadvise_dontneed.yaml} 2
dead 3251370 2018-11-13 15:07:13 2018-11-13 16:14:25 2018-11-13 16:36:24 0:21:59 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/admin_socket_objecter_requests.yaml} 2
dead 3251371 2018-11-13 15:07:14 2018-11-13 16:14:25 2018-11-13 16:36:24 0:21:59 smithi master ubuntu 18.04 rados/thrash-erasure-code-overwrites/{bluestore.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml leveldb.yaml msgr-failures/fastclose.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-pool-snaps-few-objects-overwrites.yaml} 2
dead 3251372 2018-11-13 15:07:14 2018-11-13 16:14:37 2018-11-13 16:36:36 0:21:59 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/filestore-xfs.yaml rados.yaml rocksdb.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/cache-agent-big.yaml} 2
dead 3251373 2018-11-13 15:07:15 2018-11-13 16:16:35 2018-11-13 16:36:34 0:19:59 0:11:00 0:08:59 smithi master rhel 7.5 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/simple.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml supported-random-distro$/{rhel_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-agent-small.yaml} 2
Failure Reason:

SSH connection to smithi148 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op copy_from 50 --op write_excl 50 --op delete 50 --pool base'

dead 3251374 2018-11-13 15:07:16 2018-11-13 16:18:25 2018-11-13 16:36:24 0:17:59 smithi master ubuntu 18.04 rados/singleton/{all/mon-seesaw.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
dead 3251375 2018-11-13 15:07:17 2018-11-13 16:18:25 2018-11-13 16:36:24 0:17:59 smithi master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} mon_kv_backend/leveldb.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_striper.yaml} 2
dead 3251376 2018-11-13 15:07:17 2018-11-13 16:18:35 2018-11-13 16:36:34 0:17:59 smithi master centos 7.5 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml} 2
dead 3251377 2018-11-13 15:07:18 2018-11-13 16:20:28 2018-11-13 16:36:27 0:15:59 smithi master ubuntu 16.04 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/jewel.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml two-plus-three.yaml} d-balancer/crush-compat.yaml distro$/{ubuntu_16.04.yaml} msgr-failures/osd-delay.yaml msgr/random.yaml rados.yaml rocksdb.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/test_rbd_api.yaml} 3
dead 3251378 2018-11-13 15:07:19 2018-11-13 16:20:40 2018-11-13 16:34:40 0:14:00 smithi master ubuntu 18.04 rados/singleton-nomsgr/{all/large-omap-object-warnings.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
dead 3251379 2018-11-13 15:07:19 2018-11-13 16:26:30 2018-11-13 16:36:29 0:09:59 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/osd-delay.yaml msgr/random.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml}
dead 3251380 2018-11-13 15:07:20 2018-11-13 16:26:30 2018-11-13 16:36:29 0:09:59 smithi master centos 7.5 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/fastclose.yaml msgr/simple.yaml objectstore/filestore-xfs.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/cache-snaps.yaml}
dead 3251381 2018-11-13 15:07:21 2018-11-13 16:28:26 2018-11-13 16:36:25 0:07:59 smithi master rhel 7.5 rados/perf/{ceph.yaml objectstore/bluestore-bitmap.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_latest.yaml} workloads/cosbench_64K_write.yaml} 1
dead 3251382 2018-11-13 15:07:22 2018-11-13 16:30:42 2018-11-13 16:34:41 0:03:59 smithi master ubuntu 18.04 rados/objectstore/{backends/fusestore.yaml supported-random-distro$/{ubuntu_latest.yaml}}
dead 3251383 2018-11-13 15:07:23 2018-11-13 16:30:42 2018-11-13 16:34:41 0:03:59 smithi master ubuntu 16.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml rocksdb.yaml supported-random-distro$/{ubuntu_16.04.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/cache.yaml}
dead 3251384 2018-11-13 15:07:23 2018-11-13 16:32:37 2018-11-13 16:36:36 0:03:59 smithi master ubuntu 16.04 rados/monthrash/{ceph.yaml clusters/9-mons.yaml mon_kv_backend/leveldb.yaml msgr-failures/few.yaml msgr/simple.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_16.04.yaml} thrashers/many.yaml workloads/rados_mon_workunits.yaml}
dead 3251647 2018-11-13 16:34:43 2018-11-13 16:34:43 2018-11-14 04:37:11 12:02:28 11:48:12 0:14:16 smithi master centos 7.5 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml supported-random-distro$/{centos_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/pool-snaps-few-objects.yaml} 2
Failure Reason:

SSH connection to smithi059 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op snap_remove 50 --op snap_create 50 --op rollback 50 --op read 100 --op copy_from 50 --op write 50 --op write_excl 50 --op delete 50 --pool unique_pool_0'