Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 4230776 2019-08-19 18:21:35 2019-08-19 18:22:39 2019-08-19 18:50:38 0:27:59 0:21:56 0:06:03 smithi master ubuntu 18.04 rados/singleton-nomsgr/{all/full-tiering.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

Exiting scrub checking -- not all pgs scrubbed.

pass 4230777 2019-08-19 18:21:36 2019-08-19 18:22:40 2019-08-19 19:10:39 0:47:59 0:35:22 0:12:37 smithi master centos 7.6 rados/monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/one.yaml workloads/rados_mon_workunits.yaml} 2
pass 4230778 2019-08-19 18:21:37 2019-08-19 18:22:47 2019-08-19 19:16:47 0:54:00 0:16:46 0:37:14 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/mimic-v1only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/rbd_cls.yaml} 4
dead 4230779 2019-08-19 18:21:38 2019-08-19 18:22:48 2019-08-20 06:25:14 12:02:26 smithi master rhel 7.6 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/osd-delay.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-small-objects.yaml} 2
pass 4230780 2019-08-19 18:21:39 2019-08-19 18:22:51 2019-08-19 21:12:53 2:50:02 2:41:24 0:08:38 smithi master rhel 7.6 rados/objectstore/{backends/filestore-idempotent-aio-journal.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4230781 2019-08-19 18:21:40 2019-08-19 18:25:01 2019-08-19 18:55:00 0:29:59 0:24:51 0:05:08 smithi master centos 7.6 rados/standalone/{supported-random-distro$/{centos_7.yaml} workloads/misc.yaml} 1
dead 4230782 2019-08-19 18:21:41 2019-08-19 18:25:01 2019-08-20 06:27:26 12:02:25 10:41:10 1:21:15 smithi master ubuntu 18.04 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
Failure Reason:

SSH connection to smithi012 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op snap_remove 50 --op snap_create 50 --op rollback 50 --op setattr 25 --op read 100 --op copy_from 50 --op write 0 --op rmattr 25 --op append 100 --op delete 50 --pool unique_pool_0'

dead 4230783 2019-08-19 18:21:42 2019-08-19 18:26:56 2019-08-20 06:29:19 12:02:23 smithi master rhel 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml} 2
pass 4230784 2019-08-19 18:21:43 2019-08-19 18:26:56 2019-08-19 18:42:55 0:15:59 0:09:05 0:06:54 smithi master centos 7.6 rados/singleton/{all/admin-socket.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
fail 4230785 2019-08-19 18:21:44 2019-08-19 18:26:56 2019-08-19 21:44:58 3:18:02 3:10:21 0:07:41 smithi master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_stress_watch.yaml} 2
Failure Reason:

Command failed (workunit test rados/stress_watch.sh) on smithi140 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/stress_watch.sh'

fail 4230786 2019-08-19 18:21:45 2019-08-19 18:29:04 2019-08-19 18:49:03 0:19:59 0:11:15 0:08:44 smithi master centos 7.6 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-low-osd-mem-target.yaml supported-random-distro$/{centos_7.yaml} tasks/progress.yaml} 2
Failure Reason:

Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress)

pass 4230787 2019-08-19 18:21:46 2019-08-19 18:29:04 2019-08-19 18:49:03 0:19:59 0:14:44 0:05:15 smithi master centos 7.6 rados/perf/{ceph.yaml objectstore/bluestore-bitmap.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{centos_7.yaml} workloads/radosbench_4M_rand_read.yaml} 1
pass 4230788 2019-08-19 18:21:47 2019-08-19 18:29:04 2019-08-19 19:01:04 0:32:00 0:18:54 0:13:06 smithi master rhel 7.6 rados/multimon/{clusters/6.yaml msgr-failures/many.yaml msgr/async.yaml no_pools.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/mon_recovery.yaml} 2
fail 4230789 2019-08-19 18:21:48 2019-08-19 18:29:05 2019-08-19 22:05:08 3:36:03 3:24:04 0:11:59 smithi master rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/default/{default.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml tasks/rados_cls_all.yaml validater/lockdep.yaml} 2
Failure Reason:

Command failed (workunit test cls/test_cls_journal.sh) on smithi088 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_journal.sh'

fail 4230790 2019-08-19 18:21:49 2019-08-19 18:29:07 2019-08-19 19:17:07 0:48:00 0:37:23 0:10:37 smithi master rhel 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/cache-snaps.yaml} 2
Failure Reason:

"2019-08-19T18:56:15.874421+0000 mon.a (mon.0) 776 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

pass 4230791 2019-08-19 18:21:50 2019-08-19 18:32:56 2019-08-19 18:56:55 0:23:59 0:14:52 0:09:07 smithi master rhel 7.6 rados/singleton/{all/deduptool.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4230792 2019-08-19 18:21:51 2019-08-19 18:32:56 2019-08-19 18:58:55 0:25:59 0:15:29 0:10:30 smithi master rhel 7.6 rados/singleton-nomsgr/{all/health-warnings.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
dead 4230793 2019-08-19 18:21:52 2019-08-19 18:32:56 2019-08-20 06:35:20 12:02:24 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/cache.yaml} 2
pass 4230794 2019-08-19 18:21:53 2019-08-19 18:32:56 2019-08-19 18:54:56 0:22:00 0:11:03 0:10:57 smithi master centos 7.6 rados/singleton/{all/divergent_priors.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
pass 4230795 2019-08-19 18:21:54 2019-08-19 18:32:56 2019-08-19 19:02:56 0:30:00 0:18:47 0:11:13 smithi master rhel 7.6 rados/perf/{ceph.yaml objectstore/bluestore-comp.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/radosbench_4M_seq_read.yaml} 1
dead 4230796 2019-08-19 18:21:55 2019-08-19 18:36:51 2019-08-20 06:39:19 12:02:28 11:17:05 0:45:23 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/mimic.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/few.yaml rados.yaml thrashers/careful.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml}
Failure Reason:

psutil.NoSuchProcess process no longer exists (pid=16210)

dead 4230797 2019-08-19 18:21:56 2019-08-19 18:38:28 2019-08-20 06:40:53 12:02:25 smithi master rhel 7.6 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
dead 4230798 2019-08-19 18:21:57 2019-08-19 18:38:39 2019-08-20 06:41:08 12:02:29 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/dedup_tier.yaml} 2
pass 4230799 2019-08-19 18:21:58 2019-08-19 18:38:47 2019-08-19 18:54:46 0:15:59 0:09:19 0:06:40 smithi master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_striper.yaml} 2
dead 4230800 2019-08-19 18:21:59 2019-08-19 18:40:41 2019-08-20 06:43:06 12:02:25 smithi master rhel 7.6 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=4-m=2.yaml} 3
dead 4230801 2019-08-19 18:22:00 2019-08-19 18:41:10 2019-08-20 06:43:40 12:02:30 smithi master centos 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/osd-delay.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{centos_7.yaml} thrashers/minsize_recovery.yaml thrashosds-health.yaml workloads/ec-snaps-few-objects-overwrites.yaml} 2
pass 4230802 2019-08-19 18:22:01 2019-08-19 18:42:56 2019-08-19 18:58:55 0:15:59 0:08:25 0:07:34 smithi master ubuntu 18.04 rados/singleton/{all/divergent_priors2.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
pass 4230803 2019-08-19 18:22:02 2019-08-19 18:42:56 2019-08-19 19:08:55 0:25:59 0:16:04 0:09:55 smithi master rhel 7.6 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-stupid.yaml supported-random-distro$/{rhel_7.yaml} tasks/prometheus.yaml} 2
fail 4230804 2019-08-19 18:22:03 2019-08-19 18:42:56 2019-08-19 19:02:56 0:20:00 0:14:58 0:05:02 smithi master rhel 7.6 rados/singleton-nomsgr/{all/large-omap-object-warnings.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

"2019-08-19T19:00:55.177775+0000 mon.a (mon.0) 82 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log

dead 4230805 2019-08-19 18:22:04 2019-08-19 18:43:15 2019-08-20 06:45:43 12:02:28 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/pool-snaps-few-objects.yaml} 2
pass 4230806 2019-08-19 18:22:05 2019-08-19 18:43:36 2019-08-19 21:23:38 2:40:02 2:32:29 0:07:33 smithi master rhel 7.6 rados/objectstore/{backends/filestore-idempotent.yaml supported-random-distro$/{rhel_7.yaml}} 1
dead 4230807 2019-08-19 18:22:06 2019-08-19 18:45:05 2019-08-20 06:47:31 12:02:26 smithi master ubuntu 18.04 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/fastclose.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=clay-k=4-m=2.yaml} 2
pass 4230808 2019-08-19 18:22:07 2019-08-19 18:47:10 2019-08-19 19:05:09 0:17:59 0:11:41 0:06:18 smithi master centos 7.6 rados/singleton/{all/dump-stuck.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
pass 4230809 2019-08-19 18:22:08 2019-08-19 18:47:10 2019-08-19 19:07:09 0:19:59 0:13:20 0:06:39 smithi master centos 7.6 rados/perf/{ceph.yaml objectstore/bluestore-low-osd-mem-target.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{centos_7.yaml} workloads/radosbench_4M_write.yaml} 1
pass 4230810 2019-08-19 18:22:09 2019-08-19 18:49:05 2019-08-19 19:23:04 0:33:59 0:25:14 0:08:45 smithi master ubuntu 18.04 rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/sync-many.yaml workloads/snaps-few-objects.yaml} 2
fail 4230811 2019-08-19 18:22:10 2019-08-19 18:49:05 2019-08-19 22:25:08 3:36:03 3:26:58 0:09:05 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on smithi191 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

dead 4230812 2019-08-19 18:22:11 2019-08-19 18:50:53 2019-08-20 06:53:18 12:02:25 smithi master ubuntu 18.04 rados/singleton/{all/ec-lost-unfound.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
pass 4230813 2019-08-19 18:22:12 2019-08-19 18:51:01 2019-08-19 19:19:00 0:27:59 0:15:58 0:12:01 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/nautilus-v1only.yaml backoff/normal.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/default.yaml thrashosds-health.yaml workloads/test_rbd_api.yaml} 4
pass 4230814 2019-08-19 18:22:13 2019-08-19 18:52:41 2019-08-19 19:36:41 0:44:00 0:36:07 0:07:53 smithi master rhel 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/rados_workunit_loadgen_big.yaml} 2
fail 4230815 2019-08-19 18:22:14 2019-08-19 18:52:56 2019-08-19 20:14:56 1:22:00 1:11:47 0:10:13 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/radosbench.yaml} 2
Failure Reason:

reached maximum tries (500) after waiting for 3000 seconds

fail 4230816 2019-08-19 18:22:15 2019-08-19 18:53:27 2019-08-19 19:25:27 0:32:00 0:25:06 0:06:54 smithi master rhel 7.6 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

Command failed (workunit test cephtool/test.sh) on smithi092 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'

pass 4230817 2019-08-19 18:22:16 2019-08-19 18:53:31 2019-08-19 19:07:30 0:13:59 0:07:18 0:06:41 smithi master ubuntu 18.04 rados/singleton-nomsgr/{all/lazy_omap_stats_output.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
fail 4230818 2019-08-19 18:22:18 2019-08-19 18:55:01 2019-08-19 19:15:00 0:19:59 0:13:50 0:06:09 smithi master rhel 7.6 rados/standalone/{supported-random-distro$/{rhel_7.yaml} workloads/mon.yaml} 1
Failure Reason:

Command failed (workunit test mon/health-mute.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/health-mute.sh'

dead 4230819 2019-08-19 18:22:19 2019-08-19 18:55:01 2019-08-20 06:57:14 12:02:13 smithi master ubuntu 18.04 rados/upgrade/nautilus-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/nautilus.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-octopus.yaml 8-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashosds-health.yaml}
fail 4230820 2019-08-19 18:22:20 2019-08-19 18:55:01 2019-08-20 06:33:13 11:38:12 smithi master rhel 7.6 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
Failure Reason:

Command failed on smithi012 with status 1: 'sudo package-cleanup -y --oldkernels'

pass 4230821 2019-08-19 18:22:21 2019-08-19 18:55:01 2019-08-19 19:11:01 0:16:00 0:07:23 0:08:37 smithi master ubuntu 18.04 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/filestore-xfs.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/ssh_orchestrator.yaml} 2
pass 4230822 2019-08-19 18:22:22 2019-08-19 18:55:09 2019-08-19 19:27:08 0:31:59 0:26:20 0:05:39 smithi master rhel 7.6 rados/perf/{ceph.yaml objectstore/bluestore-stupid.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/radosbench_omap_write.yaml} 1
pass 4230823 2019-08-19 18:22:23 2019-08-19 18:57:09 2019-08-19 19:15:08 0:17:59 0:13:03 0:04:56 smithi master rhel 7.6 rados/singleton/{all/erasure-code-nonregression.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4230824 2019-08-19 18:22:24 2019-08-19 18:57:10 2019-08-19 19:17:09 0:19:59 0:07:06 0:12:53 smithi master ubuntu 18.04 rados/multimon/{clusters/9.yaml msgr-failures/few.yaml msgr/async-v1only.yaml no_pools.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/mon_clock_no_skews.yaml} 3
pass 4230825 2019-08-19 18:22:25 2019-08-19 18:57:10 2019-08-19 19:33:09 0:35:59 0:27:44 0:08:15 smithi master centos rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/none.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-stupid.yaml rados.yaml tasks/mon_recovery.yaml validater/valgrind.yaml} 2
fail 4230826 2019-08-19 18:22:26 2019-08-19 18:58:56 2019-08-19 19:38:56 0:40:00 0:31:28 0:08:32 smithi master rhel 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/redirect.yaml} 2
Failure Reason:

"2019-08-19T19:24:17.893799+0000 mon.a (mon.0) 339 : cluster [WRN] Health check failed: 2 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4230827 2019-08-19 18:22:27 2019-08-19 18:58:57 2019-08-19 19:34:56 0:35:59 0:28:22 0:07:37 smithi master centos 7.6 rados/singleton/{all/lost-unfound-delete.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

timed out waiting for admin_socket to appear after osd.1 restart

dead 4230828 2019-08-19 18:22:28 2019-08-19 18:58:59 2019-08-20 07:01:27 12:02:28 smithi master ubuntu 18.04 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
dead 4230829 2019-08-19 18:22:29 2019-08-19 18:59:03 2019-08-20 07:01:30 12:02:27 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/redirect_promote_tests.yaml} 2
pass 4230830 2019-08-19 18:22:30 2019-08-19 19:01:19 2019-08-19 19:15:18 0:13:59 0:06:09 0:07:50 smithi master ubuntu 18.04 rados/objectstore/{backends/fusestore.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
dead 4230831 2019-08-19 18:22:31 2019-08-19 19:03:10 2019-08-20 07:05:37 12:02:27 smithi master ubuntu 18.04 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml} 3
dead 4230832 2019-08-19 18:22:32 2019-08-19 19:03:10 2019-08-20 07:05:33 12:02:23 smithi master centos 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/fastclose.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{centos_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-pool-snaps-few-objects-overwrites.yaml} 2
pass 4230833 2019-08-19 18:22:33 2019-08-19 19:03:10 2019-08-19 19:19:09 0:15:59 0:09:20 0:06:39 smithi master ubuntu 18.04 rados/perf/{ceph.yaml objectstore/filestore-xfs.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{ubuntu_latest.yaml} workloads/sample_fio.yaml} 1
pass 4230834 2019-08-19 18:22:34 2019-08-19 19:05:05 2019-08-19 19:25:05 0:20:00 0:14:40 0:05:20 smithi master rhel 7.6 rados/singleton-nomsgr/{all/librados_hello_world.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4230835 2019-08-19 18:22:35 2019-08-19 19:05:06 2019-08-19 19:45:05 0:39:59 0:32:09 0:07:50 smithi master rhel 7.6 rados/singleton/{all/lost-unfound.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

timed out waiting for admin_socket to appear after osd.1 restart

pass 4230836 2019-08-19 18:22:36 2019-08-19 19:05:06 2019-08-19 19:33:05 0:27:59 0:18:40 0:09:19 smithi master ubuntu 18.04 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/rados_workunit_loadgen_mix.yaml} 2
dead 4230837 2019-08-19 18:22:37 2019-08-19 19:05:11 2019-08-20 07:09:39 12:04:28 11:43:28 0:21:00 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/nautilus-v2only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/mapgap.yaml thrashosds-health.yaml workloads/cache-snaps.yaml}
Failure Reason:

psutil.NoSuchProcess process no longer exists (pid=25868)

dead 4230838 2019-08-19 18:22:39 2019-08-19 19:06:58 2019-08-20 07:09:26 12:02:28 smithi master centos 7.6 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{centos_7.yaml} thrashers/fastread.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=2-m=1.yaml} 2
dead 4230839 2019-08-19 18:22:40 2019-08-19 19:07:10 2019-08-20 07:09:37 12:02:27 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/redirect_set_object.yaml} 2
pass 4230840 2019-08-19 18:22:41 2019-08-19 19:07:32 2019-08-19 19:35:31 0:27:59 0:12:50 0:15:09 smithi master centos 7.6 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-bitmap.yaml supported-random-distro$/{centos_7.yaml} tasks/workunits.yaml} 2
pass 4230841 2019-08-19 18:22:42 2019-08-19 19:09:10 2019-08-19 19:29:09 0:19:59 0:13:40 0:06:19 smithi master rhel 7.6 rados/singleton/{all/max-pg-per-osd.from-mon.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4230842 2019-08-19 18:22:43 2019-08-19 19:10:54 2019-08-19 20:18:54 1:08:00 0:57:24 0:10:36 smithi master rhel 7.6 rados/monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/sync.yaml workloads/pool-create-delete.yaml} 2
dead 4230843 2019-08-19 18:22:44 2019-08-19 19:11:02 2019-08-20 07:13:29 12:02:27 11:54:46 0:07:41 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/set-chunk-promote-flush.yaml}
Failure Reason:

psutil.NoSuchProcess process no longer exists (pid=5452)

pass 4230844 2019-08-19 18:22:45 2019-08-19 19:13:10 2019-08-19 19:37:09 0:23:59 0:17:07 0:06:52 smithi master rhel 7.6 rados/perf/{ceph.yaml objectstore/bluestore-bitmap.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/sample_radosbench.yaml} 1
pass 4230845 2019-08-19 18:22:46 2019-08-19 19:15:15 2019-08-19 19:45:15 0:30:00 0:23:00 0:07:00 smithi master centos 7.6 rados/singleton/{all/max-pg-per-osd.from-primary.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
pass 4230846 2019-08-19 18:22:47 2019-08-19 19:15:16 2019-08-19 19:45:15 0:29:59 0:24:03 0:05:56 smithi master centos 7.6 rados/singleton-nomsgr/{all/msgr.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
dead 4230847 2019-08-19 18:22:48 2019-08-19 19:15:16 2019-08-20 07:17:43 12:02:27 smithi master rhel 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/set-chunks-read.yaml} 2
pass 4230848 2019-08-19 18:22:49 2019-08-19 19:15:20 2019-08-19 19:53:19 0:37:59 0:30:58 0:07:01 smithi master rhel 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/rados_workunit_loadgen_mostlyread.yaml} 2
pass 4230849 2019-08-19 18:22:50 2019-08-19 19:17:03 2019-08-19 19:39:02 0:21:59 0:14:55 0:07:04 smithi master centos 7.6 rados/singleton/{all/max-pg-per-osd.from-replica.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
fail 4230850 2019-08-19 18:22:51 2019-08-19 19:17:03 2019-08-19 21:01:03 1:44:00 1:20:21 0:23:39 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/nautilus.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/few.yaml rados.yaml thrashers/morepggrow.yaml thrashosds-health.yaml workloads/radosbench.yaml} 4
Failure Reason:

reached maximum tries (500) after waiting for 3000 seconds

dead 4230851 2019-08-19 18:22:52 2019-08-19 19:17:08 2019-08-20 07:19:35 12:02:27 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/small-objects.yaml} 2
pass 4230852 2019-08-19 18:22:53 2019-08-19 19:17:08 2019-08-19 19:37:08 0:20:00 0:13:15 0:06:45 smithi master rhel 7.6 rados/objectstore/{backends/keyvaluedb.yaml supported-random-distro$/{rhel_7.yaml}} 1
dead 4230853 2019-08-19 18:22:54 2019-08-19 19:17:10 2019-08-20 07:19:37 12:02:27 11:55:35 0:06:52 smithi master rhel 7.6 rados/standalone/{supported-random-distro$/{rhel_7.yaml} workloads/osd.yaml} 1
Failure Reason:

psutil.NoSuchProcess process no longer exists (pid=25206)

pass 4230854 2019-08-19 18:22:55 2019-08-19 19:17:33 2019-08-19 19:37:32 0:19:59 0:13:13 0:06:46 smithi master centos 7.6 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-bitmap.yaml supported-random-distro$/{centos_7.yaml} tasks/crash.yaml} 2
pass 4230855 2019-08-19 18:22:56 2019-08-19 19:18:58 2019-08-19 19:40:57 0:21:59 0:16:29 0:05:30 smithi master ubuntu 18.04 rados/perf/{ceph.yaml objectstore/bluestore-bitmap.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{ubuntu_latest.yaml} workloads/cosbench_64K_read_write.yaml} 1
dead 4230856 2019-08-19 18:22:57 2019-08-19 19:19:02 2019-08-20 07:21:28 12:02:26 smithi master rhel 7.6 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
dead 4230857 2019-08-19 18:22:58 2019-08-19 19:19:10 2019-08-20 07:21:38 12:02:28 smithi master ubuntu 18.04 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
pass 4230858 2019-08-19 18:22:59 2019-08-19 19:21:14 2019-08-19 19:39:13 0:17:59 0:11:24 0:06:35 smithi master centos 7.6 rados/singleton/{all/mon-auth-caps.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
pass 4230859 2019-08-19 18:23:00 2019-08-19 19:23:02 2019-08-19 19:39:02 0:16:00 0:06:55 0:09:05 smithi master ubuntu 18.04 rados/multimon/{clusters/21.yaml msgr-failures/many.yaml msgr/async-v2only.yaml no_pools.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/mon_clock_with_skews.yaml} 3
fail 4230860 2019-08-19 18:23:01 2019-08-19 19:23:03 2019-08-20 01:59:09 6:36:06 6:27:56 0:08:10 smithi master rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/default/{default.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml tasks/rados_api_tests.yaml validater/lockdep.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on smithi037 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4230861 2019-08-19 18:23:02 2019-08-19 19:23:06 2019-08-19 20:01:05 0:37:59 0:29:17 0:08:42 smithi master ubuntu 18.04 rados/singleton-nomsgr/{all/multi-backfill-reject.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 2
Failure Reason:

"2019-08-19T19:43:17.056648+0000 mon.a (mon.0) 244 : cluster [WRN] Health check failed: 2 daemons have recently crashed (RECENT_CRASH)" in cluster log

pass 4230862 2019-08-19 18:23:03 2019-08-19 19:25:21 2019-08-19 20:01:20 0:35:59 0:29:57 0:06:02 smithi master rhel 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} 2
dead 4230863 2019-08-19 18:23:04 2019-08-19 19:25:28 2019-08-20 07:27:56 12:02:28 smithi master ubuntu 18.04 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=4-m=2.yaml} 3
dead 4230864 2019-08-19 18:23:05 2019-08-19 19:26:43 2019-08-20 07:29:11 12:02:28 smithi master ubuntu 18.04 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/few.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-small-objects-fast-read-overwrites.yaml} 2
dead 4230865 2019-08-19 18:23:06 2019-08-19 19:27:10 2019-08-20 07:29:37 12:02:27 smithi master ubuntu 18.04 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/osd-delay.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/minsize_recovery.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=3-m=1.yaml} 2
pass 4230866 2019-08-19 18:23:07 2019-08-19 19:29:26 2019-08-19 19:49:25 0:19:59 0:12:33 0:07:26 smithi master centos 7.6 rados/singleton/{all/mon-config-key-caps.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
dead 4230867 2019-08-19 18:23:08 2019-08-19 19:31:10 2019-08-20 07:33:33 12:02:23 smithi master rhel 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/write_fadvise_dontneed.yaml} 2
pass 4230868 2019-08-19 18:23:09 2019-08-19 19:32:49 2019-08-19 19:56:48 0:23:59 0:18:08 0:05:51 smithi master rhel 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/readwrite.yaml} 2
fail 4230869 2019-08-19 18:23:10 2019-08-19 19:33:07 2019-08-19 20:05:06 0:31:59 0:25:16 0:06:43 smithi master rhel 7.6 rados/perf/{ceph.yaml objectstore/bluestore-comp.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/cosbench_64K_write.yaml} 1
Failure Reason:

Command failed on smithi124 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'

pass 4230870 2019-08-19 18:23:11 2019-08-19 19:33:11 2019-08-19 19:57:10 0:23:59 0:16:15 0:07:44 smithi master centos 7.6 rados/singleton/{all/mon-config-keys.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
pass 4230871 2019-08-19 18:23:12 2019-08-19 19:34:38 2019-08-19 19:54:37 0:19:59 0:14:28 0:05:31 smithi master rhel 7.6 rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/force-sync-many.yaml workloads/rados_5925.yaml} 2
fail 4230872 2019-08-19 18:23:13 2019-08-19 19:34:58 2019-08-19 21:28:59 1:54:01 1:46:08 0:07:53 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/admin_socket_objecter_requests.yaml} 2
Failure Reason:

reached maximum tries (800) after waiting for 4800 seconds

dead 4230873 2019-08-19 18:23:14 2019-08-19 19:35:33 2019-08-20 07:38:01 12:02:28 11:43:59 0:18:29 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/hammer.yaml backoff/normal.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/none.yaml thrashosds-health.yaml workloads/rbd_cls.yaml} 4
Failure Reason:

psutil.NoSuchProcess process no longer exists (pid=4904)

fail 4230874 2019-08-19 18:23:15 2019-08-19 19:36:58 2019-08-19 20:08:58 0:32:00 0:23:11 0:08:49 smithi master ubuntu 18.04 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-comp.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/dashboard.yaml} 2
Failure Reason:

Test failure: test_create_get_update_delete_w_tenant (tasks.mgr.dashboard.test_rgw.RgwBucketTest)

fail 4230875 2019-08-19 18:23:16 2019-08-19 19:37:09 2019-08-19 20:07:08 0:29:59 0:21:42 0:08:17 smithi master centos 7.6 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

Command failed (workunit test cephtool/test.sh) on smithi068 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'

fail 4230876 2019-08-19 18:23:17 2019-08-19 19:37:10 2019-08-19 19:49:10 0:12:00 0:05:28 0:06:32 smithi master ubuntu 18.04 rados/singleton-nomsgr/{all/osd_stale_reads.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

'NoneType' object is not iterable

pass 4230877 2019-08-19 18:23:18 2019-08-19 19:37:34 2019-08-19 19:57:33 0:19:59 0:14:18 0:05:41 smithi master rhel 7.6 rados/singleton/{all/mon-config.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
dead 4230878 2019-08-19 18:23:19 2019-08-19 19:39:12 2019-08-20 07:41:39 12:02:27 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/cache-agent-big.yaml} 2
pass 4230879 2019-08-19 18:23:20 2019-08-19 19:39:12 2019-08-19 19:57:11 0:17:59 0:12:21 0:05:38 smithi master centos 7.6 rados/perf/{ceph.yaml objectstore/bluestore-low-osd-mem-target.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{centos_7.yaml} workloads/fio_4K_rand_read.yaml} 1
pass 4230880 2019-08-19 18:23:21 2019-08-19 19:39:12 2019-08-19 20:15:12 0:36:00 0:27:41 0:08:19 smithi master centos 7.6 rados/objectstore/{backends/objectcacher-stress.yaml supported-random-distro$/{centos_7.yaml}} 1
dead 4230881 2019-08-19 18:23:22 2019-08-19 19:39:15 2019-08-20 07:41:42 12:02:27 smithi master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/repair_test.yaml} 2
dead 4230882 2019-08-19 18:23:23 2019-08-19 19:40:59 2019-08-20 07:43:26 12:02:27 smithi master rhel 7.6 rados/singleton/{all/osd-backfill.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4230883 2019-08-19 18:23:24 2019-08-19 19:45:16 2019-08-19 20:23:15 0:37:59 0:27:40 0:10:19 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/cache-agent-small.yaml} 2
Failure Reason:

timed out waiting for admin_socket to appear after osd.1 restart

dead 4230884 2019-08-19 18:23:25 2019-08-19 19:45:16 2019-08-20 07:47:39 12:02:23 smithi master centos 7.6 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
pass 4230885 2019-08-19 18:23:26 2019-08-19 19:45:16 2019-08-19 20:01:15 0:15:59 0:07:20 0:08:39 smithi master ubuntu 18.04 rados/singleton-nomsgr/{all/pool-access.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
fail 4230886 2019-08-19 18:23:27 2019-08-19 19:45:17 2019-08-19 20:13:16 0:27:59 0:16:03 0:11:56 smithi master rhel 7.6 rados/singleton/{all/osd-recovery-incomplete.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

Command failed on smithi065 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats'

dead 4230887 2019-08-19 18:23:28 2019-08-19 19:47:07 2019-08-20 07:49:35 12:02:28 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/cache-pool-snaps-readproxy.yaml} 2
fail 4230888 2019-08-19 18:23:29 2019-08-19 19:49:09 2019-08-19 21:53:10 2:04:01 1:49:55 0:14:06 smithi master rhel 7.6 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/fastclose.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/ec-radosbench.yaml} 2
Failure Reason:

reached maximum tries (800) after waiting for 4800 seconds

dead 4230889 2019-08-19 18:23:30 2019-08-19 19:49:11 2019-08-20 07:51:34 12:02:23 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/jewel-v1only.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/pggrow.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml} 4
pass 4230890 2019-08-19 18:23:31 2019-08-19 19:49:26 2019-08-19 20:13:26 0:24:00 0:13:41 0:10:19 smithi master centos 7.6 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-low-osd-mem-target.yaml supported-random-distro$/{centos_7.yaml} tasks/failover.yaml} 2
pass 4230891 2019-08-19 18:23:32 2019-08-19 19:53:06 2019-08-19 20:09:05 0:15:59 0:09:15 0:06:44 smithi master ubuntu 18.04 rados/perf/{ceph.yaml objectstore/bluestore-stupid.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{ubuntu_latest.yaml} workloads/fio_4K_rand_rw.yaml} 1
fail 4230892 2019-08-19 18:23:33 2019-08-19 19:53:21 2019-08-19 23:09:23 3:16:02 3:07:59 0:08:03 smithi master ubuntu 18.04 rados/standalone/{supported-random-distro$/{ubuntu_latest.yaml} workloads/scrub.yaml} 1
Failure Reason:

Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi162 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh'

dead 4230893 2019-08-19 18:23:34 2019-08-19 19:54:52 2019-08-20 07:57:19 12:02:27 smithi master ubuntu 18.04 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/few.yaml objectstore/filestore-xfs.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml} 3
dead 4230894 2019-08-19 18:23:35 2019-08-19 19:55:41 2019-08-20 07:58:09 12:02:28 smithi master ubuntu 18.04 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/osd-delay.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-small-objects-overwrites.yaml} 2
dead 4230895 2019-08-19 18:23:36 2019-08-19 19:57:04 2019-08-20 07:59:17 12:02:13 smithi master rhel 7.6 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml}
fail 4230896 2019-08-19 18:23:37 2019-08-19 19:57:11 2019-08-19 20:45:11 0:48:00 0:38:20 0:09:40 smithi master rhel 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/cache-pool-snaps.yaml} 2
Failure Reason:

"2019-08-19T20:24:23.154627+0000 mon.a (mon.0) 738 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

pass 4230897 2019-08-19 18:23:38 2019-08-19 19:57:13 2019-08-19 20:15:12 0:17:59 0:10:12 0:07:47 smithi master ubuntu 18.04 rados/singleton/{all/osd-recovery.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
pass 4230898 2019-08-19 18:23:39 2019-08-19 19:57:34 2019-08-19 20:21:34 0:24:00 0:16:15 0:07:45 smithi master centos 7.6 rados/multimon/{clusters/3.yaml msgr-failures/few.yaml msgr/async.yaml no_pools.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/mon_recovery.yaml} 2
pass 4230899 2019-08-19 18:23:40 2019-08-19 20:01:21 2019-08-19 21:09:21 1:08:00 1:00:37 0:07:23 smithi master centos rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/none.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml tasks/rados_cls_all.yaml validater/valgrind.yaml} 2
pass 4230900 2019-08-19 18:23:41 2019-08-19 20:01:21 2019-08-19 20:37:21 0:36:00 0:24:35 0:11:25 smithi master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/rgw_snaps.yaml} 2
pass 4230901 2019-08-19 18:23:42 2019-08-19 20:01:21 2019-08-19 20:41:21 0:40:00 0:35:00 0:05:00 smithi master rhel 7.6 rados/singleton-nomsgr/{all/recovery-unfound-found.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
dead 4230902 2019-08-19 18:23:43 2019-08-19 20:02:57 2019-08-20 08:05:21 12:02:24 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/cache-snaps.yaml} 2
pass 4230903 2019-08-19 18:23:44 2019-08-19 20:05:21 2019-08-19 20:19:20 0:13:59 0:06:41 0:07:18 smithi master ubuntu 18.04 rados/singleton/{all/peer.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
pass 4230904 2019-08-19 18:23:45 2019-08-19 20:07:08 2019-08-19 20:23:07 0:15:59 0:09:07 0:06:52 smithi master ubuntu 18.04 rados/perf/{ceph.yaml objectstore/filestore-xfs.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{ubuntu_latest.yaml} workloads/fio_4M_rand_read.yaml} 1
fail 4230905 2019-08-19 18:23:46 2019-08-19 20:07:08 2019-08-19 23:35:10 3:28:02 3:19:43 0:08:19 smithi master centos 7.6 rados/monthrash/{ceph.yaml clusters/9-mons.yaml msgr-failures/mon-delay.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/many.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on smithi114 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

dead 4230906 2019-08-19 18:23:47 2019-08-19 20:07:10 2019-08-20 08:09:33 12:02:23 smithi master rhel 7.6 rados/objectstore/{backends/objectstore.yaml supported-random-distro$/{rhel_7.yaml}} 1
dead 4230907 2019-08-19 18:23:48 2019-08-19 20:07:38 2019-08-20 08:10:01 12:02:23 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/cache.yaml} 2
fail 4230908 2019-08-19 18:23:49 2019-08-19 20:09:13 2019-08-19 20:29:12 0:19:59 0:09:54 0:10:05 smithi master ubuntu 18.04 rados/singleton/{all/pg-autoscaler.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 2
Failure Reason:

Found coredumps on ubuntu@smithi060.front.sepia.ceph.com

pass 4230909 2019-08-19 18:23:50 2019-08-19 20:09:13 2019-08-19 20:29:12 0:19:59 0:09:43 0:10:16 smithi master ubuntu 18.04 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-stupid.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/insights.yaml} 2
pass 4230910 2019-08-19 18:23:51 2019-08-19 20:11:17 2019-08-19 21:23:17 1:12:00 0:16:12 0:55:48 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/jewel.yaml backoff/peering_and_degraded.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/few.yaml rados.yaml thrashers/careful.yaml thrashosds-health.yaml workloads/test_rbd_api.yaml} 4
dead 4230911 2019-08-19 18:23:52 2019-08-19 20:11:17 2019-08-20 08:13:45 12:02:28 smithi master rhel 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/dedup_tier.yaml} 2
pass 4230912 2019-08-19 18:23:52 2019-08-19 20:13:13 2019-08-19 20:35:12 0:21:59 0:16:24 0:05:35 smithi master rhel 7.6 rados/perf/{ceph.yaml objectstore/bluestore-bitmap.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/fio_4M_rand_rw.yaml} 1
pass 4230913 2019-08-19 18:23:53 2019-08-19 20:13:13 2019-08-19 20:33:12 0:19:59 0:13:49 0:06:10 smithi master centos 7.6 rados/singleton/{all/pg-removal-interruption.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
pass 4230914 2019-08-19 18:23:54 2019-08-19 20:13:17 2019-08-19 20:33:16 0:19:59 0:14:07 0:05:52 smithi master centos 7.6 rados/singleton-nomsgr/{all/version-number-sanity.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
pass 4230915 2019-08-19 18:23:55 2019-08-19 20:13:27 2019-08-19 20:37:26 0:23:59 0:16:38 0:07:21 smithi master rhel 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/scrub_test.yaml} 2
dead 4230916 2019-08-19 18:23:56 2019-08-19 20:15:11 2019-08-20 08:17:39 12:02:28 smithi master ubuntu 18.04 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
dead 4230917 2019-08-19 18:23:57 2019-08-19 20:15:13 2019-08-20 08:17:41 12:02:28 smithi master ubuntu 18.04 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/few.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-small-objects-fast-read.yaml} 2
fail 4230918 2019-08-19 18:23:58 2019-08-19 20:15:13 2019-08-19 21:39:14 1:24:01 1:12:03 0:11:58 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/pool-snaps-few-objects.yaml} 2
Failure Reason:

failed to become clean before timeout expired

fail 4230919 2019-08-19 18:23:59 2019-08-19 20:17:11 2019-08-19 23:39:13 3:22:02 3:13:37 0:08:25 smithi master centos 7.6 rados/singleton/{all/radostool.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
Failure Reason:

Command failed (workunit test rados/test_rados_tool.sh) on smithi083 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_rados_tool.sh'

dead 4230920 2019-08-19 18:24:00 2019-08-19 20:17:11 2019-08-20 08:19:35 12:02:24 smithi master rhel 7.6 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-bitmap.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-rados-plugin=jerasure-k=4-m=2.yaml} 3
dead 4230921 2019-08-19 18:24:01 2019-08-19 20:19:10 2019-08-20 08:21:37 12:02:27 smithi master rhel 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/normal.yaml msgr-failures/fastclose.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/ec-snaps-few-objects-overwrites.yaml} 2
fail 4230922 2019-08-19 18:24:02 2019-08-19 20:19:10 2019-08-19 23:55:12 3:36:02 3:23:46 0:12:16 smithi master ubuntu 18.04 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/morepggrow.yaml thrashosds-health.yaml workloads/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on smithi190 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

fail 4230923 2019-08-19 18:24:03 2019-08-19 20:19:21 2019-08-19 20:49:21 0:30:00 0:22:06 0:07:54 smithi master ubuntu 18.04 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/filestore-xfs.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/module_selftest.yaml} 2
Failure Reason:

"2019-08-19T20:41:15.642908+0000 mon.a (mon.0) 1192 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 4230924 2019-08-19 18:24:04 2019-08-19 20:19:25 2019-08-19 20:37:24 0:17:59 0:09:17 0:08:42 smithi master ubuntu 18.04 rados/perf/{ceph.yaml objectstore/bluestore-comp.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{ubuntu_latest.yaml} workloads/fio_4M_rand_write.yaml} 1
pass 4230925 2019-08-19 18:24:05 2019-08-19 20:21:21 2019-08-19 20:51:20 0:29:59 0:23:03 0:06:56 smithi master ubuntu 18.04 rados/singleton/{all/random-eio.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 2
dead 4230926 2019-08-19 18:24:06 2019-08-19 20:21:35 2019-08-20 08:23:48 12:02:13 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-install/luminous-v1only.yaml backoff/normal.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/off.yaml distro$/{centos_latest.yaml} msgr-failures/osd-delay.yaml rados.yaml thrashers/default.yaml thrashosds-health.yaml workloads/cache-snaps.yaml}
fail 4230927 2019-08-19 18:24:07 2019-08-19 20:23:22 2019-08-19 23:37:24 3:14:02 3:07:25 0:06:37 smithi master ubuntu 18.04 rados/objectstore/{backends/alloc-hint.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
Failure Reason:

Command failed (workunit test rados/test_alloc_hint.sh) on smithi113 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_alloc_hint.sh'

pass 4230928 2019-08-19 18:24:08 2019-08-19 20:23:22 2019-08-19 20:43:21 0:19:59 0:14:16 0:05:43 smithi master rhel 7.6 rados/rest/{mgr-restful.yaml supported-random-distro$/{rhel_7.yaml}} 1
fail 4230929 2019-08-19 18:24:09 2019-08-19 20:25:11 2019-08-19 20:59:11 0:34:00 0:25:20 0:08:40 smithi master rhel 7.6 rados/singleton-bluestore/{all/cephtool.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

Command failed (workunit test cephtool/test.sh) on smithi065 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'

pass 4230930 2019-08-19 18:24:10 2019-08-19 20:29:27 2019-08-19 20:57:26 0:27:59 0:20:04 0:07:55 smithi master centos rados/singleton-flat/valgrind-leaks.yaml 1
pass 4230931 2019-08-19 18:24:11 2019-08-19 20:29:27 2019-08-19 20:57:26 0:27:59 0:18:39 0:09:20 smithi master rhel 7.6 rados/singleton-nomsgr/{all/admin_socket_output.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4230932 2019-08-19 18:24:12 2019-08-19 20:33:26 2019-08-19 20:51:26 0:18:00 0:10:22 0:07:38 smithi master ubuntu 18.04 rados/standalone/{supported-random-distro$/{ubuntu_latest.yaml} workloads/crush.yaml} 1
fail 4230933 2019-08-19 18:24:13 2019-08-19 20:33:27 2019-08-20 08:31:39 11:58:12 0:41:18 11:16:54 smithi master ubuntu 18.04 rados/upgrade/mimic-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/mimic.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-nautilus.yaml 8-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashosds-health.yaml} 4
Failure Reason:

failed to become clean before timeout expired

dead 4230934 2019-08-19 18:24:14 2019-08-19 20:35:27 2019-08-20 08:37:54 12:02:27 smithi master rhel 7.6 rados/thrash-erasure-code-shec/{ceph.yaml clusters/{fixed-4.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml} 4
fail 4230935 2019-08-19 18:24:15 2019-08-19 20:37:26 2019-08-19 22:05:27 1:28:01 1:15:52 0:12:09 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/none.yaml thrashosds-health.yaml workloads/radosbench.yaml} 2
Failure Reason:

reached maximum tries (500) after waiting for 3000 seconds

fail 4230936 2019-08-19 18:24:16 2019-08-19 20:37:27 2019-08-20 00:05:29 3:28:02 3:18:34 0:09:28 smithi master rhel 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/rados_api_tests.yaml} 2
Failure Reason:

Command failed (workunit test rados/test.sh) on smithi002 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=475286020efbcbe469ec6169e1c477e69c61c6c6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

pass 4230937 2019-08-19 18:24:17 2019-08-19 20:37:28 2019-08-19 21:27:27 0:49:59 0:13:33 0:36:26 smithi master rhel 7.6 rados/multimon/{clusters/6.yaml msgr-failures/many.yaml msgr/async-v1only.yaml no_pools.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} tasks/mon_clock_no_skews.yaml} 2
fail 4230938 2019-08-19 18:24:18 2019-08-19 20:41:15 2019-08-19 21:13:15 0:32:00 0:20:51 0:11:09 smithi master rados/verify/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-thrash/default/{default.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-comp.yaml rados.yaml tasks/mon_recovery.yaml validater/lockdep.yaml} 2
Failure Reason:

"2019-08-19T21:03:53.254475+0000 mon.a (mon.0) 527 : cluster [WRN] Health check failed: 1 daemons have recently crashed (RECENT_CRASH)" in cluster log

fail 4230939 2019-08-19 18:24:19 2019-08-19 20:41:22 2019-08-19 21:05:22 0:24:00 0:15:55 0:08:05 smithi master rhel 7.6 rados/singleton/{all/rebuild-mondb.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
Failure Reason:

+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store *** stack smashing detected ***: ceph-objectstore-tool terminated ======= Backtrace: ========= /lib64/libc.so.6(__fortify_fail+0x37)[0x7fadca990b67] /lib64/libc.so.6(+0x117b22)[0x7fadca990b22] ceph-objectstore-tool(_Z13update_mon_dbR11ObjectStoreR13OSDSuperblockRKSsS4_+0x2c51)[0x5581bb233a91] ceph-objectstore-tool(main+0x557e)[0x5581bb1b805e] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fadca89b495] ceph-objectstore-tool(+0x3cfe50)[0x5581bb1e7e50] ======= Memory map: ======== 5581bae18000-5581bbcf9000 r-xp 00000000 08:01 1728 /usr/bin/ceph-objectstore-tool 5581bbef8000-5581bbf30000 r--p 00ee0000 08:01 1728 /usr/bin/ceph-objectstore-tool 5581bbf30000-5581bbf44000 rw-p 00f18000 08:01 1728 /usr/bin/ceph-objectstore-tool 5581bbf44000-5581bc05c000 rw-p 00000000 00:00 0 5581bd85f000-5581bfdc2000 rw-p 00000000 00:00 0 [heap] 7fadb36ed000-7fadb36ee000 ---p 00000000 00:00 0 7fadb36ee000-7fadb3eee000 rw-p 00000000 00:00 0 7fadb3eee000-7fadb3eef000 ---p 00000000 00:00 0 7fadb3eef000-7fadb46ef000 rw-p 00000000 00:00 0 7fadb46ef000-7fadb46f0000 ---p 00000000 00:00 0 7fadb46f0000-7fadb4ef0000 rw-p 00000000 00:00 0 7fadb4ef0000-7fadb4ef1000 ---p 00000000 00:00 0 7fadb4ef1000-7fadb56f1000 rw-p 00000000 00:00 0 7fadbaefc000-7fadbaefd000 ---p 00000000 00:00 0 7fadbaefd000-7fadbb6fd000 rw-p 00000000 00:00 0 7fadbb6fd000-7fadbb6fe000 ---p 00000000 00:00 0 7fadbb6fe000-7fadbbefe000 rw-p 00000000 00:00 0 7fadbbefe000-7fadbbeff000 ---p 00000000 00:00 0 7fadbbeff000-7fadbc6ff000 rw-p 00000000 00:00 0 7fadbc6ff000-7fadbc700000 ---p 00000000 00:00 0 7fadbc700000-7fadbcf00000 rw-p 00000000 00:00 0 7fadbcf00000-7fadbcf01000 ---p 00000000 00:00 0 7fadbcf01000-7fadbd701000 rw-p 00000000 00:00 0 7fadbd701000-7fadbd702000 ---p 00000000 00:00 0 7fadbd702000-7fadbdf02000 rw-p 00000000 00:00 0 7fadbdf02000-7fadbdf03000 ---p 00000000 00:00 0 7fadbdf03000-7fadbe703000 rw-p 00000000 00:00 0 7fadbe703000-7fadbe704000 ---p 00000000 00:00 0 7fadbe704000-7fadbef04000 rw-p 00000000 00:00 0 7fadbef04000-7fadbef05000 ---p 00000000 00:00 0 7fadbef05000-7fadbf705000 rw-p 00000000 00:00 0 7fadbf705000-7fadbf706000 ---p 00000000 00:00 0 7fadbf706000-7fadbff06000 rw-p 00000000 00:00 0 7fadbff06000-7fadbff07000 ---p 00000000 00:00 0 7fadbff07000-7fadc0707000 rw-p 00000000 00:00 0 7fadc0707000-7fadc0708000 ---p 00000000 00:00 0 7fadc0708000-7fadc0f08000 rw-p 00000000 00:00 0 7fadc0f08000-7fadc0f09000 ---p 00000000 00:00 0 7fadc0f09000-7fadc1709000 rw-p 00000000 00:00 0 7fadc1709000-7fadc170a000 ---p 00000000 00:00 0 7fadc170a000-7fadc1f0a000 rw-p 00000000 00:00 0 7fadc1f0a000-7fadc1f0b000 ---p 00000000 00:00 0 7fadc1f0b000-7fadc270b000 rw-p 00000000 00:00 0 7fadc270b000-7fadc270c000 ---p 00000000 00:00 0 7fadc270c000-7fadc2f0c000 rw-p 00000000 00:00 0 7fadc2f0c000-7fadc2f0d000 ---p 00000000 00:00 0 7fadc2f0d000-7fadc370d000 rw-p 00000000 00:00 0 7fadc370d000-7fadc370e000 ---p 00000000 00:00 0 7fadc370e000-7fadc3f0e000 rw-p 00000000 00:00 0 7fadc3f0e000-7fadc3f0f000 ---p 00000000 00:00 0 7fadc3f0f000-7fadc470f000 rw-p 00000000 00:00 0 7fadc470f000-7fadc4710000 ---p 00000000 00:00 0 7fadc4710000-7fadc4f10000 rw-p 00000000 00:00 0 7fadc4f10000-7fadc4f11000 ---p 00000000 00:00 0 7fadc4f11000-7fadc5711000 rw-p 00000000 00:00 0 7fadc5711000-7fadc5712000 ---p 00000000 00:00 0 7fadc5712000-7fadc5f12000 rw-p 00000000 00:00 0 7fadc5f12000-7fadc5f13000 ---p 00000000 00:00 0 7fadc5f13000-7fadc6713000 rw-p 00000000 00:00 0 7fadc6713000-7fadc6714000 ---p 00000000 00:00 0 7fadc6714000-7fadc6f14000 rw-p 00000000 00:00 0 7fadc6f14000-7fadc6f15000 ---p 00000000 00:00 0 7fadc6f15000-7fadc7715000 rw-p 00000000 00:00 0 7fadc7715000-7fadc7716000 ---p 00000000 00:00 0 7fadc7716000-7fadc7f16000 rw-p 00000000 00:00 0 7fadc7f16000-7fadc7f17000 ---p 00000000 00:00 0 7fadc7f17000-7fadc8717000 rw-p 00000000 00:00 0 7fadc8717000-7fadc8718000 ---p 00000000 00:00 0 7fadc8718000-7fadc8f18000 rw-p 00000000 00:00 0 7fadc8f18000-7fadc8f1a000 r-xp 00000000 08:01 1406 /usr/lib64/liburcu-common.so.1.0.0 7fadc8f1a000-7fadc9119000 ---p 00002000 08:01 1406 /usr/lib64/liburcu-common.so.1.0.0 7fadc9119000-7fadc911a000 r--p 00001000 08:01 1406 /usr/lib64/liburcu-common.so.1.0.0 7fadc911a000-7fadc911b000 rw-p 00002000 08:01 1406 /usr/lib64/liburcu-common.so.1.0.0 7fadc911b000-7fadc9121000 r-xp 00000000 08:01 1404 /usr/lib64/liburcu-cds.so.1.0.0 7fadc9121000-7fadc9320000 ---p 00006000 08:01 1404 /usr/lib64/liburcu-cds.so.1.0.0 7fadc9320000-7fadc9321000 r--p 00005000 08:01 1404 /usr/lib64/liburcu-cds.so.1.0.0 7fadc9321000-7fadc9322000 rw-p 00006000 08:01 1404 /usr/lib64/liburcu-cds.so.1.0.0 7fadc9322000-7fadc9328000 r-xp 00000000 08:01 1402 /usr/lib64/liburcu-bp.so.1.0.0 7fadc9328000-7fadc9528000 ---p 00006000 08:01 1402 /usr/lib64/liburcu-bp.so.1.0.0 7fadc9528000-7fadc9529000 r--p 00006000 08:01 1402 /usr/lib64/liburcu-bp.so.1.0.0 7fadc9529000-7fadc952a000 rw-p 00007000 08:01 1402 /usr/lib64/liburcu-bp.so.1.0.0 7fadc952a000-7fadc9533000 r-xp 00000000 08:01 1430 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 7fadc9533000-7fadc9732000 ---p 00009000 08:01 1430 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 7fadc9732000-7fadc9733000 r--p 00008000 08:01 1430 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 7fadc9733000-7fadc9734000 rw-p 00009000 08:01 1430 /usr/lib64/liblttng-ust-tracepoint.so.0.0.0 7fadc9734000-7fadc9744000 rw-p 00000000 00:00 0 7fadc9744000-7fadc9753000 r-xp 00000000 08:01 9223 /usr/lib64/libbz2.so.1.0.6 7fadc9753000-7fadc9952000 ---p 0000f000 08:01 9223 /usr/lib64/libbz2.so.1.0.6 7fadc9952000-7fadc9953000 r--p 0000e000 08:01 9223 /usr/lib64/libbz2.so.1.0.6 7fadc9953000-7fadc9954000 rw-p 0000f000 08:01 9223 /usr/lib64/libbz2.so.1.0.6 7fadc9954000-7fadc9979000 r-xp 00000000 08:01 9216 /usr/lib64/liblzma.so.5.2.2 7fadc9979000-7fadc9b78000 ---p 00025000 08:01 9216 /usr/lib64/liblzma.so.5.2.2 7fadc9b78000-7fadc9b79000 r--p 00024000 08:01 9216 /usr/lib64/liblzma.so.5.2.2 7fadc9b79000-7fadc9b7a000 rw-p 00025000 08:01 9216 /usr/lib64/liblzma.so.5.2.2 7fadc9b7a000-7fadc9b91000 r-xp 00000000 08:01 9317 /usr/lib64/libelf-0.172.so 7fadc9b91000-7fadc9d90000 ---p 00017000 08:01 9317 /usr/lib64/libelf-0.172.so 7fadc9d90000-7fadc9d91000 r--p 00016000 08:01 9317 /usr/lib64/libelf-0.172.so 7fadc9d91000-7fadc9d92000 rw-p 00017000 08:01 9317 /usr/lib64/libelf-0.172.so 7fadc9d92000-7fadc9d96000 r-xp 00000000 08:01 9364 /usr/lib64/libattr.so.1.1.0 7fadc9d96000-7fadc9f95000 ---p 00004000 08:01 9364 /usr/lib64/libattr.so.1.1.0 7fadc9f95000-7fadc9f96000 r--p 00003000 08:01 9364 /usr/lib64/libattr.so.1.1.0 7fadc9f96000-7fadc9f97000 rw-p 00004000 08:01 9364 /usr/lib64/libattr.so.1.1.0 7fadc9f97000-7fadc9fb5000 r-xp 00000000 08:01 9529 /usr/lib64/libnl-3.so.200.23.0 7fadc9fb5000-7fadca1b5000 ---p 0001e000 08:01 9529 /usr/lib64/libnl-3.so.200.23.0 7fadca1b5000-7fadca1b7000 r--p 0001e000 08:01 9529 /usr/lib64/libnl-3.so.200.23.0 7fadca1b7000-7fadca1b8000 rw-p 00020000 08:01 9529 /usr/lib64/libnl-3.so.200.23.0 7fadca1b8000-7fadca21c000 r-xp 00000000 08:01 9537 /usr/lib64/libnl-route-3.so.200.23.0 7fadca21c000-7fadca41b000 ---p 00064000 08:01 9537 /usr/lib64/libnl-route-3.so.200.23.0 7fadca41b000-7fadca41e000 r--p 00063000 08:01 9537 /usr/lib64/libnl-route-3.so.200.23.0 7fadca41e000-7fadca423000 rw-p 00066000 08:01 9537 /usr/lib64/libnl-route-3.so.200.23.0 7fadca423000-7fadca425000 rw-p 00000000 00:00 0 7fadca425000-7fadca471000 r-xp 00000000 08:01 11388 /usr/lib64/libdw-0.172.so 7fadca471000-7fadca671000 ---p 0004c000 08:01 11388 /usr/lib64/libdw-0.172.so 7fadca671000-7fadca673000 r--p 0004c000 08:01 11388 /usr/lib64/libdw-0.172.so 7fadca673000-7fadca674000 rw-p 0004e000 08:01 11388 /usr/lib64/libdw-0.172.so 7fadca674000-7fadca678000 r-xp 00000000 08:01 9366 /usr/lib64/libcap.so.2.22 7fadca678000-7fadca877000 ---p 00004000 08:01 9366 /usr/lib64/libcap.so.2.22 7fadca877000-7fadca878000 r--p 00003000 08:01 9366 /usr/lib64/libcap.so.2.22 7fadca878000-7fadca879000 rw-p 00004000 08:01 9366 /usr/lib64/libcap.so.2.22 7fadca879000-7fadcaa3b000 r-xp 00000000 08:01 9086 /usr/lib64/libc-2.17.so 7fadcaa3b000-7fadcac3b000 ---p 001c2000 08:01 9086 /usr/lib64/libc-2.17.so 7fadcac3b000-7fadcac3f000 r--p 001c2000 08:01 9086 /usr/lib64/libc-2.17.so 7fadcac3f000-7fadcac41000 rw-p 001c6000 08:01 9086 /usr/lib64/libc-2.17.so 7fadcac41000-7fadcac46000 rw-p 00000000 00:00 0 7fadcac46000-7fadcac5b000 r-xp 00000000 08:01 9799 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fadcac5b000-7fadcae5a000 ---p 00015000 08:01 9799 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fadcae5a000-7fadcae5b000 r--p 00014000 08:01 9799 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fadcae5b000-7fadcae5c000 rw-p 00015000 08:01 9799 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7fadcae5c000-7fadcaf5d000 r-xp 00000000 08:01 9091 /usr/lib64/libm-2.17.so 7fadcaf5d000-7fadcb15c000 ---p 00101000 08:01 9091 /usr/lib64/libm-2.17.so 7fadcb15c000-7fadcb15d000 r--p 00100000 08:01 9091 /usr/lib64/libm-2.17.so 7fadcb15d000-7fadcb15e000 rw-p 00101000 08:01 9091 /usr/lib64/libm-2.17.so 7fadcb15e000-7fadcb247000 r-xp 00000000 08:01 9119 /usr/lib64/libstdc++.so.6.0.19 7fadcb247000-7fadcb446000 ---p 000e9000 08:01 9119 /usr/lib64/libstdc++.so.6.0.19 7fadcb446000-7fadcb44e000 r--p 000e8000 08:01 9119 /usr/lib64/libstdc++.so.6.0.19 7fadcb44e000-7fadcb450000 rw-p 000f0000 08:01 9119 /usr/lib64/libstdc++.so.6.0.19 7fadcb450000-7fadcb465000 rw-p 00000000 00:00 0 7fadcb465000-7fadcb47b000 r-xp 00000000 08:01 9106 /usr/lib64/libresolv-2.17.so 7fadcb47b000-7fadcb67a000 ---p 00016000 08:01 9106 /usr/lib64/libresolv-2.17.so 7fadcb67a000-7fadcb67b000 r--p 00015000 08:01 9106 /usr/lib64/libresolv-2.17.so 7fadcb67b000-7fadcb67c000 rw-p 00016000 08:01 9106 /usr/lib64/libresolv-2.17.so 7fadcb67c000-7fadcb67e000 rw-p 00000000 00:00 0 7fadcb67e000-7fadcb685000 r-xp 00000000 08:01 9107 /usr/lib64/librt-2.17.so 7fadcb685000-7fadcb884000 ---p 00007000 08:01 9107 /usr/lib64/librt-2.17.so 7fadcb884000-7fadcb885000 r--p 00006000 08:01 9107 /usr/lib64/librt-2.17.so 7fadcb885000-7fadcb886000 rw-p 00007000 08:01 9107 /usr/lib64/librt-2.17.so 7fadcb886000-7fadcb89b000 r-xp 00000000 08:01 14974 /usr/lib64/librdmacm.so.1.1.17.2 7fadcb89b000-7fadcba9a000 ---p 00015000 08:01 14974 /usr/lib64/librdmacm.so.1.1.17.2 7fadcba9a000-7fadcba9b000 r--p 00014000 08:01 14974 /usr/lib64/librdmacm.so.1.1.17.2 7fadcba9b000-7fadcba9c000 rw-p 00015000 08:01 14974 /usr/lib64/librdmacm.so.1.1.17.2 7fadcba9c000-7fadcba9d000 rw-p 00000000 00:00 0 7fadcba9d000-7fadcbab2000 r-xp 00000000 08:01 14968 /usr/lib64/libibverbs.so.1.1.17.2 7fadcbab2000-7fadcbcb2000 ---p 00015000 08:01 14968 /usr/lib64/libibverbs.so.1.1.17.2 7fadcbcb2000-7fadcbcb3000 r--p 00015000 08:01 14968 /usr/lib64/libibverbs.so.1.1.17.2 7fadcbcb3000-7fadcbcb4000 rw-p 00016000 08:01 14968 /usr/lib64/libibverbs.so.1.1.17.2 7fadcbcb4000-7fadcbcc9000 r-xp 00000000 08:01 11403 /usr/lib64/libudev.so.1.6.2 7fadcbcc9000-7fadcbec8000 ---p 00015000 08:01 11403 /usr/lib64/libudev.so.1.6.2 7fadcbec8000-7fadcbec9000 r--p 00014000 08:01 11403 /usr/lib64/libudev.so.1.6.2 7fadcbec9000-7fadcbeca000 rw-p 00015000 08:01 11403 /usr/lib64/libudev.so.1.6.2 7fadcbeca000-7fadcbee1000 r-xp 00000000 08:01 9105 /usr/lib64/libpthread-2.17.so 7fadcbee1000-7fadcc0e0000 ---p 00017000 08:01 9105 /usr/lib64/libpthread-2.17.so 7fadcc0e0000-7fadcc0e1000 r--p 00016000 08:01 9105 /usr/lib64/libpthread-2.17.so 7fadcc0e1000-7fadcc0e2000 rw-p 00017000 08:01 9105 /usr/lib64/libpthread-2.17.so 7fadcc0e2000-7fadcc0e6000 rw-p 00000000 00:00 0 7fadcc0e6000-7fadcc31a000 r-xp 00000000 08:01 10149 /usr/lib64/libcrypto.so.1.0.2k 7fadcc31a000-7fadcc51a000 ---p 00234000 08:01 10149 /usr/lib64/libcrypto.so.1.0.2k 7fadcc51a000-7fadcc536000 r--p 00234000 08:01 10149 /usr/lib64/libcrypto.so.1.0.2k 7fadcc536000-7fadcc543000 rw-p 00250000 08:01 10149 /usr/lib64/libcrypto.so.1.0.2k 7fadcc543000-7fadcc547000 rw-p 00000000 00:00 0 7fadcc547000-7fadcc583000 r-xp 00000000 08:01 10273 /usr/lib64/libblkid.so.1.1.0 7fadcc583000-7fadcc782000 ---p 0003c000 08:01 10273 /usr/lib64/libblkid.so.1.1.0 7fadcc782000-7fadcc785000 r--p 0003b000 08:01 10273 /usr/lib64/libblkid.so.1.1.0 7fadcc785000-7fadcc786000 rw-p 0003e000 08:01 10273 /usr/lib64/libblkid.so.1.1.0 7fadcc786000-7fadcc787000 rw-p 00000000 00:00 0 7fadcc787000-7fadcd10d000 r-xp 00000000 08:01 393372 /usr/lib64/ceph/libceph-common.so.0 7fadcd10d000-7fadcd30d000 ---p 00986000 08:01 393372 /usr/lib64/ceph/libceph-common.so.0 7fadcd30d000-7fadcd32c000 r--p 00986000 08:01 393372 /usr/lib64/ceph/libceph-common.so.0 7fadcd32c000-7fadcd335000 rw-p 009a5000 08:01 393372 /usr/lib64/ceph/libceph-common.so.0 7fadcd335000-7fadd5555000 rw-p 00000000 00:00 0 7fadd5555000-7fadd559b000 r-xp 00000000 08:01 14944 /usr/lib64/libtcmalloc.so.4.4.5 7fadd559b000-7fadd579b000 ---p 00046000 08:01 14944 /usr/lib64/libtcmalloc.so.4.4.5 7fadd579b000-7fadd579c000 r--p 00046000 08:01 14944 /usr/lib64/libtcmalloc.so.4.4.5 7fadd579c000-7fadd579d000 rw-p 00047000 08:01 14944 /usr/lib64/libtcmalloc.so.4.4.5 7fadd579d000-7fadd594a000 rw-p 00000000 00:00 0 7fadd594a000-7fadd594e000 r-xp 00000000 08:01 9221 /usr/lib64/libuuid.so.1.3.0 7fadd594e000-7fadd5b4d000 ---p 00004000 08:01 9221 /usr/lib64/libuuid.so.1.3.0 7fadd5b4d000-7fadd5b4e000 r--p 00003000 08:01 9221 /usr/lib64/libuuid.so.1.3.0 7fadd5b4e000-7fadd5b4f000 rw-p 00004000 08:01 9221 /usr/lib64/libuuid.so.1.3.0 7fadd5b4f000-7fadd5b50000 r-xp 00000000 08:01 9579 /usr/lib64/libaio.so.1.0.1 7fadd5b50000-7fadd5d4f000 ---p 00001000 08:01 9579 /usr/lib64/libaio.so.1.0.1 7fadd5d4f000-7fadd5d50000 r--p 00000000 08:01 9579 /usr/lib64/libaio.so.1.0.1 7fadd5d50000-7fadd5d51000 rw-p 00001000 08:01 9579 /usr/lib64/libaio.so.1.0.1 7fadd5d51000-7fadd5d66000 r-xp 00000000 08:01 9206 /usr/lib64/libz.so.1.2.7 7fadd5d66000-7fadd5f65000 ---p 00015000 08:01 9206 /usr/lib64/libz.so.1.2.7 7fadd5f65000-7fadd5f66000 r--p 00014000 08:01 9206 /usr/lib64/libz.so.1.2.7 7fadd5f66000-7fadd5f67000 rw-p 00015000 08:01 9206 /usr/lib64/libz.so.1.2.7 7fadd5f67000-7fadd5f7b000 r-xp 00000000 08:01 9677 /usr/lib64/liblz4.so.1.7.5 7fadd5f7b000-7fadd617a000 ---p 00014000 08:01 9677 /usr/lib64/liblz4.so.1.7.5 7fadd617a000-7fadd617b000 r--p 00013000 08:01 9677 /usr/lib64/liblz4.so.1.7.5 7fadd617b000-7fadd617c000 rw-p 00014000 08:01 9677 /usr/lib64/liblz4.so.1.7.5 7fadd617c000-7fadd6181000 r-xp 00000000 08:01 10098 /usr/lib64/libsnappy.so.1.1.4 7fadd6181000-7fadd6380000 ---p 00005000 08:01 10098 /usr/lib64/libsnappy.so.1.1.4 7fadd6380000-7fadd6381000 r--p 00004000 08:01 10098 /usr/lib64/libsnappy.so.1.1.4 7fadd6381000-7fadd6382000 rw-p 00005000 08:01 10098 /usr/lib64/libsnappy.so.1.1.4 7fadd6382000-7fadd63d4000 r-xp 00000000 08:01 1270 /usr/lib64/libleveldb.so.1.0.7 7fadd63d4000-7fadd65d3000 ---p 00052000 08:01 1270 /usr/lib64/libleveldb.so.1.0.7 7fadd65d3000-7fadd65d5000 r--p 00051000 08:01 1270 /usr/lib64/libleveldb.so.1.0.7 7fadd65d5000-7fadd65d6000 rw-p 00053000 08:01 1270 /usr/lib64/libleveldb.so.1.0.7 7fadd65d6000-7fadd6601000 r-xp 00000000 08:01 14619 /usr/lib64/libfuse.so.2.9.2 7fadd6601000-7fadd6801000 ---p 0002b000 08:01 14619 /usr/lib64/libfuse.so.2.9.2 7fadd6801000-7fadd6813000 r--p 0002b000 08:01 14619 /usr/lib64/libfuse.so.2.9.2 7fadd6813000-7fadd6814000 rw-p 0003d000 08:01 14619 /usr/lib64/libfuse.so.2.9.2 7fadd6814000-7fadd6816000 r-xp 00000000 08:01 9090 /usr/lib64/libdl-2.17.so 7fadd6816000-7fadd6a16000 ---p 00002000 08:01 9090 /usr/lib64/libdl-2.17.so 7fadd6a16000-7fadd6a17000 r--p 00002000 08:01 9090 /usr/lib64/libdl-2.17.so 7fadd6a17000-7fadd6a18000 rw-p 00003000 08:01 9090 /usr/lib64/libdl-2.17.so 7fadd6a18000-7fadd6a3a000 r-xp 00000000 08:01 9080 /usr/lib64/ld-2.17.so 7fadd6c07000-7fadd6c2b000 rw-p 00000000 00:00 0 7fadd6c37000-7fadd6c39000 rw-p 00000000 00:00 0 7fadd6c39000-7fadd6c3a000 r--p 00021000 08:01 9080 /usr/lib64/ld-2.17.so 7fadd6c3a000-7fadd6c3b000 rw-p 00022000 08:01 9080 /usr/lib64/ld-2.17.so 7fadd6c3b000-7fadd6c3c000 rw-p 00000000 00:00 0 7ffc1265e000-7ffc12717000 rw-p 00000000 00:00 0 [stack] 7ffc1272b000-7ffc1272d000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] *** Caught signal (Aborted) ** in thread 7fadd6c1a980 thread_name:ceph-objectstor ceph version 15.0.0-4067-g4752860 (475286020efbcbe469ec6169e1c477e69c61c6c6) octopus (dev) 1: (()+0xf5d0) [0x7fadcbed95d0] 2: (gsignal()+0x37) [0x7fadca8af2c7] 3: (abort()+0x148) [0x7fadca8b09b8] 4: (()+0x78e17) [0x7fadca8f1e17] 5: (__fortify_fail()+0x37) [0x7fadca990b67] 6: (()+0x117b22) [0x7fadca990b22] 7: (update_mon_db(ObjectStore&, OSDSuperblock&, std::string const&, std::string const&)+0x2c51) [0x5581bb233a91] 8: (main()+0x557e) [0x5581bb1b805e] 9: (__libc_start_main()+0xf5) [0x7fadca89b495] 10: (()+0x3cfe50) [0x5581bb1e7e50]

pass 4230940 2019-08-19 18:24:20 2019-08-19 20:43:37 2019-08-19 21:45:37 1:02:00 0:49:59 0:12:01 smithi master ubuntu 18.04 rados/monthrash/{ceph.yaml clusters/3-mons.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml} thrashers/one.yaml workloads/rados_mon_osdmap_prune.yaml} 2
dead 4230941 2019-08-19 18:24:21 2019-08-19 20:45:26 2019-08-20 08:47:49 12:02:23 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/filestore-xfs.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/redirect.yaml} 2
pass 4230942 2019-08-19 18:24:22 2019-08-19 20:49:36 2019-08-19 21:13:35 0:23:59 0:18:17 0:05:42 smithi master rhel 7.6 rados/perf/{ceph.yaml objectstore/bluestore-low-osd-mem-target.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{rhel_7.yaml} workloads/radosbench_4K_rand_read.yaml} 1
dead 4230943 2019-08-19 18:24:23 2019-08-19 20:51:23 2019-08-20 08:53:57 12:02:34 smithi master centos 7.6 rados/singleton/{all/recovery-preemption.yaml msgr-failures/few.yaml msgr/async-v1only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
pass 4230944 2019-08-19 18:24:24 2019-08-19 20:51:27 2019-08-19 21:15:26 0:23:59 0:13:59 0:10:00 smithi master rhel 7.6 rados/singleton-nomsgr/{all/cache-fs-trunc.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
dead 4230945 2019-08-19 18:24:25 2019-08-19 20:57:41 2019-08-20 09:00:09 12:02:28 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{default.yaml} backoff/peering_and_degraded.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/off.yaml msgr-failures/few.yaml msgr/async-v2only.yaml objectstore/bluestore-bitmap.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/redirect_promote_tests.yaml} 2
pass 4230946 2019-08-19 18:24:26 2019-08-19 20:57:41 2019-08-19 21:49:41 0:52:00 0:09:23 0:42:37 smithi master ubuntu 18.04 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-bitmap.yaml supported-random-distro$/{ubuntu_latest.yaml} tasks/orchestrator_cli.yaml} 2
dead 4230947 2019-08-19 18:24:27 2019-08-19 20:59:26 2019-08-20 09:01:54 12:02:28 smithi master rhel 7.6 rados/thrash-erasure-code/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/osd-delay.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{rhel_7.yaml} thrashers/careful.yaml thrashosds-health.yaml workloads/ec-small-objects-many-deletes.yaml} 2
dead 4230948 2019-08-19 18:24:28 2019-08-19 21:01:19 2019-08-20 09:03:42 12:02:23 smithi master ubuntu 18.04 rados/thrash-erasure-code-isa/{arch/x86_64.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/osd-delay.yaml objectstore/bluestore-stupid.yaml rados.yaml recovery-overrides/{default.yaml} supported-random-distro$/{ubuntu_latest.yaml} thrashers/pggrow.yaml thrashosds-health.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml} 2
pass 4230949 2019-08-19 18:24:29 2019-08-19 21:05:36 2019-08-19 21:33:36 0:28:00 0:14:37 0:13:23 smithi master rhel 7.6 rados/singleton/{all/resolve_stuck_peering.yaml msgr-failures/many.yaml msgr/async-v2only.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 2
dead 4230950 2019-08-19 18:24:30 2019-08-19 21:09:34 2019-08-20 06:31:44 9:22:10 smithi master centos 7.6 rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-install/luminous.yaml backoff/peering.yaml ceph.yaml clusters/{openstack.yaml three-plus-one.yaml} d-balancer/crush-compat.yaml distro$/{centos_latest.yaml} msgr-failures/fastclose.yaml rados.yaml thrashers/mapgap.yaml thrashosds-health.yaml workloads/radosbench.yaml} 4
Failure Reason:

SSH connection to smithi115 was lost: 'rpm -q kernel --last | head -n 1'

pass 4230951 2019-08-19 18:24:31 2019-08-19 21:13:07 2019-08-19 21:51:07 0:38:00 0:22:58 0:15:02 smithi master centos 7.6 rados/basic/{ceph.yaml clusters/{fixed-2.yaml openstack.yaml} msgr-failures/many.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} tasks/rados_cls_all.yaml} 2
dead 4230952 2019-08-19 18:24:32 2019-08-19 21:13:16 2019-08-20 09:15:40 12:02:24 smithi master centos 7.6 rados/thrash/{0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/upmap.yaml msgr-failures/osd-delay.yaml msgr/async.yaml objectstore/bluestore-comp.yaml rados.yaml supported-random-distro$/{centos_7.yaml} thrashers/default.yaml thrashosds-health.yaml workloads/redirect_set_object.yaml} 2
pass 4230953 2019-08-19 18:24:33 2019-08-19 21:13:36 2019-08-19 21:33:35 0:19:59 0:10:48 0:09:11 smithi master ubuntu 18.04 rados/perf/{ceph.yaml objectstore/bluestore-stupid.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{ubuntu_latest.yaml} workloads/radosbench_4K_seq_read.yaml} 1
pass 4230954 2019-08-19 18:24:34 2019-08-19 21:15:41 2019-08-19 21:35:40 0:19:59 0:08:14 0:11:45 smithi master ubuntu 18.04 rados/singleton/{all/test-crash.yaml msgr-failures/few.yaml msgr/async.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{ubuntu_latest.yaml}} 1
dead 4230955 2019-08-19 18:24:35 2019-08-19 21:23:34 2019-08-20 09:25:57 12:02:23 smithi master rhel 7.6 rados/objectstore/{backends/ceph_objectstore_tool.yaml supported-random-distro$/{rhel_7.yaml}} 1
dead 4230956 2019-08-19 18:24:36 2019-08-19 21:23:39 2019-08-20 09:26:02 12:02:23 smithi master centos 7.6 rados/thrash-erasure-code-big/{ceph.yaml cluster/{12-osds.yaml openstack.yaml} msgr-failures/fastclose.yaml objectstore/bluestore-comp.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/fastread.yaml thrashosds-health.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml} 3
dead 4230957 2019-08-19 18:24:37 2019-08-19 21:27:43 2019-08-20 09:30:11 12:02:28 smithi master centos 7.6 rados/thrash-erasure-code-overwrites/{bluestore-bitmap.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} fast/fast.yaml msgr-failures/few.yaml rados.yaml recovery-overrides/{more-active-recovery.yaml} supported-random-distro$/{centos_7.yaml} thrashers/fastread.yaml thrashosds-health.yaml workloads/ec-pool-snaps-few-objects-overwrites.yaml} 2
dead 4230958 2019-08-19 18:24:38 2019-08-19 21:29:14 2019-08-20 09:31:42 12:02:28 smithi master rhel 7.6 rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml 2-recovery-overrides/{more-active-recovery.yaml} backoff/peering.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-balancer/crush-compat.yaml msgr-failures/fastclose.yaml msgr/async-v1only.yaml objectstore/bluestore-low-osd-mem-target.yaml rados.yaml supported-random-distro$/{rhel_7.yaml} thrashers/mapgap.yaml thrashosds-health.yaml workloads/set-chunk-promote-flush.yaml} 2
pass 4230959 2019-08-19 18:24:39 2019-08-19 21:33:50 2019-08-19 21:55:50 0:22:00 0:13:46 0:08:14 smithi master rhel 7.6 rados/singleton-nomsgr/{all/ceph-kvstore-tool.yaml rados.yaml supported-random-distro$/{rhel_7.yaml}} 1
pass 4230960 2019-08-19 18:24:40 2019-08-19 21:33:50 2019-08-19 22:05:50 0:32:00 0:22:08 0:09:52 smithi master centos 7.6 rados/singleton/{all/test_envlibrados_for_rocksdb.yaml msgr-failures/many.yaml msgr/async-v1only.yaml objectstore/bluestore-stupid.yaml rados.yaml supported-random-distro$/{centos_7.yaml}} 1
fail 4230961 2019-08-19 18:24:41 2019-08-19 21:35:56 2019-08-19 22:01:55 0:25:59 0:11:27 0:14:32 smithi master centos 7.6 rados/mgr/{clusters/{2-node-mgr.yaml openstack.yaml} debug/mgr.yaml objectstore/bluestore-comp.yaml supported-random-distro$/{centos_7.yaml} tasks/progress.yaml} 2
Failure Reason:

Test failure: test_osd_came_back (tasks.mgr.test_progress.TestProgress)

pass 4230962 2019-08-19 18:24:42 2019-08-19 21:39:29 2019-08-19 21:59:28 0:19:59 0:13:35 0:06:24 smithi master centos 7.6 rados/perf/{ceph.yaml objectstore/filestore-xfs.yaml openstack.yaml settings/optimized.yaml supported-random-distro$/{centos_7.yaml} workloads/radosbench_4M_rand_read.yaml} 1