Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 4693557 2020-01-22 12:43:22 2020-01-22 12:44:25 2020-01-22 13:22:24 0:37:59 0:25:28 0:12:31 smithi master upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap.yaml clusters/1-mds-2-client-micro.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} overrides/{frag_enable.yaml multimds/no.yaml pg-warn.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/{0-mimic.yaml 1-client.yaml 2-upgrade.yaml 3-compat_client/mimic.yaml}} 3
Failure Reason:

Command failed on smithi101 with status 124: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_health_to_clog=true"

fail 4693558 2020-01-22 12:43:23 2020-01-22 12:44:26 2020-01-22 13:14:25 0:29:59 0:12:44 0:17:15 smithi master ubuntu 18.04 upgrade/mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/bluestore-bitmap.yaml ubuntu_latest.yaml} 4
Failure Reason:

"2020-01-22 13:06:09.852539 mon.b mon.0 172.21.15.25:6789/0 81 : cluster [WRN] Health check failed: too few PGs per OSD (2 < min 30) (TOO_FEW_PGS)" in cluster log

pass 4693559 2020-01-22 12:43:24 2020-01-22 12:44:51 2020-01-22 14:32:54 1:48:03 1:30:53 0:17:10 smithi master ubuntu 18.04 upgrade/mimic-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/mimic.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-octopus.yaml 8-workload/{rbd-python.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} 4
fail 4693560 2020-01-22 12:43:25 2020-01-22 12:44:59 2020-01-22 13:02:59 0:18:00 0:03:25 0:14:35 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/bluestore-bitmap.yaml octopus.yaml ubuntu_latest.yaml} 4
Failure Reason:

Command failed on smithi149 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

fail 4693561 2020-01-22 12:43:26 2020-01-22 12:47:14 2020-01-22 13:07:14 0:20:00 0:03:21 0:16:39 smithi master ubuntu 18.04 upgrade/nautilus-x-singleton/{0-cluster/{openstack.yaml start.yaml} 1-install/nautilus.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 5-workload/{radosbench.yaml rbd_api.yaml} 6-finish-upgrade.yaml 7-octopus.yaml 8-workload/{rbd-python.yaml snaps-many-objects.yaml} bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} 4
Failure Reason:

Command failed on smithi160 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic'

fail 4693562 2020-01-22 12:43:27 2020-01-22 12:47:14 2020-01-22 13:55:15 1:08:01 0:13:15 0:54:46 smithi master ubuntu 18.04 upgrade/mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi139 with status 22: "sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog"

fail 4693563 2020-01-22 12:43:28 2020-01-22 12:47:31 2020-01-22 13:05:30 0:17:59 0:03:37 0:14:22 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi100 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

pass 4693564 2020-01-22 12:43:29 2020-01-22 12:49:17 2020-01-22 13:19:17 0:30:00 0:18:15 0:11:45 smithi master upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap.yaml clusters/1-mds-2-client-micro.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} overrides/{frag_enable.yaml multimds/no.yaml pg-warn.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/{0-mimic.yaml 1-client.yaml 2-upgrade.yaml 3-client-upgrade.yaml 4-compat_client.yaml 5-client-sanity.yaml}} 3
fail 4693565 2020-01-22 12:43:30 2020-01-22 12:49:17 2020-01-22 13:45:19 0:56:02 0:12:59 0:43:03 smithi master ubuntu 18.04 upgrade/mimic-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-luminous-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-ec-workload.yaml 5-finish-upgrade.yaml 7-final-workload.yaml objectstore/bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi027 with status 22: "sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog"

fail 4693566 2020-01-22 12:43:31 2020-01-22 12:51:01 2020-01-22 13:45:01 0:54:00 0:03:43 0:50:17 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-nautilus-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 3.1-objectstore/bluestore-bitmap.yaml 4-ec-workload/{rados-ec-workload.yaml rbd-ec-workload.yaml} 5-finish-upgrade.yaml 7-final-workload.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi098 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

fail 4693567 2020-01-22 12:43:32 2020-01-22 12:51:01 2020-01-22 13:19:01 0:28:00 0:12:21 0:15:39 smithi master ubuntu 18.04 upgrade/mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/filestore-xfs.yaml ubuntu_latest.yaml} 4
Failure Reason:

"2020-01-22 13:10:50.060548 mon.a mon.0 172.21.15.2:6789/0 141 : cluster [WRN] Health check failed: too few PGs per OSD (2 < min 30) (TOO_FEW_PGS)" in cluster log

fail 4693568 2020-01-22 12:43:33 2020-01-22 12:51:01 2020-01-22 13:11:01 0:20:00 0:03:16 0:16:44 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/filestore-xfs.yaml octopus.yaml ubuntu_latest.yaml} 4
Failure Reason:

Command failed on smithi150 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

pass 4693569 2020-01-22 12:43:34 2020-01-22 12:51:01 2020-01-22 13:35:01 0:44:00 0:32:33 0:11:27 smithi master upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap.yaml clusters/1-mds-2-client-micro.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} overrides/{frag_enable.yaml multimds/yes.yaml pg-warn.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/{0-mimic.yaml 1-client.yaml 2-upgrade.yaml 3-compat_client/no.yaml}} 3
fail 4693570 2020-01-22 12:43:35 2020-01-22 12:53:01 2020-01-22 13:29:01 0:36:00 0:12:37 0:23:23 smithi master ubuntu 18.04 upgrade/mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/filestore-xfs.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi029 with status 22: "sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog"

fail 4693571 2020-01-22 12:43:36 2020-01-22 12:53:01 2020-01-22 13:21:00 0:27:59 0:03:38 0:24:21 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/filestore-xfs.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi117 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

fail 4693572 2020-01-22 12:43:37 2020-01-22 12:55:15 2020-01-22 13:31:15 0:36:00 0:12:39 0:23:21 smithi master ubuntu 18.04 upgrade/mimic-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-luminous-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-ec-workload.yaml 5-finish-upgrade.yaml 7-final-workload.yaml objectstore/filestore-xfs.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi197 with status 22: "sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog"

fail 4693573 2020-01-22 12:43:38 2020-01-22 12:56:52 2020-01-22 13:20:52 0:24:00 0:03:35 0:20:25 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-nautilus-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 3.1-objectstore/filestore-xfs.yaml 4-ec-workload/{rados-ec-workload.yaml rbd-ec-workload.yaml} 5-finish-upgrade.yaml 7-final-workload.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi123 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

pass 4693574 2020-01-22 12:43:39 2020-01-22 12:56:59 2020-01-22 13:34:59 0:38:00 0:25:15 0:12:45 smithi master upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap.yaml clusters/1-mds-2-client-micro.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} overrides/{frag_enable.yaml multimds/yes.yaml pg-warn.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/{0-mimic.yaml 1-client.yaml 2-upgrade.yaml 3-compat_client/mimic.yaml}} 3
fail 4693575 2020-01-22 12:43:40 2020-01-22 12:59:27 2020-01-22 13:35:27 0:36:00 0:12:28 0:23:32 smithi master ubuntu 18.04 upgrade/mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/bluestore-bitmap.yaml ubuntu_latest.yaml} 4
Failure Reason:

"2020-01-22 13:26:33.002139 mon.c mon.0 172.21.15.157:6789/0 88 : cluster [WRN] Health check failed: too few PGs per OSD (2 < min 30) (TOO_FEW_PGS)" in cluster log

fail 4693576 2020-01-22 12:43:41 2020-01-22 13:01:02 2020-01-22 13:17:01 0:15:59 0:03:15 0:12:44 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/bluestore-bitmap.yaml octopus.yaml ubuntu_latest.yaml} 4
Failure Reason:

Command failed on smithi184 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

fail 4693577 2020-01-22 12:43:42 2020-01-22 13:03:24 2020-01-22 13:39:24 0:36:00 0:13:14 0:22:46 smithi master ubuntu 18.04 upgrade/mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi172 with status 22: "sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog"

fail 4693578 2020-01-22 12:43:43 2020-01-22 13:04:38 2020-01-22 13:32:37 0:27:59 0:03:37 0:24:22 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi109 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

pass 4693579 2020-01-22 12:43:44 2020-01-22 13:05:00 2020-01-22 13:35:00 0:30:00 0:18:31 0:11:29 smithi master upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap.yaml clusters/1-mds-2-client-micro.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} overrides/{frag_enable.yaml multimds/yes.yaml pg-warn.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/{0-mimic.yaml 1-client.yaml 2-upgrade.yaml 3-client-upgrade.yaml 4-compat_client.yaml 5-client-sanity.yaml}} 3
fail 4693580 2020-01-22 12:43:45 2020-01-22 13:05:32 2020-01-22 13:31:31 0:25:59 0:12:50 0:13:09 smithi master ubuntu 18.04 upgrade/mimic-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-luminous-install/mimic.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-ec-workload.yaml 5-finish-upgrade.yaml 7-final-workload.yaml objectstore/bluestore-bitmap.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi068 with status 22: "sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog"

fail 4693581 2020-01-22 12:43:46 2020-01-22 13:06:39 2020-01-22 13:30:39 0:24:00 0:03:39 0:20:21 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-nautilus-install/nautilus.yaml 1.1-pg-log-overrides/short_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 3.1-objectstore/bluestore-bitmap.yaml 4-ec-workload/{rados-ec-workload.yaml rbd-ec-workload.yaml} 5-finish-upgrade.yaml 7-final-workload.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi026 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

fail 4693582 2020-01-22 12:43:47 2020-01-22 13:07:01 2020-01-22 13:37:00 0:29:59 0:12:28 0:17:31 smithi master ubuntu 18.04 upgrade/mimic-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 4-msgr2.yaml 4-nautilus.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/filestore-xfs.yaml ubuntu_latest.yaml} 4
Failure Reason:

"2020-01-22 13:28:28.304593 mon.a mon.0 172.21.15.38:6789/0 129 : cluster [WRN] Health check failed: too few PGs per OSD (2 < min 30) (TOO_FEW_PGS)" in cluster log

fail 4693583 2020-01-22 12:43:48 2020-01-22 13:07:15 2020-01-22 13:29:15 0:22:00 0:03:22 0:18:38 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-workload/{blogbench.yaml ec-rados-default.yaml rados_api.yaml rados_loadgenbig.yaml rgw_ragweed_prepare.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-upgrade-sequence/upgrade-all.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw.yaml rgw_ragweed_check.yaml} objectstore/filestore-xfs.yaml octopus.yaml ubuntu_latest.yaml} 4
Failure Reason:

Command failed on smithi098 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

fail 4693584 2020-01-22 12:43:49 2020-01-22 13:10:30 2020-01-22 13:48:30 0:38:00 0:25:56 0:12:04 smithi master upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap.yaml clusters/1-mds-2-client-micro.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} overrides/{frag_enable.yaml multimds/no.yaml pg-warn.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/{0-mimic.yaml 1-client.yaml 2-upgrade.yaml 3-compat_client/no.yaml}} 3
Failure Reason:

Command failed on smithi118 with status 124: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_health_to_clog=true"

fail 4693585 2020-01-22 12:43:50 2020-01-22 13:10:52 2020-01-22 13:40:52 0:30:00 0:12:50 0:17:10 smithi master ubuntu 18.04 upgrade/mimic-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/filestore-xfs.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi190 with status 22: "sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog"

fail 4693586 2020-01-22 12:43:51 2020-01-22 13:10:53 2020-01-22 13:50:53 0:40:00 0:03:40 0:36:20 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-ceph-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml rgw_ragweed_prepare.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-msgr2.yaml 6-nautilus.yaml 7-final-workload/{rbd-python.yaml snaps-many-objects.yaml} objectstore/filestore-xfs.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi192 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'

fail 4693587 2020-01-22 12:43:52 2020-01-22 13:10:55 2020-01-22 13:40:55 0:30:00 0:12:51 0:17:09 smithi master ubuntu 18.04 upgrade/mimic-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-luminous-install/mimic.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-ec-workload.yaml 5-finish-upgrade.yaml 7-final-workload.yaml objectstore/filestore-xfs.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi060 with status 22: "sudo ceph --cluster ceph --mon-client-directed-command-retry 5 tell 'mon.*' injectargs -- --no-mon-health-to-clog"

fail 4693588 2020-01-22 12:43:53 2020-01-22 13:11:02 2020-01-22 13:49:02 0:38:00 0:03:41 0:34:19 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-nautilus-install/nautilus.yaml 1.1-pg-log-overrides/normal_pg_log.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 3.1-objectstore/filestore-xfs.yaml 4-ec-workload/{rados-ec-workload.yaml rbd-ec-workload.yaml} 5-finish-upgrade.yaml 7-final-workload.yaml thrashosds-health.yaml ubuntu_latest.yaml} 5
Failure Reason:

Command failed on smithi136 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=14.2.6-315-g47afc1d-1bionic cephadm=14.2.6-315-g47afc1d-1bionic ceph-mds=14.2.6-315-g47afc1d-1bionic ceph-mgr=14.2.6-315-g47afc1d-1bionic ceph-common=14.2.6-315-g47afc1d-1bionic ceph-fuse=14.2.6-315-g47afc1d-1bionic ceph-test=14.2.6-315-g47afc1d-1bionic radosgw=14.2.6-315-g47afc1d-1bionic python3-rados=14.2.6-315-g47afc1d-1bionic python3-rgw=14.2.6-315-g47afc1d-1bionic python3-cephfs=14.2.6-315-g47afc1d-1bionic python3-rbd=14.2.6-315-g47afc1d-1bionic libcephfs2=14.2.6-315-g47afc1d-1bionic libcephfs-dev=14.2.6-315-g47afc1d-1bionic librados2=14.2.6-315-g47afc1d-1bionic librbd1=14.2.6-315-g47afc1d-1bionic rbd-fuse=14.2.6-315-g47afc1d-1bionic'