Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
dead 1481365 2017-08-04 04:23:41 2017-08-06 06:40:07 2017-08-06 18:45:28 12:05:21 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
fail 1481368 2017-08-04 04:23:41 2017-08-06 06:40:08 2017-08-06 07:32:05 0:51:57 0:44:23 0:07:34 smithi master centos 7.3 upgrade:jewel-x/point-to-point-x/{distros/centos_7.3.yaml point-to-point-upgrade.yaml} 3
Failure Reason:

Command failed on smithi011 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

pass 1481371 2017-08-04 04:23:42 2017-08-06 06:40:47 2017-08-06 09:22:47 2:42:00 2:36:03 0:05:57 smithi master centos 7.3 upgrade:jewel-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-luminous.yaml 7-final-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} distros/centos_latest.yaml thrashosds-health.yaml} 3
pass 1481374 2017-08-04 04:23:43 2017-08-06 06:41:00 2017-08-06 08:47:01 2:06:01 1:09:41 0:56:20 smithi master centos 7.3 upgrade:jewel-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/ec-rados-default.yaml 5-finish-upgrade.yaml 6-luminous.yaml 7-final-workload/ec-rados-plugin=jerasure-k=3-m=1.yaml distros/centos_latest.yaml thrashosds-health.yaml} 3
fail 1481377 2017-08-04 04:23:43 2017-08-06 06:41:17 2017-08-06 09:47:16 3:05:59 0:48:59 2:17:00 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/cache-pool-snaps.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

'wait_until_healthy' reached maximum tries (150) after waiting for 900 seconds

fail 1481380 2017-08-04 04:23:44 2017-08-06 06:41:27 2017-08-06 08:33:27 1:52:00 1:18:09 0:33:51 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/ec-rados-default.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

"2017-08-06 07:20:03.623607 mon.b mon.0 172.21.15.110:6789/0 151 : cluster [WRN] Health check failed: 3 osds down (OSD_DOWN)" in cluster log

fail 1481384 2017-08-04 04:23:45 2017-08-06 06:41:27 2017-08-06 07:03:25 0:21:58 0:10:49 0:11:09 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_api.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

Command failed on smithi034 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 1481387 2017-08-04 04:23:46 2017-08-06 06:42:30 2017-08-06 08:12:30 1:30:00 1:21:50 0:08:10 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_loadgenbig.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

"2017-08-06 06:53:32.895514 mon.a mon.0 172.21.15.64:6789/0 69 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 1481391 2017-08-04 04:23:46 2017-08-06 06:42:47 2017-08-06 07:10:41 0:27:54 0:10:25 0:17:29 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_api.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

Command failed on smithi083 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 1481394 2017-08-04 04:23:47 2017-08-06 06:42:49 2017-08-06 08:00:48 1:17:59 1:11:58 0:06:01 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_python.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

"2017-08-06 06:54:46.229376 mon.b mon.0 172.21.15.132:6789/0 125 : cluster [WRN] Health check failed: 3 osds down (OSD_DOWN)" in cluster log

dead 1481397 2017-08-04 04:23:48 2017-08-06 06:42:51 2017-08-06 18:48:42 12:05:51 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
fail 1481399 2017-08-04 04:23:48 2017-08-06 06:43:22 2017-08-06 07:45:20 1:01:58 0:51:53 0:10:05 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/cache-pool-snaps.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

'wait_until_healthy' reached maximum tries (150) after waiting for 900 seconds

fail 1481403 2017-08-04 04:23:49 2017-08-06 06:43:37 2017-08-06 07:23:36 0:39:59 0:10:15 0:29:44 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/ec-rados-default.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

Command failed on smithi002 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 1481406 2017-08-04 04:23:50 2017-08-06 06:44:40 2017-08-06 09:32:36 2:47:56 0:12:56 2:35:00 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_api.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

Command failed (workunit test cls/test_cls_rgw.sh) on smithi061 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=jewel TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

fail 1481408 2017-08-04 04:23:50 2017-08-06 06:44:36 2017-08-06 08:50:36 2:06:00 1:25:57 0:40:03 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_loadgenbig.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

"2017-08-06 07:33:13.592843 mon.b mon.0 172.21.15.90:6789/0 117 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 1481411 2017-08-04 04:23:51 2017-08-06 06:45:10 2017-08-06 08:11:06 1:25:56 1:07:44 0:18:12 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_api.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

"2017-08-06 07:10:01.528178 mon.a mon.0 172.21.15.12:6789/0 173 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET)" in cluster log

fail 1481414 2017-08-04 04:23:52 2017-08-06 06:45:04 2017-08-06 06:51:00 0:05:56 0:00:20 0:05:36 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_python.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

{'smithi172.front.sepia.ceph.com': {'_ansible_parsed': False, 'exception': 'Traceback (most recent call last):\n File "/tmp/ansible_LbbWzy/ansible_module_apt.py", line 965, in <module>\n main()\n File "/tmp/ansible_LbbWzy/ansible_module_apt.py", line 886, in main\n cache.open(progress=None)\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 151, in open\n self._cache = apt_pkg.Cache(progress)\nSystemError: E:Problem renaming the file /var/cache/apt/pkgcache.bin.PyCaSg to /var/cache/apt/pkgcache.bin - rename (2: No such file or directory), W:You may want to run apt-get update to correct these problems\n', '_ansible_no_log': False, 'module_stderr': 'Traceback (most recent call last):\n File "/tmp/ansible_LbbWzy/ansible_module_apt.py", line 965, in <module>\n main()\n File "/tmp/ansible_LbbWzy/ansible_module_apt.py", line 886, in main\n cache.open(progress=None)\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 151, in open\n self._cache = apt_pkg.Cache(progress)\nSystemError: E:Problem renaming the file /var/cache/apt/pkgcache.bin.PyCaSg to /var/cache/apt/pkgcache.bin - rename (2: No such file or directory), W:You may want to run apt-get update to correct these problems\n', 'changed': False, 'module_stdout': '', 'failed': True, 'rc': 1, 'msg': 'MODULE FAILURE'}, 'smithi169.front.sepia.ceph.com': {'_ansible_parsed': False, 'exception': 'Traceback (most recent call last):\n File "/tmp/ansible_A1OkPq/ansible_module_apt.py", line 965, in <module>\n main()\n File "/tmp/ansible_A1OkPq/ansible_module_apt.py", line 886, in main\n cache.open(progress=None)\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 151, in open\n self._cache = apt_pkg.Cache(progress)\nSystemError: E:Problem renaming the file /var/cache/apt/pkgcache.bin.DvETHC to /var/cache/apt/pkgcache.bin - rename (2: No such file or directory), W:You may want to run apt-get update to correct these problems\n', '_ansible_no_log': False, 'module_stderr': 'Traceback (most recent call last):\n File "/tmp/ansible_A1OkPq/ansible_module_apt.py", line 965, in <module>\n main()\n File "/tmp/ansible_A1OkPq/ansible_module_apt.py", line 886, in main\n cache.open(progress=None)\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 151, in open\n self._cache = apt_pkg.Cache(progress)\nSystemError: E:Problem renaming the file /var/cache/apt/pkgcache.bin.DvETHC to /var/cache/apt/pkgcache.bin - rename (2: No such file or directory), W:You may want to run apt-get update to correct these problems\n', 'changed': False, 'module_stdout': '', 'failed': True, 'rc': 1, 'msg': 'MODULE FAILURE'}, 'smithi048.front.sepia.ceph.com': {'_ansible_parsed': False, 'exception': 'Traceback (most recent call last):\n File "/tmp/ansible_DZjnIf/ansible_module_apt.py", line 965, in <module>\n main()\n File "/tmp/ansible_DZjnIf/ansible_module_apt.py", line 886, in main\n cache.open(progress=None)\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 151, in open\n self._cache = apt_pkg.Cache(progress)\nSystemError: E:Problem renaming the file /var/cache/apt/pkgcache.bin.bpuWwX to /var/cache/apt/pkgcache.bin - rename (2: No such file or directory), W:You may want to run apt-get update to correct these problems\n', '_ansible_no_log': False, 'module_stderr': 'Traceback (most recent call last):\n File "/tmp/ansible_DZjnIf/ansible_module_apt.py", line 965, in <module>\n main()\n File "/tmp/ansible_DZjnIf/ansible_module_apt.py", line 886, in main\n cache.open(progress=None)\n File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 151, in open\n self._cache = apt_pkg.Cache(progress)\nSystemError: E:Problem renaming the file /var/cache/apt/pkgcache.bin.bpuWwX to /var/cache/apt/pkgcache.bin - rename (2: No such file or directory), W:You may want to run apt-get update to correct these problems\n', 'changed': False, 'module_stdout': '', 'failed': True, 'rc': 1, 'msg': 'MODULE FAILURE'}}

dead 1481417 2017-08-04 04:23:52 2017-08-06 06:45:12 2017-08-06 18:51:08 12:05:56 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
pass 1481420 2017-08-04 04:23:53 2017-08-06 06:45:15 2017-08-06 11:33:20 4:48:05 2:20:07 2:27:58 smithi master ubuntu 14.04 upgrade:jewel-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-luminous.yaml 7-final-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} distros/ubuntu_14.04.yaml thrashosds-health.yaml} 3
fail 1481423 2017-08-04 04:23:54 2017-08-06 06:46:32 2017-08-06 11:32:39 4:46:07 1:19:59 3:26:08 smithi master ubuntu 14.04 upgrade:jewel-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/ec-rados-default.yaml 5-finish-upgrade.yaml 6-luminous.yaml 7-final-workload/ec-rados-plugin=jerasure-k=3-m=1.yaml distros/ubuntu_14.04.yaml thrashosds-health.yaml} 3
Failure Reason:

timed out waiting for mon to be updated with osd.1: 0 < 81604379468

fail 1481426 2017-08-04 04:23:54 2017-08-06 06:46:39 2017-08-06 07:10:34 0:23:55 0:13:23 0:10:32 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/cache-pool-snaps.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

Command failed on smithi037 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 1481429 2017-08-04 04:23:55 2017-08-06 06:46:38 2017-08-06 10:04:37 3:17:59 1:43:42 1:34:17 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/ec-rados-default.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

"2017-08-06 08:24:06.939645 mon.b mon.0 172.21.15.47:6789/0 126 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 1481432 2017-08-04 04:23:56 2017-08-06 06:46:41 2017-08-06 08:24:45 1:38:04 1:17:56 0:20:08 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_api.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

"2017-08-06 07:13:03.092499 mon.b mon.0 172.21.15.84:6789/0 111 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 1481435 2017-08-04 04:23:56 2017-08-06 06:46:43 2017-08-06 08:52:43 2:06:00 1:24:52 0:41:08 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_loadgenbig.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

"2017-08-06 07:35:02.966131 mon.a mon.0 172.21.15.34:6789/0 78 : cluster [WRN] Health check failed: 4 osds down (OSD_DOWN)" in cluster log

fail 1481438 2017-08-04 04:23:57 2017-08-06 06:46:54 2017-08-06 08:06:53 1:19:59 1:09:58 0:10:01 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_api.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

"2017-08-06 07:00:53.893265 mon.b mon.0 172.21.15.48:6789/0 9 : cluster [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET)" in cluster log

fail 1481441 2017-08-04 04:23:58 2017-08-06 06:47:41 2017-08-06 08:15:37 1:27:56 1:19:56 0:08:00 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_python.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

"2017-08-06 07:00:44.022187 mon.b mon.0 172.21.15.97:6789/0 120 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 1481444 2017-08-04 04:23:58 2017-08-06 06:48:08 2017-08-06 08:09:54 1:21:46 1:16:18 0:05:28 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

"2017-08-06 07:02:28.081105 mon.b mon.0 172.21.15.130:6789/0 69 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 1481447 2017-08-04 04:23:59 2017-08-06 06:48:15 2017-08-06 07:48:10 0:59:55 0:51:17 0:08:38 smithi master ubuntu 14.04 upgrade:jewel-x/point-to-point-x/{distros/ubuntu_14.04.yaml point-to-point-upgrade.yaml} 3
Failure Reason:

Command failed (workunit test rados/test-upgrade-v11.0.0.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=jewel TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CLS_RBD_GTEST_FILTER=\'*:-TestClsRbd.mirror_image\' adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/rados/test-upgrade-v11.0.0.sh'

fail 1481450 2017-08-04 04:24:00 2017-08-06 06:50:44 2017-08-06 09:04:42 2:13:58 0:53:09 1:20:49 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/cache-pool-snaps.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

'wait_until_healthy' reached maximum tries (150) after waiting for 900 seconds

fail 1481452 2017-08-04 04:24:00 2017-08-06 06:50:40 2017-08-06 09:34:41 2:44:01 1:22:26 1:21:35 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/ec-rados-default.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

"2017-08-06 08:19:56.467914 mon.b mon.0 172.21.15.92:6789/0 253 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 1481455 2017-08-04 04:24:01 2017-08-06 06:50:42 2017-08-06 07:22:40 0:31:58 0:16:12 0:15:46 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_api.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

Command failed (workunit test cls/test_cls_rgw.sh) on smithi025 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=jewel TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

fail 1481458 2017-08-04 04:24:02 2017-08-06 06:51:10 2017-08-06 08:33:02 1:41:52 1:24:41 0:17:11 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_loadgenbig.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

"2017-08-06 07:16:14.440550 mon.b mon.0 172.21.15.72:6789/0 70 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 1481461 2017-08-04 04:24:02 2017-08-06 06:51:03 2017-08-06 11:41:06 4:50:03 1:04:15 3:45:48 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_api.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

"2017-08-06 10:40:22.447881 mon.a mon.0 172.21.15.87:6789/0 187 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 1481464 2017-08-04 04:24:03 2017-08-06 06:51:06 2017-08-06 09:23:07 2:32:01 1:19:16 1:12:45 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_python.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

"2017-08-06 08:13:18.081324 mon.b mon.0 172.21.15.137:6789/0 194 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 1481467 2017-08-04 04:24:04 2017-08-06 06:51:17 2017-08-06 12:37:15 5:45:58 1:00:44 4:45:14 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

"2017-08-06 11:39:56.728063 mon.a mon.0 172.21.15.172:6789/0 69 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 1481470 2017-08-04 04:24:04 2017-08-06 06:52:23 2017-08-06 09:20:21 2:27:58 2:23:02 0:04:56 smithi master ubuntu 16.04 upgrade:jewel-x/stress-split/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/{radosbench.yaml rbd-cls.yaml rbd-import-export.yaml rbd_api.yaml readwrite.yaml snaps-few-objects.yaml} 5-finish-upgrade.yaml 6-luminous.yaml 7-final-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} distros/ubuntu_latest.yaml thrashosds-health.yaml} 3
fail 1481473 2017-08-04 04:24:05 2017-08-06 06:52:24 2017-08-06 08:08:22 1:15:58 1:09:50 0:06:08 smithi master ubuntu 16.04 upgrade:jewel-x/stress-split-erasure-code/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-workload/ec-rados-default.yaml 5-finish-upgrade.yaml 6-luminous.yaml 7-final-workload/ec-rados-plugin=jerasure-k=3-m=1.yaml distros/ubuntu_latest.yaml thrashosds-health.yaml} 3
Failure Reason:

timed out waiting for mon to be updated with osd.1: 0 < 85899346742

fail 1481475 2017-08-04 04:24:06 2017-08-06 06:52:49 2017-08-06 07:14:46 0:21:57 0:15:09 0:06:48 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/cache-pool-snaps.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

Command failed on smithi082 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 1481479 2017-08-04 04:24:07 2017-08-06 06:52:48 2017-08-06 08:20:49 1:28:01 1:21:53 0:06:08 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/ec-rados-default.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

"2017-08-06 07:04:45.813910 mon.b mon.0 172.21.15.15:6789/0 144 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 1481481 2017-08-04 04:24:07 2017-08-06 06:53:01 2017-08-06 08:30:52 1:37:51 0:10:14 1:27:37 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_api.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

Command failed on smithi075 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 1481484 2017-08-04 04:24:08 2017-08-06 06:53:58 2017-08-06 10:19:58 3:26:00 1:21:09 2:04:51 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_loadgenbig.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

"2017-08-06 09:04:30.351280 mon.a mon.0 172.21.15.183:6789/0 69 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 1481487 2017-08-04 04:24:09 2017-08-06 06:54:15 2017-08-06 08:59:57 2:05:42 1:18:35 0:47:07 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_api.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

"2017-08-06 07:50:40.138247 mon.b mon.0 172.21.15.24:6789/0 219 : cluster [WRN] Health check failed: 2 osds down (OSD_DOWN)" in cluster log

fail 1481490 2017-08-04 04:24:09 2017-08-06 06:54:18 2017-08-06 09:14:17 2:19:59 1:12:08 1:07:51 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_python.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

"2017-08-06 08:03:41.761996 mon.b mon.0 172.21.15.40:6789/0 109 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

dead 1481493 2017-08-04 04:24:10 2017-08-06 06:54:32 2017-08-06 18:59:51 12:05:19 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/blogbench.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
fail 1481496 2017-08-04 04:24:11 2017-08-06 06:54:55 2017-08-06 08:00:50 1:05:55 0:52:42 0:13:13 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/cache-pool-snaps.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

'wait_until_healthy' reached maximum tries (150) after waiting for 900 seconds

fail 1481499 2017-08-04 04:24:11 2017-08-06 06:54:55 2017-08-06 12:49:07 5:54:12 1:10:18 4:43:54 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/ec-rados-default.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

"2017-08-06 11:44:14.619322 mon.b mon.0 172.21.15.49:6789/0 236 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 1481502 2017-08-04 04:24:12 2017-08-06 06:55:11 2017-08-06 07:21:09 0:25:58 0:17:56 0:08:02 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_api.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

Command failed (workunit test cls/test_cls_rgw.sh) on smithi195 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=jewel TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh'

fail 1481506 2017-08-04 04:24:13 2017-08-06 06:55:12 2017-08-06 08:37:11 1:41:59 0:19:07 1:22:52 smithi master centos 7.3 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/rados_loadgenbig.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/centos_latest.yaml} 3
Failure Reason:

Command failed on smithi012 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'

fail 1481509 2017-08-04 04:24:13 2017-08-06 06:55:43 2017-08-06 08:07:44 1:12:01 1:05:02 0:06:59 smithi master ubuntu 14.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_api.yaml 3-upgrade-sequence/upgrade-all.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_14.04.yaml} 3
Failure Reason:

"2017-08-06 07:05:03.834405 mon.a mon.0 172.21.15.156:6789/0 128 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

fail 1481512 2017-08-04 04:24:14 2017-08-06 06:56:47 2017-08-06 08:14:48 1:18:01 0:16:09 1:01:52 smithi master ubuntu 16.04 upgrade:jewel-x/parallel/{0-cluster/{openstack.yaml start.yaml} 1-jewel-install/jewel.yaml 1.5-final-scrub.yaml 2-workload/test_rbd_python.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-luminous.yaml 5-final-workload/{blogbench.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_swift.yaml} distros/ubuntu_latest.yaml} 3
Failure Reason:

Command failed on smithi091 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'