Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 5552287 2020-10-23 22:00:27 2020-10-23 22:01:17 2020-10-23 22:39:17 0:38:00 0:25:10 0:12:50 smithi master ubuntu upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

timeout expired in wait_until_healthy

fail 5552288 2020-10-23 22:00:28 2020-10-23 22:02:48 2020-10-23 22:04:47 0:01:59 0 smithi master ubuntu upgrade/mimic-x/parallel/mon_election/classic
Failure Reason:

list index out of range

fail 5552289 2020-10-23 22:00:29 2020-10-23 22:02:48 2020-10-23 22:04:47 0:01:59 0 smithi master ubuntu upgrade/mimic-x-singleton/mon_election/classic
Failure Reason:

list index out of range

fail 5552290 2020-10-23 22:00:30 2020-10-23 22:02:48 2020-10-23 22:40:48 0:38:00 0:19:51 0:18:09 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552291 2020-10-23 22:00:31 2020-10-23 22:02:49 2020-10-24 04:28:56 6:26:07 4:48:18 1:37:49 smithi master ubuntu 18.04 upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/classic thrashosds-health ubuntu_latest} 4
Failure Reason:

"2020-10-24T01:30:52.456445+0000 mon.c (mon.0) 15 : cluster [WRN] Health check failed: There are daemons running an older version of ceph (DAEMON_OLD_VERSION)" in cluster log

fail 5552292 2020-10-23 22:00:33 2020-10-23 22:03:18 2020-10-23 22:45:18 0:42:00 0:19:26 0:22:34 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552293 2020-10-23 22:00:34 2020-10-23 22:03:47 2020-10-23 22:39:47 0:36:00 0:18:51 0:17:09 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552294 2020-10-23 22:00:35 2020-10-23 22:05:08 2020-10-23 23:11:09 1:06:01 0:22:29 0:43:32 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552295 2020-10-23 22:00:36 2020-10-23 22:05:08 2020-10-23 22:39:08 0:34:00 0:20:03 0:13:57 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552296 2020-10-23 22:00:40 2020-10-23 22:05:08 2020-10-23 23:29:09 1:24:01 1:11:24 0:12:37 smithi master ubuntu 18.04 upgrade/cephfs/volumes/import-legacy/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-verify} ubuntu_18.04} 2
Failure Reason:

timeout expired in wait_until_healthy

fail 5552297 2020-10-23 22:00:41 2020-10-23 22:05:18 2020-10-24 02:01:22 3:56:04 3:04:39 0:51:25 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} 5
Failure Reason:

"2020-10-24T00:38:35.352444+0000 mon.c (mon.0) 25 : cluster [WRN] Health check failed: There are daemons running an older version of ceph (DAEMON_OLD_VERSION)" in cluster log

fail 5552298 2020-10-23 22:00:42 2020-10-23 22:05:40 2020-10-23 22:07:39 0:01:59 0 smithi master ubuntu upgrade/mimic-x/stress-split/mon_election/classic
Failure Reason:

list index out of range

fail 5552299 2020-10-23 22:00:44 2020-10-23 22:06:05 2020-10-24 01:12:08 3:06:03 0:19:41 2:46:22 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552300 2020-10-23 22:00:45 2020-10-23 22:06:08 2020-10-23 22:52:08 0:46:00 0:18:24 0:27:36 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552301 2020-10-23 22:00:47 2020-10-23 22:06:59 2020-10-23 22:48:59 0:42:00 0:21:28 0:20:32 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552302 2020-10-23 22:00:48 2020-10-23 22:07:17 2020-10-23 22:43:17 0:36:00 0:19:09 0:16:51 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552303 2020-10-23 22:00:51 2020-10-23 22:07:40 2020-10-23 22:31:39 0:23:59 0:13:28 0:10:31 smithi master ubuntu upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

timeout expired in wait_until_healthy

fail 5552304 2020-10-23 22:00:52 2020-10-23 22:07:45 2020-10-24 00:01:46 1:54:01 0:18:55 1:35:06 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552305 2020-10-23 22:00:54 2020-10-23 22:07:57 2020-10-23 22:09:56 0:01:59 0 smithi master ubuntu upgrade/mimic-x/stress-split-erasure-code/mon_election/classic
Failure Reason:

list index out of range

fail 5552306 2020-10-23 22:00:55 2020-10-23 22:08:44 2020-10-23 22:46:43 0:37:59 0:19:10 0:18:49 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552307 2020-10-23 22:00:56 2020-10-23 22:08:44 2020-10-23 23:18:44 1:10:00 0:20:10 0:49:50 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552308 2020-10-23 22:00:57 2020-10-23 22:08:44 2020-10-23 23:18:44 1:10:00 0:18:17 0:51:43 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552309 2020-10-23 22:00:59 2020-10-23 22:08:58 2020-10-23 22:42:58 0:34:00 0:23:34 0:10:26 smithi master ubuntu upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}} 3
Failure Reason:

timeout expired in wait_until_healthy

fail 5552310 2020-10-23 22:01:00 2020-10-23 22:09:08 2020-10-23 23:31:09 1:22:01 0:18:24 1:03:37 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552311 2020-10-23 22:01:01 2020-10-23 22:10:30 2020-10-23 23:48:31 1:38:01 0:18:47 1:19:14 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552312 2020-10-23 22:01:02 2020-10-23 22:10:30 2020-10-23 22:12:30 0:02:00 0 smithi master ubuntu upgrade/mimic-x/parallel/mon_election/connectivity
Failure Reason:

list index out of range

fail 5552313 2020-10-23 22:01:03 2020-10-23 22:10:30 2020-10-23 22:12:30 0:02:00 0 smithi master ubuntu upgrade/mimic-x-singleton/mon_election/connectivity
Failure Reason:

list index out of range

fail 5552314 2020-10-23 22:01:04 2020-10-23 22:13:07 2020-10-23 22:47:07 0:34:00 0:20:26 0:13:34 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552315 2020-10-23 22:01:05 2020-10-23 22:13:07 2020-10-24 03:29:14 5:16:07 4:54:23 0:21:44 smithi master ubuntu 18.04 upgrade/nautilus-x-singleton/{0-cluster/{openstack start} 1-install/nautilus 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap mon_election/connectivity thrashosds-health ubuntu_latest} 4
Failure Reason:

"2020-10-23T22:44:25.948655+0000 mon.b (mon.0) 7 : cluster [WRN] Health check failed: There are daemons running an older version of ceph (DAEMON_OLD_VERSION)" in cluster log

fail 5552316 2020-10-23 22:01:06 2020-10-23 22:13:54 2020-10-24 04:44:01 6:30:07 5:13:19 1:16:48 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} 5
Failure Reason:

"2020-10-24T01:24:52.797582+0000 mon.c (mon.0) 22 : cluster [WRN] Health check failed: There are daemons running an older version of ceph (DAEMON_OLD_VERSION)" in cluster log

fail 5552317 2020-10-23 22:01:07 2020-10-23 22:13:59 2020-10-24 02:34:04 4:20:05 3:09:03 1:11:02 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} 5
Failure Reason:

"2020-10-23T23:57:37.377259+0000 mon.c (mon.0) 20 : cluster [WRN] Health check failed: There are daemons running an older version of ceph (DAEMON_OLD_VERSION)" in cluster log

fail 5552318 2020-10-23 22:01:08 2020-10-23 22:15:44 2020-10-23 22:51:44 0:36:00 0:23:57 0:12:03 smithi master ubuntu upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

timeout expired in wait_until_healthy

fail 5552319 2020-10-23 22:01:09 2020-10-23 22:15:44 2020-10-23 22:47:44 0:32:00 0:19:09 0:12:51 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552320 2020-10-23 22:01:10 2020-10-23 22:15:47 2020-10-23 22:53:47 0:38:00 0:20:17 0:17:43 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552321 2020-10-23 22:01:13 2020-10-23 22:15:53 2020-10-24 03:33:59 5:18:06 3:20:51 1:57:15 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/filestore-xfs thrashosds-health ubuntu_18.04} 5
Failure Reason:

"2020-10-24T02:12:20.860763+0000 mon.c (mon.0) 34 : cluster [WRN] Health check failed: There are daemons running an older version of ceph (DAEMON_OLD_VERSION)" in cluster log

fail 5552322 2020-10-23 22:01:15 2020-10-23 22:16:05 2020-10-23 22:18:05 0:02:00 0 smithi master ubuntu upgrade/mimic-x/stress-split/mon_election/connectivity
Failure Reason:

list index out of range

fail 5552323 2020-10-23 22:01:16 2020-10-23 22:16:32 2020-10-24 03:22:38 5:06:06 1:53:05 3:13:01 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} 5
Failure Reason:

"2020-10-24T02:00:00.021554+0000 mon.c (mon.0) 29 : cluster [WRN] Health check failed: There are daemons running an older version of ceph (DAEMON_OLD_VERSION)" in cluster log

fail 5552324 2020-10-23 22:01:17 2020-10-23 22:17:44 2020-10-23 22:43:44 0:26:00 0:14:32 0:11:28 smithi master ubuntu upgrade/cephfs/featureful_client/upgraded_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

timeout expired in wait_until_healthy

fail 5552325 2020-10-23 22:01:19 2020-10-23 22:17:53 2020-10-23 23:13:53 0:56:00 0:22:17 0:33:43 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552326 2020-10-23 22:01:20 2020-10-23 22:18:06 2020-10-23 23:16:06 0:58:00 0:21:00 0:37:00 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/bluestore-bitmap ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552327 2020-10-23 22:01:21 2020-10-23 22:18:09 2020-10-23 23:00:08 0:41:59 0:18:36 0:23:23 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/connectivity objectstore/bluestore-bitmap thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552328 2020-10-23 22:01:23 2020-10-23 22:18:09 2020-10-23 23:46:10 1:28:01 0:19:13 1:08:48 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/classic thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552329 2020-10-23 22:01:24 2020-10-23 22:18:16 2020-10-23 22:20:16 0:02:00 0 smithi master ubuntu upgrade/mimic-x/stress-split-erasure-code/mon_election/connectivity
Failure Reason:

list index out of range

fail 5552330 2020-10-23 22:01:25 2020-10-23 22:20:30 2020-10-23 23:14:30 0:54:00 0:19:33 0:34:27 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/filestore-xfs ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552331 2020-10-23 22:01:26 2020-10-23 22:20:30 2020-10-23 23:00:30 0:40:00 0:24:21 0:15:39 smithi master ubuntu upgrade/cephfs/featureful_client/old_client/{bluestore-bitmap clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{frag_enable multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-nautilus 1-client 2-upgrade 3-compat_client/pacific}} 3
Failure Reason:

timeout expired in wait_until_healthy

fail 5552332 2020-10-23 22:01:28 2020-10-23 22:22:21 2020-10-23 23:20:22 0:58:01 0:22:26 0:35:35 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/short_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-mon-osd-mds 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/connectivity objectstore/bluestore-bitmap ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy

fail 5552333 2020-10-23 22:01:29 2020-10-23 22:22:21 2020-10-24 02:42:26 4:20:05 3:16:03 1:04:02 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{radosbench rbd-cls rbd-import-export rbd_api readwrite rgw_ragweed_prepare snaps-few-objects} 5-finish-upgrade 6-octopus 7-msgr2 8-final-workload/{rbd-python snaps-many-objects} mon_election/classic objectstore/filestore-xfs thrashosds-health ubuntu_18.04} 5
Failure Reason:

"2020-10-24T01:21:07.302204+0000 mon.c (mon.0) 24 : cluster [WRN] Health check failed: There are daemons running an older version of ceph (DAEMON_OLD_VERSION)" in cluster log

fail 5552334 2020-10-23 22:01:30 2020-10-23 22:22:22 2020-10-24 00:42:24 2:20:02 0:19:00 2:01:02 smithi master ubuntu 18.04 upgrade/nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/bluestore-bitmap 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload mon_election/connectivity thrashosds-health ubuntu_18.04} 5
Failure Reason:

timeout expired in wait_until_healthy

fail 5552335 2020-10-23 22:01:32 2020-10-23 22:22:21 2020-10-23 23:30:22 1:08:01 0:18:39 0:49:22 smithi master ubuntu 18.04 upgrade/nautilus-x/parallel/{0-cluster/{openstack start} 1-ceph-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-workload/{blogbench ec-rados-default rados_api rados_loadgenbig rgw_ragweed_prepare test_rbd_api test_rbd_python} 3-upgrade-sequence/upgrade-all 4-octopus 5-final-workload/{blogbench rados-snaps-few-objects rados_loadgenmix rados_mon_thrash rbd_cls rbd_import_export rgw rgw_ragweed_check} mon_election/classic objectstore/filestore-xfs ubuntu_18.04} 4
Failure Reason:

timeout expired in wait_until_healthy