Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
fail 7436836 2023-10-25 07:18:35 2023-10-25 07:19:59 2023-10-25 08:52:25 1:32:26 1:20:44 0:11:42 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

fail 7436837 2023-10-25 07:18:35 2023-10-25 07:19:59 2023-10-25 10:12:21 2:52:22 2:37:03 0:15:19 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

fail 7436838 2023-10-25 07:18:35 2023-10-25 07:21:50 2023-10-25 11:10:22 3:48:32 3:33:49 0:14:43 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

fail 7436839 2023-10-25 07:18:35 2023-10-25 07:25:11 2023-10-25 10:49:36 3:24:25 3:13:30 0:10:55 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

fail 7436840 2023-10-25 07:18:35 2023-10-25 07:25:12 2023-10-25 10:37:46 3:12:34 3:02:17 0:10:17 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

Command failed on smithi131 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1'

fail 7436841 2023-10-25 07:18:35 2023-10-25 07:25:12 2023-10-25 10:12:17 2:47:05 2:30:36 0:16:29 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

fail 7436842 2023-10-25 07:18:35 2023-10-25 07:29:34 2023-10-25 07:46:53 0:17:19 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

failed to install new kernel version within timeout

dead 7436843 2023-10-25 07:18:35 2023-10-25 07:30:28 2023-10-25 07:38:44 0:08:16 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

Error reimaging machines: list index out of range

fail 7436844 2023-10-25 07:18:35 2023-10-25 07:30:48 2023-10-25 09:38:10 2:07:22 1:54:21 0:13:01 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

fail 7436845 2023-10-25 07:18:35 2023-10-25 07:33:19 2023-10-25 10:32:04 2:58:45 2:47:03 0:11:42 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats'

fail 7436846 2023-10-25 07:18:35 2023-10-25 07:33:30 2023-10-25 10:10:30 2:37:00 2:25:42 0:11:18 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

Command failed on smithi163 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone --depth 1 --branch quincy https://github.com/Matan-B/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0'

fail 7436847 2023-10-25 07:18:35 2023-10-25 07:33:30 2023-10-25 08:58:27 1:24:57 1:14:12 0:10:45 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

fail 7436848 2023-10-25 07:18:35 2023-10-25 07:34:21 2023-10-25 13:04:53 5:30:32 5:19:17 0:11:15 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

Command failed on smithi006 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3'

fail 7436849 2023-10-25 07:18:35 2023-10-25 07:34:21 2023-10-25 10:39:36 3:05:15 2:52:37 0:12:38 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

fail 7436850 2023-10-25 07:18:36 2023-10-25 07:35:32 2023-10-25 09:10:17 1:34:45 1:23:11 0:11:34 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

fail 7436851 2023-10-25 07:18:36 2023-10-25 07:36:12 2023-10-25 11:39:25 4:03:13 3:50:59 0:12:14 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

wait_for_recovery: failed before timeout expired

fail 7436852 2023-10-25 07:18:36 2023-10-25 07:37:23 2023-10-25 10:48:44 3:11:21 3:00:01 0:11:20 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

reached maximum tries (501) after waiting for 3000 seconds

dead 7436853 2023-10-25 07:18:36 2023-10-25 07:38:24 2023-10-25 07:50:51 0:12:27 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

SSH connection to smithi191 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

fail 7436854 2023-10-25 07:18:36 2023-10-25 07:38:54 2023-10-25 09:45:49 2:06:55 1:56:06 0:10:49 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

Command failed on smithi099 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json'

fail 7436855 2023-10-25 07:18:36 2023-10-25 07:39:25 2023-10-25 09:40:14 2:00:49 1:48:39 0:12:10 smithi main ubuntu 20.04 upgrade:quincy-p2p:quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} 3
Failure Reason:

Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats'