User | Scheduled | Started | Updated | Runtime | Suite | Branch | Machine Type | Revision | Pass | Fail | Dead |
---|---|---|---|---|---|---|---|---|---|---|---|
vshankar | 2022-01-03 12:27:45 | 2022-01-03 12:28:38 | 2022-01-03 20:10:34 | 7:41:56 | fs | wip-vshankar-testing-20220103-142738 | smithi | cbf7034 | 30 | 7 | 18 |
Status | Job ID | Links | Posted | Started | Updated | Runtime |
Duration |
In Waiting |
Machine | Teuthology Branch | OS Type | OS Version | Description | Nodes |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
fail | 6591940 | 2022-01-03 12:27:51 | 2022-01-03 12:28:37 | 2022-01-03 13:09:25 | 0:40:48 | 0:13:27 | 0:27:21 | smithi | master | centos | 8.3 | fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/centos_latest} | 2 | |
Failure Reason:
Command failed on smithi191 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:cbf70343937294a91acf495031c37c1bd818afb5 pull' |
||||||||||||||
pass | 6591941 | 2022-01-03 12:27:52 | 2022-01-03 12:28:37 | 2022-01-03 15:16:51 | 2:48:14 | 2:23:59 | 0:24:15 | smithi | master | centos | 8.3 | fs/valgrind/{begin centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} | 1 | |
pass | 6591942 | 2022-01-03 12:27:53 | 2022-01-03 12:28:37 | 2022-01-03 13:01:18 | 0:32:41 | 0:16:02 | 0:16:39 | smithi | master | ubuntu | 20.04 | fs/verify/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/lockdep} | 2 | |
dead | 6591943 | 2022-01-03 12:27:53 | 2022-01-03 12:28:38 | 2022-01-03 12:42:29 | 0:13:51 | 0:05:34 | 0:08:17 | smithi | master | rhel | 8.4 | fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} | 2 | |
Failure Reason:
{'smithi078.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi085.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591944 | 2022-01-03 12:27:54 | 2022-01-03 12:28:38 | 2022-01-03 13:03:15 | 0:34:37 | 0:23:39 | 0:10:58 | smithi | master | centos | 8.3 | fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} | 2 | |
pass | 6591945 | 2022-01-03 12:27:55 | 2022-01-03 12:28:38 | 2022-01-03 15:16:09 | 2:47:31 | 2:30:23 | 0:17:08 | smithi | master | ubuntu | 20.04 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} | 2 | |
dead | 6591946 | 2022-01-03 12:27:56 | 2022-01-03 12:28:39 | 2022-01-03 13:01:15 | 0:32:36 | 0:06:53 | 0:25:43 | smithi | master | rhel | 8.4 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{no}} | 3 | |
Failure Reason:
{'smithi122.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi146.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi137.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
dead | 6591947 | 2022-01-03 12:27:57 | 2022-01-03 12:28:39 | 2022-01-03 12:43:16 | 0:14:37 | 0:05:33 | 0:09:04 | smithi | master | rhel | 8.4 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} | 3 | |
Failure Reason:
{'smithi079.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi168.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi145.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591948 | 2022-01-03 12:27:57 | 2022-01-03 12:28:39 | 2022-01-03 13:11:17 | 0:42:38 | 0:15:53 | 0:26:45 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} | 2 | |
dead | 6591949 | 2022-01-03 12:27:58 | 2022-01-03 12:28:40 | 2022-01-03 12:43:49 | 0:15:09 | 0:05:41 | 0:09:28 | smithi | master | rhel | 8.4 | fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} | 2 | |
Failure Reason:
{'smithi023.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi032.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591950 | 2022-01-03 12:27:59 | 2022-01-03 12:28:40 | 2022-01-03 13:31:17 | 1:02:37 | 0:35:57 | 0:26:40 | smithi | master | centos | 8.stream | fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} | 2 | |
dead | 6591951 | 2022-01-03 12:28:00 | 2022-01-03 12:28:40 | 2022-01-03 12:36:46 | 0:08:06 | smithi | master | centos | 8.3 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} | 3 | |||
Failure Reason:
SSH connection to smithi065 was lost: 'rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.16.0_rc3_ceph_g1ddee0bc2048-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://1.chacra.ceph.com/r/kernel/testing/1ddee0bc204871a92d257c3548b3926bd1f06520/centos/8/flavors/default/x86_64/ --input-file=-' |
||||||||||||||
dead | 6591952 | 2022-01-03 12:28:00 | 2022-01-03 12:28:41 | 2022-01-03 12:28:43 | 0:00:02 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} | 2 | |||
Failure Reason:
Error reimaging machines: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/273/task |
||||||||||||||
pass | 6591953 | 2022-01-03 12:28:01 | 2022-01-03 12:28:41 | 2022-01-03 13:14:36 | 0:45:55 | 0:21:20 | 0:24:35 | smithi | master | centos | 8.3 | fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}} | 2 | |
pass | 6591954 | 2022-01-03 12:28:02 | 2022-01-03 12:28:41 | 2022-01-03 13:08:09 | 0:39:28 | 0:16:36 | 0:22:52 | smithi | master | centos | 8.3 | fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry} | 2 | |
pass | 6591955 | 2022-01-03 12:28:03 | 2022-01-03 12:28:42 | 2022-01-03 13:52:12 | 1:23:30 | 1:00:52 | 0:22:38 | smithi | master | centos | 8.stream | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} | 3 | |
dead | 6591956 | 2022-01-03 12:28:04 | 2022-01-03 12:28:42 | 2022-01-03 12:43:14 | 0:14:32 | 0:05:35 | 0:08:57 | smithi | master | rhel | 8.4 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} | 2 | |
Failure Reason:
{'smithi040.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi140.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
dead | 6591957 | 2022-01-03 12:28:04 | 2022-01-03 12:28:42 | 2022-01-03 12:45:09 | 0:16:27 | 0:05:38 | 0:10:49 | smithi | master | rhel | 8.4 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} | 3 | |
Failure Reason:
{'smithi073.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi062.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi129.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591958 | 2022-01-03 12:28:05 | 2022-01-03 12:28:43 | 2022-01-03 14:21:11 | 1:52:28 | 1:25:48 | 0:26:40 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} | 2 | |
pass | 6591959 | 2022-01-03 12:28:06 | 2022-01-03 12:28:43 | 2022-01-03 13:12:17 | 0:43:34 | 0:19:18 | 0:24:16 | smithi | master | centos | 8.3 | fs/32bits/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} | 2 | |
fail | 6591960 | 2022-01-03 12:28:07 | 2022-01-03 12:28:43 | 2022-01-03 12:36:47 | 0:08:04 | smithi | master | centos | 8.stream | fs/permission/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} | 2 | |||
Failure Reason:
Command failed on smithi065 with status 1: 'sudo yum upgrade -y linux-firmware' |
||||||||||||||
pass | 6591961 | 2022-01-03 12:28:07 | 2022-01-03 12:28:44 | 2022-01-03 13:11:55 | 0:43:11 | 0:18:19 | 0:24:52 | smithi | master | centos | 8.3 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} | 3 | |
pass | 6591962 | 2022-01-03 12:28:08 | 2022-01-03 12:28:44 | 2022-01-03 13:11:49 | 0:43:05 | 0:15:59 | 0:27:06 | smithi | master | centos | 8.stream | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} | 2 | |
dead | 6591963 | 2022-01-03 12:28:09 | 2022-01-03 12:28:45 | 2022-01-03 12:43:51 | 0:15:06 | 0:05:43 | 0:09:23 | smithi | master | rhel | 8.4 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} | 2 | |
Failure Reason:
{'smithi121.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi152.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591964 | 2022-01-03 12:28:10 | 2022-01-03 12:28:45 | 2022-01-03 13:19:52 | 0:51:07 | 0:24:29 | 0:26:38 | smithi | master | centos | 8.3 | fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} | 3 | |
pass | 6591965 | 2022-01-03 12:28:11 | 2022-01-03 12:28:45 | 2022-01-03 13:16:39 | 0:47:54 | 0:22:39 | 0:25:15 | smithi | master | centos | 8.3 | fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} | 2 | |
dead | 6591966 | 2022-01-03 12:28:11 | 2022-01-03 12:28:46 | 2022-01-03 12:45:22 | 0:16:36 | 0:05:39 | 0:10:57 | smithi | master | rhel | 8.4 | fs/snaps/{begin clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} | 2 | |
Failure Reason:
{'smithi074.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi156.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591967 | 2022-01-03 12:28:12 | 2022-01-03 12:28:46 | 2022-01-03 13:50:51 | 1:22:05 | 0:55:33 | 0:26:32 | smithi | master | centos | 8.stream | fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} | 2 | |
dead | 6591968 | 2022-01-03 12:28:13 | 2022-01-03 12:28:46 | 2022-01-03 12:42:30 | 0:13:44 | 0:05:32 | 0:08:12 | smithi | master | rhel | 8.4 | fs/volumes/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} | 2 | |
Failure Reason:
{'smithi123.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi160.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591969 | 2022-01-03 12:28:14 | 2022-01-03 12:28:47 | 2022-01-03 13:13:48 | 0:45:01 | 0:18:19 | 0:26:42 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} | 2 | |
dead | 6591970 | 2022-01-03 12:28:15 | 2022-01-03 12:28:47 | 2022-01-03 12:43:38 | 0:14:51 | 0:05:40 | 0:09:11 | smithi | master | rhel | 8.4 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} | 3 | |
Failure Reason:
{'smithi086.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi112.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi183.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
dead | 6591971 | 2022-01-03 12:28:15 | 2022-01-03 12:28:47 | 2022-01-03 12:44:08 | 0:15:21 | 0:05:42 | 0:09:39 | smithi | master | rhel | 8.4 | fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} | 2 | |
Failure Reason:
{'smithi049.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi059.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591972 | 2022-01-03 12:28:16 | 2022-01-03 12:28:48 | 2022-01-03 13:16:15 | 0:47:27 | 0:25:21 | 0:22:06 | smithi | master | centos | 8.3 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} | 3 | |
dead | 6591973 | 2022-01-03 12:28:17 | 2022-01-03 12:28:48 | 2022-01-03 12:43:11 | 0:14:23 | 0:05:36 | 0:08:47 | smithi | master | rhel | 8.4 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} | 2 | |
Failure Reason:
{'smithi096.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi187.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591974 | 2022-01-03 12:28:18 | 2022-01-03 12:28:48 | 2022-01-03 13:12:09 | 0:43:21 | 0:16:15 | 0:27:06 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} | 2 | |
pass | 6591975 | 2022-01-03 12:28:18 | 2022-01-03 12:28:49 | 2022-01-03 13:12:34 | 0:43:45 | 0:16:01 | 0:27:44 | smithi | master | centos | 8.3 | fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} | 2 | |
dead | 6591976 | 2022-01-03 12:28:19 | 2022-01-03 12:28:49 | 2022-01-03 13:02:44 | 0:33:55 | 0:06:55 | 0:27:00 | smithi | master | rhel | 8.4 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} | 3 | |
Failure Reason:
{'smithi158.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi164.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi179.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591977 | 2022-01-03 12:28:20 | 2022-01-03 12:28:49 | 2022-01-03 13:12:24 | 0:43:35 | 0:18:22 | 0:25:13 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} | 2 | |
fail | 6591978 | 2022-01-03 12:28:21 | 2022-01-03 12:28:50 | 2022-01-03 13:31:59 | 1:03:09 | 0:36:31 | 0:26:38 | smithi | master | centos | 8.3 | fs/thrash/multifs/{begin clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} | 2 | |
Failure Reason:
Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi107 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cbf70343937294a91acf495031c37c1bd818afb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git-ceph.sh' |
||||||||||||||
pass | 6591979 | 2022-01-03 12:28:22 | 2022-01-03 12:28:50 | 2022-01-03 13:29:34 | 1:00:44 | 0:35:26 | 0:25:18 | smithi | master | centos | 8.stream | fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} | 2 | |
fail | 6591980 | 2022-01-03 12:28:22 | 2022-01-03 12:28:50 | 2022-01-03 15:18:46 | 2:49:56 | 2:23:15 | 0:26:41 | smithi | master | centos | 8.stream | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} | 2 | |
Failure Reason:
Found coredumps on ubuntu@smithi117.front.sepia.ceph.com |
||||||||||||||
fail | 6591981 | 2022-01-03 12:28:23 | 2022-01-03 12:28:51 | 2022-01-03 13:15:24 | 0:46:33 | 0:30:32 | 0:16:01 | smithi | master | ubuntu | 20.04 | fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} | 2 | |
Failure Reason:
"2022-01-03T12:55:43.365052+0000 mon.a (mon.0) 441 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log |
||||||||||||||
dead | 6591982 | 2022-01-03 12:28:24 | 2022-01-03 12:28:51 | 2022-01-03 12:39:17 | 0:10:26 | 0:05:35 | 0:04:51 | smithi | master | rhel | 8.4 | fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} | 2 | |
Failure Reason:
{'smithi002.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi017.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591983 | 2022-01-03 12:28:25 | 2022-01-03 12:28:51 | 2022-01-03 13:04:32 | 0:35:41 | 0:19:49 | 0:15:52 | smithi | master | ubuntu | 20.04 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} | 2 | |
pass | 6591984 | 2022-01-03 12:28:25 | 2022-01-03 12:28:52 | 2022-01-03 14:55:21 | 2:26:29 | 1:59:39 | 0:26:50 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} | 2 | |
pass | 6591985 | 2022-01-03 12:28:26 | 2022-01-03 12:28:52 | 2022-01-03 13:42:34 | 1:13:42 | 1:01:06 | 0:12:36 | smithi | master | ubuntu | 20.04 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} | 3 | |
dead | 6591986 | 2022-01-03 12:28:27 | 2022-01-03 12:28:52 | 2022-01-03 13:03:09 | 0:34:17 | 0:06:56 | 0:27:21 | smithi | master | rhel | 8.4 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{rhel_8} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} | 3 | |
Failure Reason:
{'smithi172.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi018.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi038.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591987 | 2022-01-03 12:28:28 | 2022-01-03 12:29:33 | 2022-01-03 13:50:29 | 1:20:56 | 0:56:33 | 0:24:23 | smithi | master | centos | 8.3 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} | 3 | |
fail | 6591988 | 2022-01-03 12:28:28 | 2022-01-03 12:29:53 | 2022-01-03 16:27:23 | 3:57:30 | 3:32:26 | 0:25:04 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} | 2 | |
Failure Reason:
"2022-01-03T13:47:35.653484+0000 mds.a (mds.2) 1 : cluster [WRN] client.4610 isn't responding to mclientcaps(revoke), ino 0x300000007ac pending pAsLsXsFscr issued pAsLsXsFsxcrwb, sent 300.008911 seconds ago" in cluster log |
||||||||||||||
pass | 6591989 | 2022-01-03 12:28:29 | 2022-01-03 12:30:03 | 2022-01-03 15:23:43 | 2:53:40 | 2:26:44 | 0:26:56 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} | 2 | |
pass | 6591990 | 2022-01-03 12:28:30 | 2022-01-03 12:30:14 | 2022-01-03 13:29:39 | 0:59:25 | 0:33:02 | 0:26:23 | smithi | master | centos | 8.stream | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8.stream} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} | 2 | |
dead | 6591991 | 2022-01-03 12:28:31 | 2022-01-03 12:30:14 | 2022-01-03 12:45:28 | 0:15:14 | 0:05:42 | 0:09:32 | smithi | master | rhel | 8.4 | fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8.stream} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} | 3 | |
Failure Reason:
{'smithi055.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi155.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}, 'smithi063.front.sepia.ceph.com': {'attempts': 12, 'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}} |
||||||||||||||
pass | 6591992 | 2022-01-03 12:28:31 | 2022-01-03 12:31:55 | 2022-01-03 13:16:15 | 0:44:20 | 0:19:37 | 0:24:43 | smithi | master | centos | 8.3 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} | 2 | |
fail | 6591993 | 2022-01-03 12:28:32 | 2022-01-03 12:36:56 | 2022-01-03 20:10:34 | 7:33:38 | 7:19:30 | 0:14:08 | smithi | master | ubuntu | 20.04 | fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} | 2 | |
Failure Reason:
Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi061 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cbf70343937294a91acf495031c37c1bd818afb5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' |
||||||||||||||
pass | 6591994 | 2022-01-03 12:28:33 | 2022-01-03 12:38:07 | 2022-01-03 13:32:17 | 0:54:10 | 0:35:00 | 0:19:10 | smithi | master | centos | 8.stream | fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} | 2 |