| User | Scheduled | Started | Updated | Runtime | Suite | Branch | Machine Type | Revision | Pass | Fail | Dead |
|---|---|---|---|---|---|---|---|---|---|---|---|
| yuriw | 2023-10-19 16:23:25 | 2023-10-19 17:15:52 | 2023-10-20 16:27:48 | 23:11:56 | rados | wip-yuri10-testing-2023-10-11-0812 | smithi | 8a5df55 | 9 | 19 | 40 |
| Status | Job ID | Links | Posted | Started | Updated | Runtime |
Duration |
In Waiting |
Machine | Teuthology Branch | OS Type | OS Version | Description | Nodes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| dead | 7432432 |
|
2023-10-19 16:24:45 | 2023-10-19 17:15:52 | 2023-10-20 05:25:26 | 12:09:34 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432433 |
|
2023-10-19 16:24:46 | 2023-10-19 17:16:53 | 2023-10-19 17:39:05 | 0:22:12 | 0:11:26 | 0:10:46 | smithi | main | ubuntu | 22.04 | rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} | 1 |
Failure Reason:
Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' |
||||||||||||||
| fail | 7432434 |
|
2023-10-19 16:24:46 | 2023-10-19 17:16:53 | 2023-10-19 17:53:11 | 0:36:18 | 0:22:08 | 0:14:10 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} | 4 |
Failure Reason:
Command failed on smithi060 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' |
||||||||||||||
| dead | 7432435 |
|
2023-10-19 16:24:47 | 2023-10-19 17:19:34 | 2023-10-20 05:30:51 | 12:11:17 | smithi | main | ubuntu | 22.04 | rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432436 |
|
2023-10-19 16:24:48 | 2023-10-19 17:20:44 | 2023-10-19 17:59:55 | 0:39:11 | 0:28:12 | 0:10:59 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-snaps-balanced} | 2 |
Failure Reason:
Command failed on smithi132 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' |
||||||||||||||
| dead | 7432437 |
|
2023-10-19 16:24:49 | 2023-10-19 17:20:44 | 2023-10-20 05:29:47 | 12:09:03 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} | 3 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432438 |
|
2023-10-19 16:24:50 | 2023-10-19 17:20:55 | 2023-10-20 05:30:46 | 12:09:51 | smithi | main | ubuntu | 22.04 | rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7432439 |
|
2023-10-19 16:24:50 | 2023-10-19 17:21:15 | 2023-10-19 17:52:49 | 0:31:34 | 0:23:04 | 0:08:30 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects} | 2 |
| dead | 7432440 |
|
2023-10-19 16:24:51 | 2023-10-19 17:21:26 | 2023-10-20 05:33:31 | 12:12:05 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432441 |
|
2023-10-19 16:24:52 | 2023-10-19 17:24:36 | 2023-10-19 19:21:25 | 1:56:49 | 1:44:31 | 0:12:18 | smithi | main | centos | 9.stream | rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind} | 2 |
Failure Reason:
"2023-10-19T18:14:26.827312+0000 mon.a (mon.0) 363 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log |
||||||||||||||
| dead | 7432442 |
|
2023-10-19 16:24:53 | 2023-10-19 17:24:37 | 2023-10-20 05:35:17 | 12:10:40 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432443 |
|
2023-10-19 16:24:53 | 2023-10-19 17:24:57 | 2023-10-20 05:34:30 | 12:09:33 | smithi | main | ubuntu | 22.04 | rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/dedup-io-snaps} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432444 |
|
2023-10-19 16:24:54 | 2023-10-19 17:25:08 | 2023-10-20 05:34:40 | 12:09:32 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} | 3 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432445 |
|
2023-10-19 16:24:55 | 2023-10-19 17:25:08 | 2023-10-20 05:33:58 | 12:08:50 | smithi | main | ubuntu | 22.04 | rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/pool-snaps-few-objects} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432446 |
|
2023-10-19 16:24:56 | 2023-10-19 17:25:08 | 2023-10-20 05:35:14 | 12:10:06 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432447 |
|
2023-10-19 16:24:57 | 2023-10-19 17:26:09 | 2023-10-20 05:35:27 | 12:09:18 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432448 |
|
2023-10-19 16:24:58 | 2023-10-19 17:26:39 | 2023-10-20 05:36:57 | 12:10:18 | smithi | main | ubuntu | 22.04 | rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}} | 1 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432449 |
|
2023-10-19 16:24:58 | 2023-10-19 17:26:39 | 2023-10-19 21:47:25 | 4:20:46 | 4:11:52 | 0:08:54 | smithi | main | rhel | 8.6 | rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi050 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' |
||||||||||||||
| dead | 7432450 |
|
2023-10-19 16:24:59 | 2023-10-19 17:30:00 | 2023-10-20 05:39:20 | 12:09:20 | smithi | main | ubuntu | 22.04 | rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/snaps-few-objects} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432451 |
|
2023-10-19 16:25:00 | 2023-10-19 17:30:01 | 2023-10-20 05:39:24 | 12:09:23 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432452 |
|
2023-10-19 16:25:01 | 2023-10-19 17:30:11 | 2023-10-20 05:39:27 | 12:09:16 | smithi | main | centos | 9.stream | rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432453 |
|
2023-10-19 16:25:02 | 2023-10-19 17:30:21 | 2023-10-20 05:39:11 | 12:08:50 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432454 |
|
2023-10-19 16:25:02 | 2023-10-19 17:30:32 | 2023-10-19 18:23:41 | 0:53:09 | 0:44:09 | 0:09:00 | smithi | main | centos | 9.stream | rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} | 2 |
Failure Reason:
saw valgrind issues |
||||||||||||||
| dead | 7432455 |
|
2023-10-19 16:25:03 | 2023-10-19 17:30:42 | 2023-10-20 05:44:51 | 12:14:09 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} | 3 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432456 |
|
2023-10-19 16:25:04 | 2023-10-19 17:36:13 | 2023-10-20 05:46:29 | 12:10:16 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432457 |
|
2023-10-19 16:25:05 | 2023-10-19 17:36:34 | 2023-10-20 05:45:28 | 12:08:54 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432458 |
|
2023-10-19 16:25:06 | 2023-10-19 17:36:34 | 2023-10-20 05:48:25 | 12:11:51 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/small-objects-balanced} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432459 |
|
2023-10-19 16:25:06 | 2023-10-19 17:38:55 | 2023-10-20 06:01:31 | 12:22:36 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432460 |
|
2023-10-19 16:25:07 | 2023-10-20 02:43:02 | 2023-10-20 03:16:00 | 0:32:58 | 0:19:14 | 0:13:44 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} | 4 |
Failure Reason:
Command failed on smithi049 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' |
||||||||||||||
| fail | 7432461 |
|
2023-10-19 16:25:08 | 2023-10-20 02:43:22 | 2023-10-20 03:23:13 | 0:39:51 | 0:29:11 | 0:10:40 | smithi | main | centos | 8.stream | rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/e2e} | 2 |
Failure Reason:
Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8a5df55d22d3638066184c241442ff00fdc0ebdd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' |
||||||||||||||
| pass | 7432462 |
|
2023-10-19 16:25:09 | 2023-10-20 02:43:33 | 2023-10-20 03:16:49 | 0:33:16 | 0:23:03 | 0:10:13 | smithi | main | ubuntu | 22.04 | rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/small-objects} | 2 |
| dead | 7432463 |
|
2023-10-19 16:25:10 | 2023-10-20 02:45:03 | 2023-10-20 14:55:45 | 12:10:42 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} | 3 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432464 |
|
2023-10-19 16:25:11 | 2023-10-20 02:46:04 | 2023-10-20 14:59:12 | 12:13:08 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432465 |
|
2023-10-19 16:25:11 | 2023-10-20 02:50:35 | 2023-10-20 03:18:11 | 0:27:36 | 0:17:34 | 0:10:02 | smithi | main | ubuntu | 22.04 | rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}} | 1 |
Failure Reason:
Command failed on smithi131 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'mkdir $TESTDIR/kvtest && cd $TESTDIR/kvtest && ceph_test_keyvaluedb'" |
||||||||||||||
| fail | 7432466 |
|
2023-10-19 16:25:12 | 2023-10-20 02:50:35 | 2023-10-20 03:09:46 | 0:19:11 | 0:09:31 | 0:09:40 | smithi | main | centos | 9.stream | rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_latest}} | 1 |
Failure Reason:
Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_lazy_omap_stats' |
||||||||||||||
| dead | 7432467 |
|
2023-10-19 16:25:13 | 2023-10-20 02:51:06 | 2023-10-20 15:01:02 | 12:09:56 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432468 |
|
2023-10-19 16:25:14 | 2023-10-20 02:52:06 | 2023-10-20 15:01:01 | 12:08:55 | smithi | main | ubuntu | 22.04 | rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/snaps-few-objects-localized} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432469 |
|
2023-10-19 16:25:14 | 2023-10-20 02:52:07 | 2023-10-20 03:20:05 | 0:27:58 | 0:18:15 | 0:09:43 | smithi | main | centos | 9.stream | rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest} | 1 |
Failure Reason:
saw valgrind issues |
||||||||||||||
| fail | 7432470 |
|
2023-10-19 16:25:15 | 2023-10-20 02:53:27 | 2023-10-20 03:36:28 | 0:43:01 | 0:33:39 | 0:09:22 | smithi | main | centos | 9.stream | rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind} | 2 |
Failure Reason:
saw valgrind issues |
||||||||||||||
| dead | 7432471 |
|
2023-10-19 16:25:16 | 2023-10-20 02:53:27 | 2023-10-20 15:04:10 | 12:10:43 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432472 |
|
2023-10-19 16:25:17 | 2023-10-20 02:54:58 | 2023-10-20 03:34:34 | 0:39:36 | 0:21:47 | 0:17:49 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} | 4 |
Failure Reason:
Command failed on smithi081 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' |
||||||||||||||
| dead | 7432473 |
|
2023-10-19 16:25:17 | 2023-10-20 03:03:19 | 2023-10-20 15:17:44 | 12:14:25 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432474 |
|
2023-10-19 16:25:18 | 2023-10-20 03:09:51 | 2023-10-20 03:44:15 | 0:34:24 | 0:20:59 | 0:13:25 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} | 3 |
Failure Reason:
Command failed on smithi037 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' |
||||||||||||||
| dead | 7432475 |
|
2023-10-19 16:25:19 | 2023-10-20 03:11:51 | 2023-10-20 15:22:26 | 12:10:35 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432476 |
|
2023-10-19 16:25:20 | 2023-10-20 03:12:22 | 2023-10-20 03:48:00 | 0:35:38 | 0:22:03 | 0:13:35 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} | 4 |
Failure Reason:
Command failed on smithi033 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' |
||||||||||||||
| pass | 7432477 |
|
2023-10-19 16:25:21 | 2023-10-20 03:14:23 | 2023-10-20 03:47:06 | 0:32:43 | 0:16:26 | 0:16:17 | smithi | main | centos | 8.stream | rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait} | 2 |
| pass | 7432478 |
|
2023-10-19 16:25:21 | 2023-10-20 03:16:03 | 2023-10-20 03:47:01 | 0:30:58 | 0:16:32 | 0:14:26 | smithi | main | centos | 8.stream | rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/classic start} | 2 |
| dead | 7432479 |
|
2023-10-19 16:25:22 | 2023-10-20 03:16:03 | 2023-10-20 15:25:54 | 12:09:51 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7432480 |
|
2023-10-19 16:25:23 | 2023-10-20 03:16:54 | 2023-10-20 03:52:52 | 0:35:58 | 0:24:00 | 0:11:58 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy} | 2 |
| dead | 7432481 |
|
2023-10-19 16:25:24 | 2023-10-20 03:17:44 | 2023-10-20 15:29:27 | 12:11:43 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} | 3 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432482 |
|
2023-10-19 16:25:24 | 2023-10-20 03:18:15 | 2023-10-20 15:30:56 | 12:12:41 | smithi | main | ubuntu | 22.04 | rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}} | 1 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432483 |
|
2023-10-19 16:25:25 | 2023-10-20 03:20:15 | 2023-10-20 07:51:06 | 4:30:51 | 4:13:27 | 0:17:24 | smithi | main | centos | 8.stream | rados/upgrade/parallel/{0-random-distro$/{centos_8.stream_container_tools} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
Command failed (workunit test cls/test_cls_2pc_queue.sh) on smithi133 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh' |
||||||||||||||
| dead | 7432484 |
|
2023-10-19 16:25:26 | 2023-10-20 03:23:16 | 2023-10-20 15:43:14 | 12:19:58 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432485 |
|
2023-10-19 16:25:27 | 2023-10-20 03:34:38 | 2023-10-20 15:42:31 | 12:07:53 | smithi | main | centos | 9.stream | rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-small-objects-fast-read} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432486 |
|
2023-10-19 16:25:28 | 2023-10-20 03:34:38 | 2023-10-20 15:44:06 | 12:09:28 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| fail | 7432487 |
|
2023-10-19 16:25:28 | 2023-10-20 03:34:39 | 2023-10-20 04:16:36 | 0:41:57 | 0:21:44 | 0:20:13 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} | 4 |
Failure Reason:
Command failed on smithi037 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' |
||||||||||||||
| fail | 7432488 |
|
2023-10-19 16:25:29 | 2023-10-20 03:44:20 | 2023-10-20 05:42:18 | 1:57:58 | 1:46:38 | 0:11:20 | smithi | main | centos | 9.stream | rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind} | 2 |
Failure Reason:
saw valgrind issues |
||||||||||||||
| dead | 7432489 |
|
2023-10-19 16:25:30 | 2023-10-20 03:47:11 | 2023-10-20 15:55:38 | 12:08:27 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-snaps} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7432490 |
|
2023-10-19 16:25:31 | 2023-10-20 03:47:11 | 2023-10-20 04:08:53 | 0:21:42 | 0:12:27 | 0:09:15 | smithi | main | centos | 9.stream | rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/readwrite} | 2 |
| dead | 7432491 |
|
2023-10-19 16:25:31 | 2023-10-20 03:48:02 | 2023-10-20 15:56:35 | 12:08:33 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{default} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} | 3 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432492 |
|
2023-10-19 16:25:32 | 2023-10-20 03:48:02 | 2023-10-20 16:00:38 | 12:12:36 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7432493 |
|
2023-10-19 16:25:33 | 2023-10-20 03:51:03 | 2023-10-20 04:18:17 | 0:27:14 | 0:14:11 | 0:13:03 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/dedup-io-snaps} | 2 |
| pass | 7432494 |
|
2023-10-19 16:25:34 | 2023-10-20 03:52:54 | 2023-10-20 04:34:26 | 0:41:32 | 0:12:13 | 0:29:19 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} | 4 |
| dead | 7432495 |
|
2023-10-19 16:25:34 | 2023-10-20 04:13:27 | 2023-10-20 16:23:20 | 12:09:53 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| dead | 7432496 |
|
2023-10-19 16:25:35 | 2023-10-20 04:14:37 | 2023-10-20 16:25:37 | 12:11:00 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7432497 |
|
2023-10-19 16:25:36 | 2023-10-20 04:16:38 | 2023-10-20 04:50:46 | 0:34:08 | 0:23:11 | 0:10:57 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-small-objects} | 2 |
| fail | 7432498 |
|
2023-10-19 16:25:37 | 2023-10-20 04:16:38 | 2023-10-20 05:07:43 | 0:51:05 | 0:30:52 | 0:20:13 | smithi | main | centos | 8.stream | rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} tasks/e2e} | 2 |
Failure Reason:
Command failed (workunit test cephadm/test_dashboard_e2e.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8a5df55d22d3638066184c241442ff00fdc0ebdd TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' |
||||||||||||||
| dead | 7432499 |
|
2023-10-19 16:25:38 | 2023-10-20 04:18:19 | 2023-10-20 16:27:48 | 12:09:29 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/minsize_recovery thrashosds-health workloads/ec-snaps-few-objects-overwrites} | 2 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
Pulpito