User | Scheduled | Started | Updated | Runtime | Suite | Branch | Machine Type | Revision | Pass | Fail | Dead |
---|---|---|---|---|---|---|---|---|---|---|---|
cbodley | 2023-09-15 15:32:39 | 2023-09-15 15:33:40 | 2023-09-15 18:46:55 | 3:13:15 | rgw | wip-62507-quincy | smithi | 1006d97 | 14 | 40 | 3 |
Status | Job ID | Links | Posted | Started | Updated | Runtime |
Duration |
In Waiting |
Machine | Teuthology Branch | OS Type | OS Version | Description | Nodes |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
pass | 7397132 | 2023-09-15 15:32:44 | 2023-09-15 15:33:40 | 2023-09-15 16:21:45 | 0:48:05 | 0:34:22 | 0:13:43 | smithi | main | centos | 8.stream | rgw/cloud-transition/{cluster overrides supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests} | 1 | |
fail | 7397133 | 2023-09-15 15:32:45 | 2023-09-15 15:33:40 | 2023-09-15 16:00:48 | 0:27:08 | 0:18:34 | 0:08:34 | smithi | main | rhel | 8.4 | rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} | 1 | |
Failure Reason:
Command failed on smithi137 with status 1: "cd /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate && openstack project create --os-username admin --os-password ADMIN --os-user-domain-id default --os-project-name admin --os-project-domain-id default --os-identity-api-version 3 --os-auth-url http://smithi137.front.sepia.ceph.com:35357/v3 --description 'Encryption Tenant' --domain default rgwcrypt --debug" |
||||||||||||||
fail | 7397134 | 2023-09-15 15:32:46 | 2023-09-15 15:33:40 | 2023-09-15 15:51:10 | 0:17:30 | 0:07:45 | 0:09:45 | smithi | main | rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} | 2 | |||
Failure Reason:
Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1006d97f49aa663071195d74ae384013f088f080 |
||||||||||||||
pass | 7397135 | 2023-09-15 15:32:47 | 2023-09-15 15:33:41 | 2023-09-15 16:12:55 | 0:39:14 | 0:31:20 | 0:07:54 | smithi | main | rhel | 8.4 | rgw/lifecycle/{cluster overrides supported-random-distro$/{rhel_8} tasks/rgw_s3tests} | 1 | |
pass | 7397136 | 2023-09-15 15:32:48 | 2023-09-15 15:33:41 | 2023-09-15 15:58:10 | 0:24:29 | 0:11:41 | 0:12:48 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} | 2 | |
fail | 7397137 | 2023-09-15 15:32:49 | 2023-09-15 15:36:02 | 2023-09-15 16:29:15 | 0:53:13 | 0:44:44 | 0:08:29 | smithi | main | rhel | 8.4 | rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} | 2 | |
Failure Reason:
rgw multisite test failures |
||||||||||||||
fail | 7397138 | 2023-09-15 15:32:49 | 2023-09-15 15:36:02 | 2023-09-15 16:25:56 | 0:49:54 | 0:35:36 | 0:14:18 | smithi | main | centos | 8.stream | rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} | 2 | |
Failure Reason:
Command failed (bucket notification tests against different endpoints) on smithi026 with status 1: "BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a '!kafka_test,!amqp_test,!amqp_ssl_test,!modification_required,!manual_test'" |
||||||||||||||
pass | 7397139 | 2023-09-15 15:32:50 | 2023-09-15 15:41:23 | 2023-09-15 16:09:11 | 0:27:48 | 0:12:15 | 0:15:33 | smithi | main | ubuntu | 20.04 | rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} | 2 | |
pass | 7397140 | 2023-09-15 15:32:51 | 2023-09-15 15:43:44 | 2023-09-15 16:29:05 | 0:45:21 | 0:33:19 | 0:12:02 | smithi | main | rhel | 8.4 | rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{rhel_8} tasks/{0-install first ststests}} | 2 | |
fail | 7397141 | 2023-09-15 15:32:52 | 2023-09-15 15:45:34 | 2023-09-15 16:21:36 | 0:36:02 | 0:23:14 | 0:12:48 | smithi | main | ubuntu | 20.04 | rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} | 1 | |
Failure Reason:
Command failed on smithi072 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' |
||||||||||||||
pass | 7397142 | 2023-09-15 15:32:52 | 2023-09-15 15:45:45 | 2023-09-15 16:16:05 | 0:30:20 | 0:17:43 | 0:12:37 | smithi | main | ubuntu | 20.04 | rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} | 2 | |
pass | 7397143 | 2023-09-15 15:32:53 | 2023-09-15 15:47:05 | 2023-09-15 18:46:55 | 2:59:50 | 2:48:33 | 0:11:17 | smithi | main | centos | 8.stream | rgw/tools/{centos_latest cluster tasks} | 1 | |
fail | 7397144 | 2023-09-15 15:32:54 | 2023-09-15 15:47:05 | 2023-09-15 16:08:40 | 0:21:35 | 0:11:10 | 0:10:25 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} | 2 | |
Failure Reason:
Command failed on smithi079 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397145 | 2023-09-15 15:32:55 | 2023-09-15 15:47:26 | 2023-09-15 16:15:42 | 0:28:16 | 0:18:28 | 0:09:48 | smithi | main | ubuntu | 20.04 | rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} | 2 | |
Failure Reason:
Command failed (s3 tests against rgw) on smithi096 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3'" |
||||||||||||||
fail | 7397146 | 2023-09-15 15:32:55 | 2023-09-15 15:47:36 | 2023-09-15 16:26:32 | 0:38:56 | 0:26:38 | 0:12:18 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} | 2 | |
Failure Reason:
Command failed (workunit test rgw/run-reshard.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a15d831b2226304428ef0b1c91bf45a715fc16e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh' |
||||||||||||||
fail | 7397147 | 2023-09-15 15:32:56 | 2023-09-15 15:49:17 | 2023-09-15 16:09:59 | 0:20:42 | 0:07:57 | 0:12:45 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} | 2 | |
Failure Reason:
Command failed on smithi103 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397148 | 2023-09-15 15:32:57 | 2023-09-15 15:51:07 | 2023-09-15 16:09:10 | 0:18:03 | 0:08:00 | 0:10:03 | smithi | main | ubuntu | 20.04 | rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} | 2 | |
Failure Reason:
Command failed on smithi002 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397149 | 2023-09-15 15:32:58 | 2023-09-15 15:51:18 | 2023-09-15 16:14:31 | 0:23:13 | 0:10:57 | 0:12:16 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} | 2 | |
Failure Reason:
Command failed on smithi172 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397150 | 2023-09-15 15:32:59 | 2023-09-15 15:53:08 | 2023-09-15 16:24:09 | 0:31:01 | 0:15:30 | 0:15:31 | smithi | main | ubuntu | 20.04 | rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} | 1 | |
Failure Reason:
Command failed (s3 tests against rgw) on smithi044 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3'" |
||||||||||||||
fail | 7397151 | 2023-09-15 15:32:59 | 2023-09-15 15:58:19 | 2023-09-15 16:21:19 | 0:23:00 | 0:12:30 | 0:10:30 | smithi | main | rhel | 8.4 | rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} | 2 | |
Failure Reason:
Command failed on smithi084 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397152 | 2023-09-15 15:33:00 | 2023-09-15 15:59:50 | 2023-09-15 16:35:53 | 0:36:03 | 0:25:53 | 0:10:10 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} | 2 | |
Failure Reason:
Command failed (workunit test rgw/run-reshard.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a15d831b2226304428ef0b1c91bf45a715fc16e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh' |
||||||||||||||
pass | 7397153 | 2023-09-15 15:33:01 | 2023-09-15 16:00:51 | 2023-09-15 16:22:48 | 0:21:57 | 0:10:37 | 0:11:20 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_ragweed ubuntu_latest} | 2 | |
fail | 7397154 | 2023-09-15 15:33:02 | 2023-09-15 16:02:41 | 2023-09-15 16:38:39 | 0:35:58 | 0:19:45 | 0:16:13 | smithi | main | ubuntu | 20.04 | rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} | 2 | |
Failure Reason:
Command failed (s3 tests against rgw) on smithi079 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3'" |
||||||||||||||
fail | 7397155 | 2023-09-15 15:33:02 | 2023-09-15 16:08:42 | 2023-09-15 16:30:12 | 0:21:30 | 0:12:04 | 0:09:26 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} | 2 | |
Failure Reason:
Command failed on smithi002 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397156 | 2023-09-15 15:33:03 | 2023-09-15 16:09:13 | 2023-09-15 16:29:39 | 0:20:26 | 0:08:00 | 0:12:26 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_s3tests ubuntu_latest} | 2 | |
Failure Reason:
Command failed on smithi049 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397157 | 2023-09-15 15:33:04 | 2023-09-15 16:09:13 | 2023-09-15 16:45:33 | 0:36:20 | 0:25:16 | 0:11:04 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} | 2 | |
Failure Reason:
Command failed (workunit test rgw/run-reshard.sh) on smithi103 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a15d831b2226304428ef0b1c91bf45a715fc16e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh' |
||||||||||||||
fail | 7397158 | 2023-09-15 15:33:05 | 2023-09-15 16:10:04 | 2023-09-15 16:40:18 | 0:30:14 | 0:16:55 | 0:13:19 | smithi | main | centos | 8.stream | rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} | 1 | |
Failure Reason:
Command failed (s3 tests against rgw) on smithi061 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3'" |
||||||||||||||
pass | 7397159 | 2023-09-15 15:33:05 | 2023-09-15 16:13:04 | 2023-09-15 16:39:50 | 0:26:46 | 0:15:00 | 0:11:46 | smithi | main | centos | 8.stream | rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} | 2 | |
fail | 7397160 | 2023-09-15 15:33:06 | 2023-09-15 16:14:35 | 2023-09-15 16:33:47 | 0:19:12 | 0:07:59 | 0:11:13 | smithi | main | ubuntu | 20.04 | rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} | 2 | |
Failure Reason:
Command failed on smithi096 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397161 | 2023-09-15 15:33:07 | 2023-09-15 16:15:45 | 2023-09-15 16:37:58 | 0:22:13 | 0:11:05 | 0:11:08 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} | 2 | |
Failure Reason:
Command failed on smithi090 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
pass | 7397162 | 2023-09-15 15:33:08 | 2023-09-15 16:16:16 | 2023-09-15 16:41:08 | 0:24:52 | 0:10:45 | 0:14:07 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_user_quota ubuntu_latest} | 2 | |
fail | 7397163 | 2023-09-15 15:33:09 | 2023-09-15 16:21:07 | 2023-09-15 16:56:43 | 0:35:36 | 0:25:30 | 0:10:06 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} | 2 | |
Failure Reason:
Command failed (workunit test rgw/run-reshard.sh) on smithi084 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a15d831b2226304428ef0b1c91bf45a715fc16e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh' |
||||||||||||||
fail | 7397164 | 2023-09-15 15:33:09 | 2023-09-15 16:21:27 | 2023-09-15 16:49:14 | 0:27:47 | 0:18:01 | 0:09:46 | smithi | main | rhel | 8.4 | rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} | 1 | |
Failure Reason:
Command failed (s3 tests against rgw) on smithi072 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3'" |
||||||||||||||
fail | 7397165 | 2023-09-15 15:33:10 | 2023-09-15 16:21:37 | 2023-09-15 16:41:43 | 0:20:06 | 0:08:01 | 0:12:05 | smithi | main | rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} | 2 | |||
Failure Reason:
Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=1006d97f49aa663071195d74ae384013f088f080 |
||||||||||||||
fail | 7397166 | 2023-09-15 15:33:11 | 2023-09-15 16:22:58 | 2023-09-15 16:42:24 | 0:19:26 | 0:08:07 | 0:11:19 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_bucket_quota ubuntu_latest} | 2 | |
Failure Reason:
Command failed on smithi044 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
pass | 7397167 | 2023-09-15 15:33:12 | 2023-09-15 16:24:18 | 2023-09-15 17:15:09 | 0:50:51 | 0:40:22 | 0:10:29 | smithi | main | rhel | 8.4 | rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} | 2 | |
dead | 7397168 | 2023-09-15 15:33:12 | 2023-09-15 16:25:59 | 2023-09-15 16:32:28 | 0:06:29 | smithi | main | rhel | 8.4 | rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} | 2 | |||
Failure Reason:
Error reimaging machines: Expected smithi055's OS to be rhel 8.4 but found ubuntu 20.04 |
||||||||||||||
dead | 7397169 | 2023-09-15 15:33:13 | 2023-09-15 16:26:19 | 2023-09-15 16:34:28 | 0:08:09 | smithi | main | ubuntu | 20.04 | rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} | 2 | |||
Failure Reason:
SSH connection to smithi055 was lost: 'sudo apt-get update' |
||||||||||||||
fail | 7397170 | 2023-09-15 15:33:14 | 2023-09-15 16:26:20 | 2023-09-15 17:03:35 | 0:37:15 | 0:26:35 | 0:10:40 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} | 2 | |
Failure Reason:
Command failed (workunit test rgw/run-reshard.sh) on smithi046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a15d831b2226304428ef0b1c91bf45a715fc16e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh' |
||||||||||||||
fail | 7397171 | 2023-09-15 15:33:15 | 2023-09-15 16:26:20 | 2023-09-15 16:47:42 | 0:21:22 | 0:11:07 | 0:10:15 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} | 2 | |
Failure Reason:
Command failed on smithi156 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
pass | 7397172 | 2023-09-15 15:33:16 | 2023-09-15 16:26:40 | 2023-09-15 16:51:35 | 0:24:55 | 0:11:13 | 0:13:42 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_multipart_upload ubuntu_latest} | 2 | |
pass | 7397173 | 2023-09-15 15:33:16 | 2023-09-15 16:29:11 | 2023-09-15 16:53:32 | 0:24:21 | 0:13:53 | 0:10:28 | smithi | main | ubuntu | 20.04 | rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} | 2 | |
fail | 7397174 | 2023-09-15 15:33:17 | 2023-09-15 16:29:22 | 2023-09-15 17:05:57 | 0:36:35 | 0:25:30 | 0:11:05 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} | 2 | |
Failure Reason:
Command failed (workunit test rgw/run-reshard.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a15d831b2226304428ef0b1c91bf45a715fc16e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh' |
||||||||||||||
fail | 7397175 | 2023-09-15 15:33:18 | 2023-09-15 16:29:42 | 2023-09-15 16:56:04 | 0:26:22 | 0:14:28 | 0:11:54 | smithi | main | ubuntu | 20.04 | rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} | 1 | |
Failure Reason:
Command failed (s3 tests against rgw) on smithi002 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3'" |
||||||||||||||
dead | 7397176 | 2023-09-15 15:33:19 | 2023-09-15 16:30:22 | 2023-09-15 16:58:10 | 0:27:48 | smithi | main | ubuntu | 20.04 | rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} | 2 | |||
Failure Reason:
SSH connection to smithi055 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' |
||||||||||||||
fail | 7397177 | 2023-09-15 15:33:20 | 2023-09-15 16:32:43 | 2023-09-15 16:54:51 | 0:22:08 | 0:11:13 | 0:10:55 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} | 2 | |
Failure Reason:
Command failed on smithi096 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397178 | 2023-09-15 15:33:20 | 2023-09-15 16:33:54 | 2023-09-15 16:52:18 | 0:18:24 | 0:07:56 | 0:10:28 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_ragweed ubuntu_latest} | 2 | |
Failure Reason:
Command failed on smithi110 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397179 | 2023-09-15 15:33:21 | 2023-09-15 16:34:34 | 2023-09-15 16:42:43 | 0:08:09 | smithi | main | ubuntu | 20.04 | rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} | 2 | |||
Failure Reason:
Command failed on smithi055 with status 100: 'sudo apt-get clean' |
||||||||||||||
fail | 7397180 | 2023-09-15 15:33:22 | 2023-09-15 16:34:54 | 2023-09-15 17:08:59 | 0:34:05 | 0:24:32 | 0:09:33 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} | 2 | |
Failure Reason:
Command failed (workunit test rgw/run-reshard.sh) on smithi137 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a15d831b2226304428ef0b1c91bf45a715fc16e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh' |
||||||||||||||
fail | 7397181 | 2023-09-15 15:33:23 | 2023-09-15 16:36:05 | 2023-09-15 17:02:40 | 0:26:35 | 0:15:46 | 0:10:49 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_s3tests ubuntu_latest} | 2 | |
Failure Reason:
Command failed (s3 tests against rgw) on smithi067 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3'" |
||||||||||||||
fail | 7397182 | 2023-09-15 15:33:24 | 2023-09-15 16:36:25 | 2023-09-15 16:59:41 | 0:23:16 | 0:11:09 | 0:12:07 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} | 2 | |
Failure Reason:
Command failed on smithi090 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397183 | 2023-09-15 15:33:25 | 2023-09-15 16:38:06 | 2023-09-15 17:07:13 | 0:29:07 | 0:18:26 | 0:10:41 | smithi | main | centos | 8.stream | rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} | 1 | |
Failure Reason:
Command failed (s3 tests against rgw) on smithi057 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain'" |
||||||||||||||
fail | 7397184 | 2023-09-15 15:33:25 | 2023-09-15 16:38:06 | 2023-09-15 16:56:27 | 0:18:21 | 0:08:01 | 0:10:20 | smithi | main | ubuntu | 20.04 | rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} | 2 | |
Failure Reason:
Command failed on smithi018 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
pass | 7397185 | 2023-09-15 15:33:26 | 2023-09-15 16:38:27 | 2023-09-15 17:02:44 | 0:24:17 | 0:14:07 | 0:10:10 | smithi | main | ubuntu | 20.04 | rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} | 2 | |
fail | 7397186 | 2023-09-15 15:33:27 | 2023-09-15 16:38:47 | 2023-09-15 17:17:25 | 0:38:38 | 0:26:09 | 0:12:29 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} | 2 | |
Failure Reason:
Command failed (workunit test rgw/run-reshard.sh) on smithi172 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7a15d831b2226304428ef0b1c91bf45a715fc16e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh' |
||||||||||||||
fail | 7397187 | 2023-09-15 15:33:28 | 2023-09-15 16:39:57 | 2023-09-15 16:59:32 | 0:19:35 | 0:08:03 | 0:11:32 | smithi | main | ubuntu | 20.04 | rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_user_quota ubuntu_latest} | 2 | |
Failure Reason:
Command failed on smithi120 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |
||||||||||||||
fail | 7397188 | 2023-09-15 15:33:28 | 2023-09-15 16:41:18 | 2023-09-15 17:07:04 | 0:25:46 | 0:12:13 | 0:13:33 | smithi | main | centos | 8.stream | rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} | 2 | |
Failure Reason:
Command failed on smithi023 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap' |