Pulpito

rzarzynski-2024-06-03_22:14:06-rados-wip-osd-ec-partial-reads-eio-distro-default-smithi

ID
Status
Ceph Branch
Suite Branch
Teuthology Branch
Machine
OS
Nodes
Description
Failure Reason
7740338
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi063 smithi119
rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-flag}
"2024-06-03T22:46:36.162629+0000 mon.smithi063 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log
7740339
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi059 smithi138
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/valgrind}
valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun
7740340
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
ubuntu 22.04
smithi070 smithi183
rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
"2024-06-03T22:38:23.446166+0000 mon.a (mon.0) 102 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log
7740341
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi007
rados/standalone/{supported-random-distro$/{centos_latest} workloads/scrub}
Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c821233d097eb6eb4287bbd1d0b6d01638e5f90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh'
7740342
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
ubuntu 22.04
smithi038 smithi116
rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-wait}
"2024-06-03T22:38:56.169152+0000 mon.smithi038 (mon.0) 118 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log
7740343
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
ubuntu 22.04
smithi161
rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/none msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}
Command failed (workunit test cephtool/test.sh) on smithi161 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c821233d097eb6eb4287bbd1d0b6d01638e5f90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
7740344
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 8.stream
smithi005 smithi107 smithi148
rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/reef backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/cache-snaps}
Command failed on smithi005 with status 1: 'sudo yum install -y kernel'
7740345
DEAD
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi097 smithi137
rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
hit max job timeout
7740346
DEAD
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi152 smithi177
rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/off fixed-2 mon_election/classic start}
Error reimaging machines: Failed to power on smithi177
7740347
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 8.stream
smithi029 smithi062 smithi078
rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/radosbench}
Command failed on smithi078 with status 1: 'sudo yum install -y kernel'
7740348
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi069 smithi087
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}
Command failed (workunit test rados/test.sh) on smithi069 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c821233d097eb6eb4287bbd1d0b6d01638e5f90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
7740349
DEAD
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
ubuntu 22.04
smithi018 smithi191
rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/rm-zap-add}
Error reimaging machines: Failed to power on smithi018
7740350
DEAD
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi170 smithi194
rados/cephadm/smoke/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
Error reimaging machines: Failed to power on smithi194
7740351
DEAD
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi045 smithi046 smithi107 smithi148
rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites}
hit max job timeout
7740352
DEAD
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
ubuntu 22.04
smithi005
rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/erasure-code}
Error reimaging machines: Failed to power on smithi005
7740353
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi037 smithi040
rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-flag}
"2024-06-03T23:04:14.234373+0000 mon.smithi037 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log
7740354
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
ubuntu 22.04
smithi145
rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest}}
Command failed (workunit test cephtool/test.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c821233d097eb6eb4287bbd1d0b6d01638e5f90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
7740355
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi043 smithi053
rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/rm-zap-wait}
"2024-06-03T23:15:57.728326+0000 mon.smithi043 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log
7740356
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
ubuntu 22.04
smithi016 smithi033
rados/cephadm/smoke/{0-distro/ubuntu_22.04 0-nvme-loop agent/off fixed-2 mon_election/classic start}
"2024-06-03T22:54:35.982718+0000 mon.a (mon.0) 101 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log
7740357
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi188
rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest}
valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun
7740358
DEAD
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi079 smithi176
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind}
Error reimaging machines: Failed to power on smithi079
7740359
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi002 smithi029
rados/cephadm/smoke/{0-distro/centos_9.stream 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
Command failed on smithi002 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d208bbce2efc11fb0dcecbb271cb2051d1daa58 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 47727c94-21ff-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/nvme4n1'
7740360
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
ubuntu 22.04
smithi062 smithi078 smithi094 smithi155
rados/thrash-erasure-code-crush-4-nodes/{arch/x86_64 ceph mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=8-m=6-crush}
Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0'
7740361
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi027 smithi115
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind}
valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun
7740362
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 8.stream
smithi050 smithi072 smithi142
rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/pacific backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/cache-snaps}
Command failed on smithi142 with status 1: 'sudo yum install -y kernel'
7740363
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi133
rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_latest}}
Command failed (workunit test cephtool/test.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c821233d097eb6eb4287bbd1d0b6d01638e5f90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
7740364
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 8.stream
smithi073 smithi122 smithi144
rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/radosbench}
Command failed on smithi122 with status 1: 'sudo yum install -y kernel'
7740365
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi173 smithi178
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}
Command failed (workunit test rados/test.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c821233d097eb6eb4287bbd1d0b6d01638e5f90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
7740366
FAIL
wip-osd-ec-partial-reads-eio
wip-osd-ec-partial-reads-eio-fix66321
main
smithi
centos 9.stream
smithi070 smithi176
rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add}
"2024-06-03T23:26:48.643244+0000 mon.smithi070 (mon.0) 121 : cluster [WRN] Health check failed: failed to probe daemons or devices (CEPHADM_REFRESH_FAILED)" in cluster log