mrfreezeex-2022-04-08_23:03:08-rbd-wip-mrfreezeex-multipeer-distro-default-smithi

6783019

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi093 smithi101

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-bitmap policy/none supported-random-distro$/{centos_8}}

"2022-04-08T23:45:55.888458+0000 mon.a (mon.0) 218 : cluster [WRN] Health check failed: Degraded data redundancy: 6/2862 objects degraded (0.210%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783020

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi145 smithi152 smithi167

rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{centos_8}}

Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi167 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

6783038

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi123 smithi138

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-hybrid policy/none supported-random-distro$/{centos_8}}

"2022-04-09T00:06:43.022408+0000 mon.a (mon.0) 206 : cluster [WRN] Health check failed: Degraded data redundancy: 1/2154 objects degraded (0.046%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783060

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi184 smithi201

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}}

"2022-04-09T00:32:12.154287+0000 mon.a (mon.0) 203 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1856 objects degraded (0.054%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783080

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi037 smithi043

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-hybrid policy/simple supported-random-distro$/{centos_8}}

"2022-04-09T01:19:03.689900+0000 mon.a (mon.0) 1458 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

6783082

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi087 smithi092 smithi163

rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-stupid policy/simple supported-random-distro$/{centos_8}}

Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi163 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

6783084

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi026 smithi036

rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-bootstrap-workunit} msgr-failures/few objectstore/bluestore-low-osd-mem-target supported-random-distro$/{centos_8}}

Found coredumps on ubuntu@smithi036.front.sepia.ceph.com

6783088

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi174 smithi178

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-minimum} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{centos_8}}

"2022-04-09T01:00:03.948010+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783090

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi125 smithi157

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-zstd policy/simple supported-random-distro$/{centos_8}}

"2022-04-09T01:02:48.461115+0000 mon.a (mon.0) 209 : cluster [WRN] Health check failed: Degraded data redundancy: 2/2450 objects degraded (0.082%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783101

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

ubuntu 20.04

smithi002 smithi081

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{ubuntu_latest}}

"2022-04-09T04:23:36.409559+0000 mon.a (mon.0) 205 : cluster [WRN] Health check failed: Degraded data redundancy: 6/2044 objects degraded (0.294%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783106

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi124 smithi173

rbd/mirror/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{mirror-extra} workloads/rbd-mirror-snapshot-bootstrap-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 supported-random-distro$/{centos_8}}

"2022-04-09T04:27:06.643700+0000 mon.a (mon.0) 93 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783110

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

ubuntu 20.04

smithi104 smithi120

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{ubuntu_latest}}

"2022-04-09T04:30:13.555857+0000 mon.a (mon.0) 239 : cluster [WRN] Health check failed: Degraded data redundancy: 3/2598 objects degraded (0.115%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783112

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi005 smithi049 smithi178

rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-lz4 policy/none supported-random-distro$/{centos_8}}

Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi178 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

6783126

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

ubuntu 20.04

smithi017 smithi082

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{ubuntu_latest}}

"2022-04-09T04:40:11.377961+0000 mon.a (mon.0) 203 : cluster [WRN] Health check failed: Degraded data redundancy: 1/334 objects degraded (0.299%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783147

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

ubuntu 20.04

smithi005 smithi178

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/filestore-xfs policy/simple supported-random-distro$/{ubuntu_latest}}

Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi178 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh'

6783151

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi029 smithi043

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-lz4 policy/simple supported-random-distro$/{centos_8}}

"2022-04-09T05:10:17.638690+0000 mon.a (mon.0) 211 : cluster [WRN] Health check failed: Degraded data redundancy: 9/5898 objects degraded (0.153%), 2 pgs degraded (PG_DEGRADED)" in cluster log

6783153

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

ubuntu 20.04

smithi002 smithi026 smithi081

rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-snappy policy/none supported-random-distro$/{ubuntu_latest}}

Found coredumps on ubuntu@smithi081.front.sepia.ceph.com

6783159

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi136

rbd/valgrind/{base/install centos_latest clusters/{fixed-1 openstack} objectstore/bluestore-comp-lz4 validator/memcheck workloads/rbd_mirror}

"2022-04-09T05:47:23.758564+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log

6783163

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi066 smithi080 smithi189

rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-bitmap policy/simple supported-random-distro$/{centos_8}}

Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi189 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

6783184

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi055 smithi101

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-fsx-workunit} msgr-failures/few objectstore/bluestore-comp-zstd policy/none supported-random-distro$/{centos_8}}

"2022-04-09T05:39:59.888611+0000 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Degraded data redundancy: 8/3138 objects degraded (0.255%), 1 pg degraded (PG_DEGRADED)" in cluster log

6783196

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

centos 8.stream

smithi083 smithi156 smithi170

rbd/mirror-thrash/{base/{openstack} cluster/3-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-journal-workunit-multipeer} msgr-failures/few objectstore/bluestore-comp-zlib policy/none supported-random-distro$/{centos_8}}

Command failed (workunit test rbd/rbd_mirror_journal.sh) on smithi170 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=69ad68d989715754945f23c59c7f3abe9a8b52dc TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 USE_CLUSTER3=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_journal.sh'

6783204

FAIL

wip-mrfreezeex-multipeer

wip-mrfreezeex-multipeer

master

smithi

ubuntu 20.04

smithi026 smithi081

rbd/mirror-thrash/{base/{openstack} cluster/2-nodes/{base/{install mirror roles} extra/{rbd-mirror} workloads/rbd-mirror-snapshot-stress-workunit-fast-diff} msgr-failures/few objectstore/bluestore-stupid policy/none supported-random-distro$/{ubuntu_latest}}

reached maximum tries (90) after waiting for 540 seconds