wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/thrash/{0-distro/centos_8.2_container_tools_3.0 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects}
"2021-11-25T14:45:42.585181+0000 mgr.y (mgr.4107) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-lz4} supported-random-distro$/{rhel_8} tasks/prometheus}
Test failure: test_standby (tasks.mgr.test_prometheus.TestPrometheus)
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-balanced}
"2021-11-25T14:45:31.024543+0000 mgr.y (mgr.4107) 45 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T14:38:32.769634+0000 mgr.x (mgr.4101) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-localized}
"2021-11-25T14:45:45.363412+0000 mgr.x (mgr.4108) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/off mon_election/connectivity task/test_cephadm}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/pg-removal-interruption mon_election/classic msgr-failures/none msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects}
"2021-11-25T14:38:36.219418+0000 mgr.y (mgr.4106) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/client-keyring 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/write_fadvise_dontneed}
"2021-11-25T14:49:53.632575+0000 mgr.y (mgr.4103) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/admin_socket_objecter_requests}
"2021-11-25T14:40:57.113651+0000 mgr.y (mgr.4098) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
"2021-11-25T14:49:54.006267+0000 mgr.x (mgr.4102) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/repave-all}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/osd_stale_reads mon_election/connectivity rados supported-random-distro$/{centos_8}}
"2021-11-25T15:00:29.349852+0000 mgr.x (mgr.4102) 44 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/standalone/{supported-random-distro$/{centos_8} workloads/scrub}
Command failed (workunit test scrub/osd-scrub-repair.sh) on smithi173 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-agent-big}
"2021-11-25T15:05:18.855119+0000 mgr.y (mgr.4100) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/on fixed-2 mon_election/classic start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/radostool mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}}
"2021-11-25T15:06:53.927821+0000 mgr.x (mgr.4101) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T15:10:25.049425+0000 mgr.x (mgr.4102) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/cache-agent-small}
"2021-11-25T15:06:03.248645+0000 mgr.x (mgr.4107) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
Command failed (workunit test cephtool/test.sh) on smithi143 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/iscsi 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/cache-pool-snaps-readproxy}
"2021-11-25T15:13:43.274103+0000 mgr.x (mgr.4102) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/objectstore/{backends/objectstore-filestore-memstore supported-random-distro$/{ubuntu_latest}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/rados_api_tests}
Command failed (workunit test rados/test.sh) on smithi097 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps}
"2021-11-25T15:12:23.383061+0000 mgr.x (mgr.4105) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/thrash/{0-distro/centos_8.3_container_tools_3.0 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write}
"2021-11-25T15:10:06.263629+0000 mgr.x (mgr.4100) 29 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/one workloads/rados_api_tests}
"2021-11-25T15:21:01.245742+0000 mgr.x (mgr.4109) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced}
"2021-11-25T15:14:09.960775+0000 mgr.x (mgr.4112) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-radosbench}
"2021-11-25T15:25:25.179637+0000 mgr.y (mgr.4101) 47 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/pool-access mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T15:13:41.993065+0000 mgr.x (mgr.4103) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/mon_recovery validater/lockdep}
"2021-11-25T15:19:23.438545+0000 mgr.y (mgr.4108) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton/{all/random-eio mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}}
"2021-11-25T15:21:01.390455+0000 mgr.x (mgr.4102) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/cache-snaps}
"2021-11-25T15:22:02.006830+0000 mgr.y (mgr.4104) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/on mon_election/classic task/test_cephadm_repos}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{filestore-xfs} supported-random-distro$/{ubuntu_latest} tasks/workunits}
"2021-11-25T15:23:26.623861+0000 mgr.z (mgr.4108) 27 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.z: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/cache}
"2021-11-25T15:28:18.974566+0000 mgr.x (mgr.4108) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/mirror 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T15:32:23.230759+0000 mgr.x (mgr.4102) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/dedup-io-mixed}
"2021-11-25T15:32:02.775996+0000 mgr.y (mgr.4101) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/1-node k8s/1.21 net/flannel rook/1.7.0}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-add}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/dedup-io-snaps}
"2021-11-25T15:35:57.368073+0000 mgr.y (mgr.4102) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/connectivity rados supported-random-distro$/{centos_8}}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8}}
"2021-11-25T15:38:37.204717+0000 mgr.x (mgr.4109) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects}
"2021-11-25T15:44:22.250976+0000 mgr.y (mgr.4100) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/rados_api_tests}
Command failed (workunit test rados/test.sh) on smithi136 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_latest} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/cache-snaps}
Command failed on smithi081 with status 5: 'sudo systemctl stop ceph-8be646f4-4e07-11ec-8c2d-001a4aab830c@mon.a'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/with-work/{0-distro/centos_8.2_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/radosbench-high-concurrency}
"2021-11-25T15:55:33.743769+0000 mgr.y (mgr.4104) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/smoke-roleless/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/radosbench}
"2021-11-25T16:02:59.493345+0000 mgr.x (mgr.4102) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/recovery-preemption mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8}}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/defaut 3-start-upgrade 4-wait 5-upgrade-ls agent/on mon_election/classic}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/version-number-sanity mon_election/classic rados supported-random-distro$/{centos_8}}
"2021-11-25T15:57:09.047992+0000 mgr.x (mgr.4102) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-overwrites}
"2021-11-25T16:07:18.174436+0000 mgr.y (mgr.4104) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/redirect}
"2021-11-25T16:09:41.261692+0000 mgr.y (mgr.4104) 49 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests}
"2021-11-25T16:04:57.848945+0000 mgr.y (mgr.4104) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/sample_fio}
"2021-11-25T16:04:05.266082+0000 mgr.x (mgr.4100) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/multimon/{clusters/3 mon_election/classic msgr-failures/many msgr/async no_pools objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
"2021-11-25T16:16:40.772887+0000 mgr.x (mgr.4104) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-flag}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} tasks/rados_cls_all}
"2021-11-25T16:16:39.881442+0000 mgr.y (mgr.4101) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect_set_object}
"2021-11-25T16:16:35.443870+0000 mgr.y (mgr.4100) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/set-chunks-read}
"2021-11-25T16:19:35.198280+0000 mgr.x (mgr.4107) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T16:22:05.343325+0000 mgr.x (mgr.4102) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/resolve_stuck_peering mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T16:20:09.998448+0000 mgr.x (mgr.4101) 70 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/smoke-roleless/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects-balanced}
"2021-11-25T16:19:26.008568+0000 mgr.y (mgr.4104) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-lz4} tasks/dashboard}
Test failure: test_remove_from_blocklist (tasks.mgr.dashboard.test_auth.AuthTest)
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} supported-random-distro$/{ubuntu_latest} tasks/crash}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/objectstore/{backends/alloc-hint supported-random-distro$/{ubuntu_latest}}
"2021-11-25T16:21:44.609442+0000 mgr.x (mgr.4099) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/rest/{mgr-restful supported-random-distro$/{centos_8.stream}}
"2021-11-25T16:26:37.037727+0000 mgr.x (mgr.4102) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton-nomsgr/{all/admin_socket_output mon_election/classic rados supported-random-distro$/{rhel_8}}
"2021-11-25T16:32:48.572867+0000 mgr.x (mgr.4101) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/c2c}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/upgrade/parallel/{0-random-distro$/{centos_8.2_container_tools_3.0} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/valgrind-leaks/{1-start 2-inject-leak/mon centos_latest}
"2021-11-25T17:04:56.236233+0000 mgr.y (mgr.4100) 211 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/off mon_election/connectivity task/test_nfs}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-localized}
"2021-11-25T16:55:09.643413+0000 mgr.y (mgr.4110) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_mon_osdmap_prune}
"2021-11-25T16:56:21.158453+0000 mgr.x (mgr.4104) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects}
"2021-11-25T16:40:21.344188+0000 mgr.x (mgr.4110) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-balanced}
"2021-11-25T16:44:15.437433+0000 mgr.y (mgr.4107) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-hybrid rados tasks/rados_api_tests validater/valgrind}
"2021-11-25T17:15:02.038020+0000 mgr.x (mgr.4100) 234 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/with-work/{0-distro/centos_8.3_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced}
"2021-11-25T16:54:30.133222+0000 mgr.y (mgr.4107) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/test-crash mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream}}
Command failed (workunit test rados/test_crash.sh) on smithi129 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_crash.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized}
"2021-11-25T16:44:16.014709+0000 mgr.x (mgr.4100) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton-nomsgr/{all/balancer mon_election/connectivity rados supported-random-distro$/{rhel_8}}
"2021-11-25T16:53:10.754405+0000 mgr.x (mgr.4102) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zstd rados recovery-overrides/{default} supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T16:49:58.045933+0000 mgr.x (mgr.4105) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects}
"2021-11-25T16:50:59.306293+0000 mgr.y (mgr.4107) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/write_fadvise_dontneed}
"2021-11-25T17:08:25.614476+0000 mgr.y (mgr.4100) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_3.0} 2-node-mgr agent/off orchestrator_cli}
"2021-11-25T17:03:04.855280+0000 mgr.x (mgr.4712) 51 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests}
"2021-11-25T17:03:04.555507+0000 mgr.x (mgr.4105) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.stream_container_tools_crun} 1-start 2-services/basic 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/test_envlibrados_for_rocksdb mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}}
"2021-11-25T17:10:20.684535+0000 mgr.x (mgr.4101) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/sample_radosbench}
"2021-11-25T17:02:11.576845+0000 mgr.x (mgr.4100) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-agent-big}
"2021-11-25T18:08:37.285263+0000 mgr.y (mgr.4109) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/classic rados supported-random-distro$/{centos_8}}
"2021-11-25T18:11:48.773920+0000 mgr.x (mgr.4102) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-small}
"2021-11-25T18:13:48.901985+0000 mgr.y (mgr.4113) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-wait}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} tasks/rados_python}
"2021-11-25T18:11:25.279919+0000 mgr.y (mgr.4114) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy}
"2021-11-25T18:20:44.149966+0000 mgr.x (mgr.4102) 44 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps}
"2021-11-25T18:16:04.462405+0000 mgr.y (mgr.4110) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{centos_8} tasks/failover}
"2021-11-25T18:19:42.801473+0000 mgr.x (mgr.5142) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/thrash-backfill-full mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}}
"2021-11-25T18:23:47.795383+0000 mgr.x (mgr.4101) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/cache-snaps-balanced}
"2021-11-25T18:15:14.962064+0000 mgr.y (mgr.4100) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/on mon_election/classic task/test_orch_cli}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{default} supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
"2021-11-25T18:20:35.305229+0000 mgr.x (mgr.4104) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/connectivity rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T18:18:40.188219+0000 mgr.x (mgr.4101) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps}
"2021-11-25T18:27:43.410846+0000 mgr.y (mgr.4104) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/thrash/{0-distro/rhel_8.4_container_tools_3.0 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/cache}
"2021-11-25T18:24:47.338234+0000 mgr.x (mgr.4108) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/octopus backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_latest} mon_election/connectivity msgr-failures/osd-delay rados thrashers/careful thrashosds-health workloads/radosbench}
Command failed on smithi013 with status 5: 'sudo systemctl stop ceph-c1d16ad6-4e1c-11ec-8c2d-001a4aab830c@mon.a'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/objectstore/{backends/ceph_objectstore_tool supported-random-distro$/{ubuntu_latest}}
"2021-11-25T18:24:35.095114+0000 mgr.x (mgr.4099) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T18:29:33.466053+0000 mgr.x (mgr.4102) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-mixed}
"2021-11-25T18:39:35.348216+0000 mgr.y (mgr.4110) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-start-upgrade 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton/{all/thrash-eio mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T18:33:47.987829+0000 mgr.x (mgr.4100) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps}
"2021-11-25T18:40:03.363623+0000 mgr.x (mgr.4111) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/sync workloads/rados_mon_workunits}
Command failed (workunit test mon/pool_ops.sh) on smithi204 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects}
"2021-11-25T18:40:40.313644+0000 mgr.y (mgr.4110) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-fast-read}
"2021-11-25T18:39:11.261646+0000 mgr.y (mgr.4113) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/ceph-post-file mon_election/classic rados supported-random-distro$/{centos_8}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/standalone/{supported-random-distro$/{rhel_8} workloads/crush}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-snaps-few-objects-overwrites}
"2021-11-25T18:38:10.219711+0000 mgr.y (mgr.4104) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/repave-all}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados tasks/rados_cls_all validater/lockdep}
"2021-11-25T18:40:33.894266+0000 mgr.y (mgr.4110) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/rados_api_tests}
Command failed (workunit test rados/test.sh) on smithi170 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/on fixed-2 mon_election/classic start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_read}
"2021-11-25T18:38:51.048375+0000 mgr.x (mgr.4100) 31 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency}
"2021-11-25T18:41:20.709549+0000 mgr.x (mgr.4100) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/thrash-rados/{thrash-rados thrashosds-health} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}}
"2021-11-25T18:50:04.570174+0000 mgr.x (mgr.4101) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/radosbench}
"2021-11-25T18:49:31.775412+0000 mgr.y (mgr.4107) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T18:50:58.458372+0000 mgr.y (mgr.4106) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/thrash/{0-distro/rhel_8.4_container_tools_rhel8 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/redirect}
"2021-11-25T18:43:48.079520+0000 mgr.y (mgr.4111) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}
Command failed (workunit test cephtool/test.sh) on smithi121 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} tasks/rados_stress_watch}
"2021-11-25T18:40:26.776706+0000 mgr.x (mgr.4109) 31 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/crushdiff mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T18:37:30.784182+0000 mgr.x (mgr.4101) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs2 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests}
"2021-11-25T18:49:35.820805+0000 mgr.y (mgr.4104) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{rhel_8} tasks/insights}
"2021-11-25T18:52:57.608204+0000 mgr.x (mgr.5445) 60 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/redirect_set_object}
"2021-11-25T18:40:14.091139+0000 mgr.y (mgr.4111) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8}}
"2021-11-25T18:42:50.752085+0000 mgr.x (mgr.4102) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/set-chunks-read}
"2021-11-25T18:44:02.481223+0000 mgr.x (mgr.4108) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/osds/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced}
"2021-11-25T18:49:06.723714+0000 mgr.y (mgr.4101) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/smoke/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/export-after-evict mon_election/classic rados supported-random-distro$/{centos_8}}
"2021-11-25T18:42:40.864923+0000 mgr.x (mgr.4102) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/small-objects-localized}
"2021-11-25T18:39:47.582466+0000 mgr.y (mgr.4108) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/off mon_election/connectivity task/test_adoption}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/small-objects}
"2021-11-25T18:40:37.654339+0000 mgr.x (mgr.4111) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton/{all/watch-notify-same-primary mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced}
"2021-11-25T18:42:57.676217+0000 mgr.y (mgr.4100) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/multimon/{clusters/9 mon_election/classic msgr-failures/many msgr/async-v2only no_pools objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
"2021-11-25T18:42:06.508508+0000 mgr.x (mgr.4105) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/objectstore/{backends/filejournal supported-random-distro$/{rhel_8}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized}
"2021-11-25T18:46:06.932506+0000 mgr.y (mgr.4110) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/3-node k8s/1.21 net/host rook/master}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/classic objectstore/bluestore-bitmap openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw}
"2021-11-25T18:41:29.981460+0000 mgr.x (mgr.4100) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/snaps-few-objects}
"2021-11-25T18:55:15.193257+0000 mgr.y (mgr.4098) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T18:50:09.488466+0000 mgr.y (mgr.4110) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton-nomsgr/{all/full-tiering mon_election/connectivity rados supported-random-distro$/{rhel_8}}
"2021-11-25T18:58:02.811899+0000 mgr.x (mgr.4101) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed}
"2021-11-25T18:50:17.543798+0000 mgr.y (mgr.4103) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/osds/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-flag}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/force-sync-many workloads/snaps-few-objects}
"2021-11-25T19:06:51.062967+0000 mgr.x (mgr.4099) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton/{all/admin-socket mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/admin_socket_objecter_requests}
"2021-11-25T19:02:15.581032+0000 mgr.y (mgr.4108) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects-many-deletes}
"2021-11-25T19:09:04.845409+0000 mgr.x (mgr.4105) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/smoke/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/classic start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/mon_recovery validater/valgrind}
"2021-11-25T19:20:23.874346+0000 mgr.y (mgr.4107) 204 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-agent-big}
"2021-11-25T19:02:59.220878+0000 mgr.y (mgr.4100) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{ubuntu_latest} tasks/module_selftest}
"2021-11-25T19:11:58.417289+0000 mgr.x (mgr.7854) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} tasks/rados_striper}
"2021-11-25T19:03:06.694846+0000 mgr.x (mgr.4100) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/cache-agent-small}
"2021-11-25T19:06:33.890383+0000 mgr.x (mgr.4102) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/smoke-roleless/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton-nomsgr/{all/health-warnings mon_election/classic rados supported-random-distro$/{rhel_8}}
"2021-11-25T19:14:37.137406+0000 mgr.x (mgr.4102) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/standalone/{supported-random-distro$/{rhel_8} workloads/erasure-code}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/pacific backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_latest} mon_election/classic msgr-failures/fastclose rados thrashers/default thrashosds-health workloads/rbd_cls}
Command failed on smithi058 with status 5: 'sudo systemctl stop ceph-e9505dfa-4e22-11ec-8c2d-001a4aab830c@mon.a'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy}
"2021-11-25T19:09:34.902202+0000 mgr.y (mgr.4111) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T19:09:24.818966+0000 mgr.x (mgr.4111) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-start-upgrade 4-wait 5-upgrade-ls agent/on mon_election/classic}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps}
"2021-11-25T19:12:03.537675+0000 mgr.y (mgr.4104) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton/{all/backfill-toofull mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T19:07:58.309367+0000 mgr.x (mgr.4110) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-snaps-balanced}
"2021-11-25T19:10:51.081471+0000 mgr.y (mgr.4100) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/on mon_election/classic task/test_cephadm}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/cache-snaps}
"2021-11-25T19:12:57.859212+0000 mgr.y (mgr.4111) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/smoke-roleless/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-services/basic 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/e2e}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T19:09:24.791968+0000 mgr.x (mgr.4104) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache}
"2021-11-25T19:20:47.241348+0000 mgr.y (mgr.4100) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-dispatch-delay rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}
"2021-11-25T19:21:18.589586+0000 mgr.x (mgr.4111) 54 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/thrash/{0-distro/centos_8.2_container_tools_3.0 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-mixed}
"2021-11-25T19:20:18.425552+0000 mgr.y (mgr.4100) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/deduptool mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T19:14:37.369086+0000 mgr.x (mgr.4100) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/connectivity objectstore/bluestore-comp openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/fio_4M_rand_read}
"2021-11-25T19:11:14.012852+0000 mgr.x (mgr.4100) 31 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-wait}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/dedup-io-snaps}
"2021-11-25T19:15:41.158035+0000 mgr.y (mgr.4107) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/pool-snaps-few-objects}
"2021-11-25T19:16:34.908825+0000 mgr.x (mgr.4105) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/objectstore/{backends/filestore-idempotent-aio-journal supported-random-distro$/{rhel_8}}
"2021-11-25T19:22:41.286375+0000 mgr.x (mgr.4101) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/rados_api_tests}
Command failed (workunit test rados/test.sh) on smithi168 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{rhel_8}}
"2021-11-25T19:23:17.697317+0000 mgr.x (mgr.4101) 77 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/few msgr/async no_pools objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} tasks/mon_clock_with_skews}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
"2021-11-25T19:19:06.176870+0000 mgr.x (mgr.4108) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/radosbench-high-concurrency}
"2021-11-25T19:17:25.394397+0000 mgr.y (mgr.4104) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/divergent_priors mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T19:21:11.856218+0000 mgr.x (mgr.4101) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_8} tasks/progress}
"2021-11-25T19:20:09.994085+0000 mgr.x (mgr.4562) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 2-node-mgr agent/on orchestrator_cli}
"2021-11-25T19:30:48.346352+0000 mgr.x (mgr.4851) 51 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/radosbench}
"2021-11-25T19:28:32.227083+0000 mgr.x (mgr.4105) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} tasks/rados_workunit_loadgen_big}
"2021-11-25T19:20:59.435954+0000 mgr.y (mgr.4104) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T19:30:07.036616+0000 mgr.x (mgr.4101) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.2_container_tools_3.0} 1-start 2-services/rgw 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/redirect}
"2021-11-25T19:22:08.820825+0000 mgr.y (mgr.4103) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/client-keyring 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/many workloads/pool-create-delete}
"2021-11-25T19:22:53.727477+0000 mgr.x (mgr.4104) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/redirect_promote_tests}
"2021-11-25T19:23:18.019505+0000 mgr.y (mgr.4107) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/ec-small-objects}
"2021-11-25T19:25:05.068274+0000 mgr.y (mgr.4110) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T19:21:05.623642+0000 mgr.x (mgr.4101) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/off mon_election/connectivity task/test_cephadm_repos}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/redirect_set_object}
"2021-11-25T19:32:41.882926+0000 mgr.x (mgr.4099) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/divergent_priors2 mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8}}
"2021-11-25T19:32:31.298548+0000 mgr.x (mgr.4101) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/filestore-xfs rados tasks/rados_api_tests validater/lockdep}
Command failed (workunit test rados/test.sh) on smithi062 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/repave-all}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/set-chunks-read}
"2021-11-25T19:30:09.324425+0000 mgr.x (mgr.4105) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/classic start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/small-objects-balanced}
"2021-11-25T19:30:50.221879+0000 mgr.x (mgr.4110) 45 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T19:33:57.103757+0000 mgr.y (mgr.4107) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/thrash/{0-distro/centos_8.3_container_tools_3.0 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v2only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/classic objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw}
"2021-11-25T19:29:27.332161+0000 mgr.x (mgr.4100) 31 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/small-objects-localized}
"2021-11-25T19:32:27.433478+0000 mgr.x (mgr.4105) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/iscsi 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{rhel_8}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mgr}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/valgrind-leaks/{1-start 2-inject-leak/none centos_latest}
"2021-11-25T19:51:12.288548+0000 mgr.x (mgr.4121) 211 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects}
"2021-11-25T19:35:38.190635+0000 mgr.x (mgr.4102) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton/{all/dump-stuck mon_election/classic msgr-failures/none msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8}}
"2021-11-25T19:35:31.634594+0000 mgr.x (mgr.4100) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-balanced}
"2021-11-25T19:45:46.360323+0000 mgr.y (mgr.4107) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-start-upgrade 4-wait 5-upgrade-ls agent/off mon_election/connectivity}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-localized}
"2021-11-25T19:41:41.361661+0000 mgr.x (mgr.4108) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/snaps-few-objects}
"2021-11-25T19:49:09.494016+0000 mgr.y (mgr.4103) 31 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v1only backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_latest} mon_election/connectivity msgr-failures/few rados thrashers/mapgap thrashosds-health workloads/snaps-few-objects}
Command failed on smithi136 with status 5: 'sudo systemctl stop ceph-7e255c2e-4e27-11ec-8c2d-001a4aab830c@mon.a'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{filestore-xfs} supported-random-distro$/{centos_8.stream} tasks/prometheus}
Test failure: test_standby (tasks.mgr.test_prometheus.TestPrometheus)
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/mirror 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/write_fadvise_dontneed}
"2021-11-25T19:50:46.266419+0000 mgr.y (mgr.4107) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/multi-backfill-reject mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T19:41:01.019506+0000 mgr.x (mgr.4103) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton/{all/ec-inconsistent-hinfo mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}}
"2021-11-25T19:42:33.276544+0000 mgr.x (mgr.4105) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/objectstore/{backends/filestore-idempotent supported-random-distro$/{centos_8.stream}}
"2021-11-25T19:43:35.281924+0000 mgr.x (mgr.4102) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/admin_socket_objecter_requests}
"2021-11-25T19:41:54.237406+0000 mgr.y (mgr.4107) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/octopus 1-start 2-nfs 3-upgrade-with-workload 4-final}
hit max job timeout
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} tasks/rados_workunit_loadgen_mix}
"2021-11-25T19:44:55.217338+0000 mgr.x (mgr.4107) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-agent-big}
"2021-11-25T19:45:27.558779+0000 mgr.x (mgr.4099) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton-bluestore/{all/cephtool mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream}}
Command failed (workunit test cephtool/test.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/multimon/{clusters/3 mon_election/classic msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/mon_recovery}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
"2021-11-25T19:45:50.637743+0000 mgr.x (mgr.4103) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/cache-agent-small}
"2021-11-25T19:44:25.944076+0000 mgr.x (mgr.4113) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps-readproxy}
"2021-11-25T19:55:37.171117+0000 mgr.x (mgr.4108) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/ec-lost-unfound mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8}}
"2021-11-25T19:55:47.973947+0000 mgr.x (mgr.4105) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/osd_stale_reads mon_election/classic rados supported-random-distro$/{centos_8}}
"2021-11-25T19:50:25.171111+0000 mgr.x (mgr.4101) 44 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T19:52:00.144341+0000 mgr.y (mgr.4103) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/cache-pool-snaps}
"2021-11-25T19:52:15.698933+0000 mgr.y (mgr.4105) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/fastclose rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-small-objects-fast-read-overwrites}
"2021-11-25T19:51:57.807864+0000 mgr.x (mgr.4102) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/on mon_election/classic task/test_nfs}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/connectivity objectstore/bluestore-stupid openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_write}
"2021-11-25T19:50:43.475348+0000 mgr.x (mgr.4100) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/one workloads/rados_5925}
"2021-11-25T19:50:45.491440+0000 mgr.x (mgr.4107) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/cache-snaps-balanced}
"2021-11-25T19:52:10.413354+0000 mgr.y (mgr.4110) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
"2021-11-25T19:52:21.403793+0000 mgr.y (mgr.4110) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/cache-snaps}
"2021-11-25T19:57:31.492842+0000 mgr.y (mgr.4107) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/with-work/{0-distro/centos_8.2_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}
"2021-11-25T20:13:59.681756+0000 mgr.x (mgr.4107) 234 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache}
"2021-11-25T19:56:56.460465+0000 mgr.x (mgr.4101) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton/{all/erasure-code-nonregression mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/pool-access mon_election/connectivity rados supported-random-distro$/{centos_8}}
"2021-11-25T19:56:20.657051+0000 mgr.x (mgr.4101) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/dedup-io-mixed}
"2021-11-25T19:56:59.462221+0000 mgr.x (mgr.4107) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T20:01:16.367044+0000 mgr.x (mgr.4105) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-flag}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/dedup-io-snaps}
"2021-11-25T19:59:32.570393+0000 mgr.y (mgr.4103) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/classic start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-stupid} supported-random-distro$/{rhel_8} tasks/workunits}
"2021-11-25T20:07:49.097377+0000 mgr.x (mgr.4101) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/pool-snaps-few-objects}
"2021-11-25T20:00:59.012982+0000 mgr.x (mgr.4108) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-user 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/none 3-final cluster/3-node k8s/1.21 net/calico rook/1.7.0}
hit max job timeout
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/rados_api_tests}
Command failed (workunit test rados/test.sh) on smithi084 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools_crun 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v1only root}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton/{all/lost-unfound-delete mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8}}
"2021-11-25T20:05:47.275840+0000 mgr.x (mgr.4107) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/radosbench-high-concurrency}
"2021-11-25T20:09:44.708592+0000 mgr.x (mgr.4102) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} tasks/rados_workunit_loadgen_mostlyread}
"2021-11-25T20:07:26.276555+0000 mgr.y (mgr.4104) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/standalone/{supported-random-distro$/{centos_8} workloads/misc}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/off mon_election/connectivity task/test_orch_cli}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/radosbench}
"2021-11-25T20:17:19.013274+0000 mgr.x (mgr.4099) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/with-work/{0-distro/centos_8.3_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/objectstore/{backends/fusestore supported-random-distro$/{rhel_8}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/redirect}
"2021-11-25T20:08:12.921431+0000 mgr.y (mgr.4109) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/smoke-roleless/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect_promote_tests}
"2021-11-25T20:12:01.944221+0000 mgr.y (mgr.4103) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/classic objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4K_rand_read}
"2021-11-25T20:09:02.947077+0000 mgr.x (mgr.4099) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton/{all/lost-unfound mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T20:09:34.817451+0000 mgr.x (mgr.4112) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/repo_digest 3-start-upgrade 4-wait 5-upgrade-ls agent/off mon_election/classic}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_set_object}
"2021-11-25T20:10:47.207961+0000 mgr.x (mgr.4100) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton-nomsgr/{all/version-number-sanity mon_election/connectivity rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T20:13:56.203569+0000 mgr.x (mgr.4102) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/set-chunks-read}
"2021-11-25T20:21:09.942139+0000 mgr.y (mgr.4107) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/multimon/{clusters/6 mon_election/connectivity msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} tasks/mon_clock_no_skews}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
"2021-11-25T20:23:59.811229+0000 mgr.x (mgr.4101) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-wait}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/small-objects-balanced}
"2021-11-25T20:19:52.799527+0000 mgr.y (mgr.4103) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/nautilus-v2only backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_latest} mon_election/classic msgr-failures/osd-delay rados thrashers/morepggrow thrashosds-health workloads/test_rbd_api}
Command failed on smithi099 with status 5: 'sudo systemctl stop ceph-9b0e98b4-4e2c-11ec-8c2d-001a4aab830c@mon.a'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/smoke/{0-distro/ubuntu_20.04 0-nvme-loop agent/off fixed-2 mon_election/connectivity start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects-localized}
"2021-11-25T20:27:41.062921+0000 mgr.y (mgr.4110) 44 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T20:21:30.366784+0000 mgr.y (mgr.4110) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton/{all/max-pg-per-osd.from-mon mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T20:17:28.022873+0000 mgr.x (mgr.4102) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/thrash/{0-distro/rhel_8.4_container_tools_3.0 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/rados_api_tests}
Command failed (workunit test rados/test.sh) on smithi112 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/small-objects}
"2021-11-25T20:30:32.715375+0000 mgr.y (mgr.4112) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/minsize_recovery thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1}
"2021-11-25T20:31:11.309091+0000 mgr.x (mgr.4111) 46 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/dashboard}
Test failure: test_token_ttl (tasks.mgr.dashboard.test_auth.AuthTest)
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{bluestore-low-osd-mem-target} supported-random-distro$/{ubuntu_latest} tasks/crash}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/admin_socket_output mon_election/connectivity rados supported-random-distro$/{centos_8}}
"2021-11-25T20:22:41.012571+0000 mgr.x (mgr.4101) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/upgrade/parallel/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/smoke-roleless/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-services/nfs-ingress2 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/snaps-few-objects-balanced}
"2021-11-25T20:24:08.780660+0000 mgr.y (mgr.4104) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados tasks/rados_cls_all validater/lockdep}
"2021-11-25T20:25:21.988399+0000 mgr.y (mgr.4107) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/snaps-few-objects-localized}
"2021-11-25T20:33:09.549211+0000 mgr.y (mgr.4100) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/snaps-few-objects}
"2021-11-25T20:28:59.811074+0000 mgr.y (mgr.4104) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T20:37:26.367462+0000 mgr.y (mgr.4101) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/max-pg-per-osd.from-primary mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}}
"2021-11-25T20:36:15.779871+0000 mgr.x (mgr.4102) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.4 1-start 2-nfs 3-upgrade-with-workload 4-final}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/write_fadvise_dontneed}
"2021-11-25T20:39:56.264926+0000 mgr.y (mgr.4107) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream} tasks/readwrite}
"2021-11-25T20:33:34.124529+0000 mgr.y (mgr.4104) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/admin_socket_objecter_requests}
"2021-11-25T20:41:55.473074+0000 mgr.y (mgr.4100) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/orchestrator_cli/{0-random-distro$/{centos_8.stream_container_tools_crun} 2-node-mgr agent/off orchestrator_cli}
"2021-11-25T20:36:16.331603+0000 mgr.z (mgr.14674) 51 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.z: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/balancer mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T20:30:24.676958+0000 mgr.x (mgr.4101) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites}
"2021-11-25T20:41:13.152723+0000 mgr.y (mgr.4101) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/cache-agent-big}
"2021-11-25T20:35:13.706071+0000 mgr.y (mgr.4107) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/osds/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/connectivity objectstore/bluestore-bitmap openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4K_seq_read}
"2021-11-25T20:31:57.545304+0000 mgr.x (mgr.4100) 31 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-small}
"2021-11-25T20:37:52.235468+0000 mgr.x (mgr.4108) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/smoke/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/max-pg-per-osd.from-replica mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T20:38:19.956380+0000 mgr.x (mgr.4102) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/objectstore/{backends/keyvaluedb supported-random-distro$/{ubuntu_latest}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy}
"2021-11-25T20:46:27.144377+0000 mgr.x (mgr.4105) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke-singlehost/{0-random-distro$/{rhel_8.4_container_tools_3.0} 1-start 2-services/basic 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps}
"2021-11-25T20:40:21.110341+0000 mgr.y (mgr.4110) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/off mon_election/classic task/test_adoption}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton-nomsgr/{all/cache-fs-trunc mon_election/connectivity rados supported-random-distro$/{rhel_8}}
"2021-11-25T20:45:07.028006+0000 mgr.x (mgr.4101) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/standalone/{supported-random-distro$/{centos_8.stream} workloads/mon}
Command failed (workunit test mon/health-mute.sh) on smithi031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/health-mute.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-snaps-balanced}
"2021-11-25T20:38:47.880033+0000 mgr.y (mgr.4100) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/nfs 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/cache-snaps}
"2021-11-25T20:43:35.353675+0000 mgr.y (mgr.4100) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/thrash/{0-distro/rhel_8.4_container_tools_rhel8 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/mon-auth-caps mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T20:44:38.015907+0000 mgr.x (mgr.4100) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zstd} supported-random-distro$/{centos_8} tasks/failover}
"2021-11-25T20:47:08.598349+0000 mgr.x (mgr.5115) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache}
"2021-11-25T20:52:41.581847+0000 mgr.y (mgr.4110) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs2 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/multimon/{clusters/9 mon_election/classic msgr-failures/many msgr/async no_pools objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} tasks/mon_clock_with_skews}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
"2021-11-25T20:58:57.249306+0000 mgr.x (mgr.4102) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/dedup-io-mixed}
"2021-11-25T20:48:46.034909+0000 mgr.y (mgr.4104) 45 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/osds/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-ops/rm-zap-add}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/ceph-kvstore-tool mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T20:49:02.776779+0000 mgr.x (mgr.4104) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/none thrashosds-health workloads/dedup-io-snaps}
Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --set_chunk --enable_dedup --low_tier_pool low_tier --dedup_chunk_size 131072 --dedup_chunk_algo fastcdc --max-ops 1500 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 25 --op delete 10 --op snap_create 10 --op snap_remove 10 --op rollback 10 --op set_chunk 30 --op tier_promote 10 --op tier_evict 10 --op tier_flush 5 --op write_excl 25 --pool unique_pool_0'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/smoke/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T20:53:03.874967+0000 mgr.y (mgr.4107) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_mon_osdmap_prune}
"2021-11-25T20:51:20.639524+0000 mgr.x (mgr.4105) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects}
"2021-11-25T20:54:28.048260+0000 mgr.y (mgr.4107) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1}
"2021-11-25T21:02:45.115739+0000 mgr.x (mgr.4109) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/with-work/{0-distro/centos_8.stream_container_tools_crun fixed-2 mode/root mon_election/connectivity msgr/async-v2only start tasks/rados_python}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton/{all/mon-config-key-caps mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8}}
"2021-11-25T20:55:57.112214+0000 mgr.x (mgr.4102) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} tasks/repair_test}
"2021-11-25T20:54:41.409498+0000 mgr.x (mgr.4102) 47 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/rados_api_tests}
Command failed (workunit test rados/test.sh) on smithi081 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04-15.2.9 2-repo_digest/defaut 3-start-upgrade 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/classic objectstore/bluestore-comp openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_rand_read}
"2021-11-25T20:55:34.435100+0000 mgr.x (mgr.4099) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/radosbench-high-concurrency}
"2021-11-25T20:55:54.790112+0000 mgr.x (mgr.4105) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-snappy rados tasks/mon_recovery validater/valgrind}
"2021-11-25T21:16:40.159960+0000 mgr.x (mgr.4106) 204 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/radosbench}
"2021-11-25T20:58:59.776939+0000 mgr.y (mgr.4109) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton-nomsgr/{all/ceph-post-file mon_election/connectivity rados supported-random-distro$/{rhel_8}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_cephadm}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T21:10:42.178437+0000 mgr.y (mgr.4107) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/redirect}
"2021-11-25T21:03:12.007549+0000 mgr.x (mgr.4108) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-bluestore/{all/cephtool mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
Command failed (workunit test cephtool/test.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat distro$/{centos_latest} mon_election/connectivity msgr-failures/fastclose rados thrashers/none thrashosds-health workloads/cache-snaps}
Command failed on smithi090 with status 5: 'sudo systemctl stop ceph-e3d8cece-4e32-11ec-8c2d-001a4aab830c@mon.a'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-services/rgw-ingress 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/mon-config-keys mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T21:06:08.344210+0000 mgr.x (mgr.4109) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/redirect_promote_tests}
"2021-11-25T21:03:52.242568+0000 mgr.x (mgr.4103) 31 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/thrash/{0-distro/ubuntu_20.04 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/redirect_set_object}
"2021-11-25T21:06:19.772798+0000 mgr.y (mgr.4115) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/osds/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-ops/rm-zap-flag}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/objectstore/{backends/objectcacher-stress supported-random-distro$/{ubuntu_latest}}
"2021-11-25T21:03:34.664740+0000 mgr.x (mgr.4099) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/set-chunks-read}
"2021-11-25T21:07:27.491281+0000 mgr.x (mgr.4108) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton-nomsgr/{all/crushdiff mon_election/classic rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T21:07:24.764855+0000 mgr.x (mgr.4102) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools 0-nvme-loop agent/off fixed-2 mon_election/classic start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{bluestore-stupid} supported-random-distro$/{rhel_8} tasks/insights}
"2021-11-25T21:19:17.039050+0000 mgr.x (mgr.5099) 61 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects-balanced}
"2021-11-25T21:16:23.467627+0000 mgr.x (mgr.4104) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke-roleless/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-services/rgw 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton/{all/mon-config mon_election/classic msgr-failures/none msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T21:06:41.115088+0000 mgr.x (mgr.4115) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/small-objects-localized}
"2021-11-25T21:10:55.734619+0000 mgr.x (mgr.4109) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/small-objects}
"2021-11-25T21:08:33.557406+0000 mgr.y (mgr.4100) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/snaps-few-objects-balanced}
"2021-11-25T21:10:20.573923+0000 mgr.x (mgr.4105) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/export-after-evict mon_election/connectivity rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T21:11:19.475435+0000 mgr.x (mgr.4101) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/standalone/{supported-random-distro$/{rhel_8} workloads/osd-backfill}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/valgrind-leaks/{1-start 2-inject-leak/osd centos_latest}
"2021-11-25T21:29:16.115067+0000 mgr.y (mgr.4102) 211 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/thrash/{0-distro/centos_8.2_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-localized}
"2021-11-25T21:22:38.861297+0000 mgr.y (mgr.4109) 44 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/normal mon_election/connectivity msgr-failures/few rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-pool-snaps-few-objects-overwrites}
"2021-11-25T21:24:02.005901+0000 mgr.y (mgr.4110) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/dmclock_1Shard_16Threads settings/optimized ubuntu_latest workloads/radosbench_4M_seq_read}
"2021-11-25T21:14:22.349734+0000 mgr.x (mgr.4100) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/scrub_test}
"2021-11-25T21:15:13.340462+0000 mgr.x (mgr.4101) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/off mon_election/classic task/test_cephadm_repos}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T21:16:51.044007+0000 mgr.x (mgr.4105) 49 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/snaps-few-objects}
"2021-11-25T21:26:01.836206+0000 mgr.x (mgr.4105) 41 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/multimon/{clusters/21 mon_election/connectivity msgr-failures/few msgr/async-v1only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} tasks/mon_recovery}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3}
"2021-11-25T21:19:20.774727+0000 mgr.x (mgr.4105) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/smoke-roleless/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-services/basic 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/write_fadvise_dontneed}
"2021-11-25T21:30:33.349241+0000 mgr.x (mgr.4101) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/rook/smoke/{0-distro/ubuntu_20.04 0-kubeadm 0-nvme-loop 1-rook 2-workload/radosbench 3-final cluster/1-node k8s/1.21 net/flannel rook/master}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/osds/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-ops/rm-zap-wait}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/monthrash/{ceph clusters/9-mons mon_election/classic msgr-failures/mon-delay msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8} thrashers/force-sync-many workloads/rados_mon_workunits}
Command failed (workunit test mon/pool_ops.sh) on smithi106 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/admin_socket_objecter_requests}
"2021-11-25T21:23:14.431264+0000 mgr.y (mgr.4104) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-radosbench}
"2021-11-25T21:25:25.140346+0000 mgr.y (mgr.4101) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/osd-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T21:28:47.349223+0000 mgr.x (mgr.4102) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-agent-big}
"2021-11-25T21:39:00.288260+0000 mgr.x (mgr.4111) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/full-tiering mon_election/classic rados supported-random-distro$/{centos_8}}
"2021-11-25T21:29:26.436289+0000 mgr.x (mgr.4101) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-agent-small}
"2021-11-25T21:31:29.812296+0000 mgr.x (mgr.4102) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/many msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T21:32:20.519228+0000 mgr.x (mgr.4107) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mgr-nfs-upgrade/{0-centos_8.stream_container_tools 1-bootstrap/16.2.5 1-start 2-nfs 3-upgrade-with-workload 4-final}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zlib rados tasks/rados_api_tests validater/lockdep}
Command failed (workunit test rados/test.sh) on smithi025 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy}
"2021-11-25T21:32:43.729542+0000 mgr.y (mgr.4103) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/with-work/{0-distro/rhel_8.4_container_tools_rhel8 fixed-2 mode/root mon_election/connectivity msgr/async-v1only start tasks/rados_python}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8.stream} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T21:35:40.448520+0000 mgr.x (mgr.4108) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/cache-pool-snaps}
"2021-11-25T21:41:45.362234+0000 mgr.x (mgr.4099) 35 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-snappy} supported-random-distro$/{ubuntu_latest} tasks/module_selftest}
"2021-11-25T21:41:39.848340+0000 mgr.x (mgr.7866) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/smoke-roleless/{0-distro/centos_8.2_container_tools_3.0 0-nvme-loop 1-start 2-services/client-keyring 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/cache-snaps-balanced}
"2021-11-25T21:33:50.446761+0000 mgr.x (mgr.4103) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/upgrade/{1-start-distro/1-start-ubuntu_20.04 2-repo_digest/repo_digest 3-start-upgrade 4-wait 5-upgrade-ls agent/off mon_election/classic}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/health-warnings mon_election/connectivity rados supported-random-distro$/{centos_8}}
"2021-11-25T21:35:50.853150+0000 mgr.x (mgr.4101) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/cache-snaps}
"2021-11-25T21:38:52.276425+0000 mgr.y (mgr.4110) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/osd-recovery mon_election/connectivity msgr-failures/none msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8.stream}}
"2021-11-25T21:39:02.447149+0000 mgr.x (mgr.4107) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/objectstore/{backends/objectstore-bluestore-a supported-random-distro$/{centos_8}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache}
"2021-11-25T21:38:50.684608+0000 mgr.x (mgr.4101) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/thrash/{0-distro/centos_8.3_container_tools_3.0 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/dedup-io-mixed}
"2021-11-25T21:41:02.332972+0000 mgr.y (mgr.4101) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop 1-start 2-ops/repave-all}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/classic objectstore/bluestore-stupid openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/radosbench_4M_write}
"2021-11-25T21:39:08.832889+0000 mgr.x (mgr.4100) 31 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/dedup-io-snaps}
"2021-11-25T21:40:06.233103+0000 mgr.y (mgr.4110) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_3.0 0-nvme-loop agent/off fixed-2 mon_election/classic start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} tasks/libcephsqlite}
"2021-11-25T21:41:55.087463+0000 mgr.x (mgr.4105) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/large-omap-object-warnings mon_election/classic rados supported-random-distro$/{centos_8}}
"2021-11-25T21:41:20.296583+0000 mgr.x (mgr.4102) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash-old-clients/{0-size-min-size-overrides/3-size-2-min-size 1-install/octopus backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on distro$/{centos_latest} mon_election/classic msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/radosbench}
Command failed on smithi017 with status 5: 'sudo systemctl stop ceph-97dee2d2-4e38-11ec-8c2d-001a4aab830c@mon.a'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/pool-snaps-few-objects}
"2021-11-25T21:42:29.867064+0000 mgr.x (mgr.4110) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/peer mon_election/classic msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8}}
"2021-11-25T21:50:57.945541+0000 mgr.x (mgr.4108) 50 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/cephadm/smoke-roleless/{0-distro/centos_8.3_container_tools_3.0 0-nvme-loop 1-start 2-services/iscsi 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/rados_api_tests}
Command failed (workunit test rados/test.sh) on smithi184 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/on mon_election/connectivity task/test_nfs}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/radosbench-high-concurrency}
"2021-11-25T21:46:24.105019+0000 mgr.x (mgr.4107) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/with-work/{0-distro/ubuntu_20.04 fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/radosbench}
"2021-11-25T21:48:09.442073+0000 mgr.y (mgr.4099) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/multimon/{clusters/3 mon_election/classic msgr-failures/many msgr/async-v2only no_pools objectstore/filestore-xfs rados supported-random-distro$/{centos_8} tasks/mon_clock_no_skews}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/filestore-xfs rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8.stream} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2}
"2021-11-25T21:51:17.141611+0000 mgr.x (mgr.4108) 39 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/connectivity rados supported-random-distro$/{centos_8}}
"2021-11-25T21:50:20.640910+0000 mgr.x (mgr.4102) 58 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/standalone/{supported-random-distro$/{rhel_8} workloads/osd}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/none thrashosds-health workloads/redirect}
"2021-11-25T21:54:34.316068+0000 mgr.x (mgr.4108) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton/{all/pg-autoscaler-progress-off mon_election/connectivity msgr-failures/many msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream}}
Command failed (workunit test mon/pg_autoscaler.sh) on smithi008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pg_autoscaler.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/orchestrator_cli/{0-random-distro$/{rhel_8.4_container_tools_rhel8} 2-node-mgr agent/on orchestrator_cli}
"2021-11-25T22:06:41.912400+0000 mgr.x (mgr.4673) 53 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/monthrash/{ceph clusters/3-mons mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/many workloads/rados_mon_workunits}
Command failed (workunit test mon/pool_ops.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pool_ops.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect_promote_tests}
"2021-11-25T22:04:28.927087+0000 mgr.y (mgr.4107) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/ec-small-objects-balanced}
"2021-11-25T22:05:02.836767+0000 mgr.y (mgr.4104) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/mgr/{clusters/{2-node-mgr} debug/mgr mon_election/connectivity random-objectstore$/{bluestore-hybrid} supported-random-distro$/{centos_8.stream} tasks/progress}
"2021-11-25T22:00:21.193159+0000 mgr.x (mgr.4716) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/smoke-singlehost/{0-random-distro$/{centos_8.2_container_tools_3.0} 1-start 2-services/rgw 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/filestore-xfs rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1}
"2021-11-25T22:07:47.076732+0000 mgr.x (mgr.4099) 40 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/redirect_set_object}
"2021-11-25T21:59:50.117154+0000 mgr.y (mgr.4101) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools 0-nvme-loop 1-start 2-services/mirror 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/set-chunks-read}
"2021-11-25T22:09:09.848124+0000 mgr.y (mgr.4104) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/thrash/{0-distro/centos_8.stream_container_tools 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async-v1only root}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_cls_all validater/valgrind}
"2021-11-25T22:21:39.826325+0000 mgr.x (mgr.4113) 234 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/small-objects-balanced}
"2021-11-25T22:01:40.069003+0000 mgr.y (mgr.4104) 36 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton/{all/pg-autoscaler mon_election/classic msgr-failures/none msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest}}
Command failed (workunit test mon/pg_autoscaler.sh) on smithi170 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mon/pg_autoscaler.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/singleton-nomsgr/{all/librados_hello_world mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
"2021-11-25T22:00:52.487664+0000 mgr.x (mgr.4104) 37 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/osds/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop 1-start 2-ops/rm-zap-add}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-zlib rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
"2021-11-25T22:14:09.910475+0000 mgr.y (mgr.4102) 33 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/small-objects-localized}
"2021-11-25T22:15:51.772866+0000 mgr.x (mgr.4102) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/radosbench_omap_write}
"2021-11-25T22:05:41.164433+0000 mgr.x (mgr.4100) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-small-objects-fast-read-overwrites}
"2021-11-25T22:15:21.603338+0000 mgr.y (mgr.4104) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/cephadm/smoke/{0-distro/rhel_8.4_container_tools_rhel8 0-nvme-loop agent/on fixed-2 mon_election/connectivity start}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/none thrashosds-health workloads/small-objects}
"2021-11-25T22:06:33.830899+0000 mgr.x (mgr.4100) 31 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/objectstore/{backends/objectstore-bluestore-b supported-random-distro$/{rhel_8}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/smoke-roleless/{0-distro/centos_8.stream_container_tools_crun 0-nvme-loop 1-start 2-services/nfs-ingress-rgw-bucket 3-final}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/snaps-few-objects-balanced}
"2021-11-25T22:08:14.670063+0000 mgr.y (mgr.4104) 38 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} tasks/rados_api_tests}
Command failed (workunit test rados/test.sh) on smithi164 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7660f5736d47d57031adce01cfe693794700d0c1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rados/cephadm/workunits/{agent/off mon_election/classic task/test_orch_cli}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/snaps-few-objects-localized}
"2021-11-25T22:11:30.455467+0000 mgr.y (mgr.4104) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.y: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
rhel 8.4
rados/singleton/{all/pg-removal-interruption mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{rhel_8}}
"2021-11-25T22:20:01.561894+0000 mgr.x (mgr.4101) 44 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/singleton-nomsgr/{all/msgr mon_election/connectivity rados supported-random-distro$/{centos_8.stream}}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.2
rados/cephadm/with-work/{0-distro/centos_8.2_container_tools_3.0 fixed-2 mode/root mon_election/connectivity msgr/async start tasks/rados_python}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async objectstore/filestore-xfs rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/snaps-few-objects}
"2021-11-25T22:14:44.308761+0000 mgr.x (mgr.4105) 43 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/upgrade/{1-start-distro/1-start-centos_8.stream_container-tools 2-repo_digest/defaut 3-start-upgrade 4-wait 5-upgrade-ls agent/on mon_election/connectivity}
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_8.stream} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed}
"2021-11-25T22:15:21.436941+0000 mgr.x (mgr.4110) 34 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.stream
rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}}
Command failed on smithi038 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 014019fe-4e3d-11ec-8c2d-001a4aab830c -- ceph orch daemon add osd smithi038:vg_nvme/lv_2'
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/admin_socket_objecter_requests}
"2021-11-25T22:13:52.238004+0000 mgr.x (mgr.4105) 42 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
ubuntu 20.04
rados/cephadm/osds/{0-distro/ubuntu_20.04 0-nvme-loop 1-start 2-ops/rm-zap-flag}
timeout expired in wait_until_healthy
wip-sage3-testing-2021-11-24-1439
wip-sage3-testing-2021-11-24-1439
master
smithi
centos 8.3
rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-agent-big}
"2021-11-25T22:14:48.605654+0000 mgr.x (mgr.4105) 32 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: '>' not supported between instances of 'int' and 'tuple'" in cluster log
Pulpito