yuriw-2023-03-23_15:21:23-fs-quincy-release-distro-default-smithi

{'smithi039.front.sepia.ceph.com': {'_ansible_no_log': False, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi039.front.sepia.ceph.com,172.21.15.39' (ECDSA) to the list of known hosts.\r\nubuntu@smithi039.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive)."}}

{'smithi039.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['dnf-utils', 'git-all', 'sysstat', 'libedit', 'boost-thread', 'xfsprogs', 'gdisk', 'parted', 'libgcrypt', 'fuse-libs', 'openssl', 'libuuid', 'attr', 'ant', 'lsof', 'gettext', 'bc', 'xfsdump', 'blktrace', 'usbredir', 'podman', 'redhat-lsb', 'firewalld', 'wget', 'libev-devel', 'valgrind', 'nfs-utils', 'ncurses-devel', 'gcc', 'git', 'make', 'python3-nose', 'python3-virtualenv', 'genisoimage', 'qemu-img', 'qemu-kvm-core', 'qemu-kvm-block-rbd', 'libacl-devel', 'autoconf', 'gdb', 'iozone', 'lvm2'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': 'Failed to download packages: Cannot download Packages/l/librados2-12.2.7-9.el8.x86_64.rpm: All mirrors were tried', 'results': []}}

Command failed on smithi144 with status 1: 'sudo yum install -y kernel'

Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)

{'smithi039.front.sepia.ceph.com': {'changed': False, 'msg': 'All items completed', 'results': [{'_ansible_item_label': ['cpanminus', 'perl-doc'], '_ansible_no_log': False, 'ansible_loop_var': 'item', 'cache_update_time': 1679587131, 'cache_updated': False, 'changed': False, 'failed': True, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'name': ['cpanminus', 'perl-doc'], 'only_upgrade': False, 'package': ['cpanminus', 'perl-doc'], 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': None, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'item': ['cpanminus', 'perl-doc'], 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" install \'cpanminus\'\' failed: E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 26104 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'rc': 100, 'stderr': 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 26104 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'stderr_lines': ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 26104 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], 'stdout': '', 'stdout_lines': []}]}}

{'smithi039.front.sepia.ceph.com': {'changed': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi039.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}}

Command failed on smithi039 with status 1: 'sudo yum install -y kernel'

Test failure: test_mirroring_init_failure_with_recovery (tasks.cephfs.test_mirroring.TestMirroring)

Cannot connect to remote host smithi039

"2023-03-23T17:10:41.808266+0000 osd.3 (osd.3) 3 : cluster [WRN] OSD bench result of 235106.769821 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.3. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log

Command failed (workunit test suites/fsstress.sh) on smithi145 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=714f8ff94ab1a8a5b10ea54247535614e53b7234 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

"2023-03-23T18:26:33.000284+0000 mon.a (mon.0) 2157 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log

"2023-03-23T18:24:26.147113+0000 mds.c (mds.0) 1 : cluster [WRN] evicting unresponsive client smithi107:0 (6955), after waiting 48.1982 seconds during MDS startup" in cluster log

"2023-03-23T18:48:13.704648+0000 mds.b (mds.0) 1 : cluster [WRN] client.4767 isn't responding to mclientcaps(revoke), ino 0x10000004fda pending pAsLsXsFscr issued pAsLsXsFscrb, sent 300.012156 seconds ago" in cluster log

Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)