Pulpito :: Results Dashboard

Posted	Started	Updated	Runtime	Duration	In Waiting	Machine	Teuthology Branch	OS Type	OS Version	Nodes	Status
2022-02-02 12:18:24	2022-02-03 19:19:19	2022-02-03 19:31:58	0:12:39	0:05:30	0:07:09	smithi	master	rhel	8.4	4	dead

Nodes: smithi066 smithi090 smithi150 smithi156

Description: powercycle/osd/{clusters/3osd-1per-target objectstore/filestore-xfs powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_misc thrashosds-health whitelist_health}

Log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-02_12:17:03-powercycle-pacific-distro-default-smithi/6655327/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=179218bf02c241b2a3b56ebfd4360b13

Failure Reason:

{'Failure object was': {'smithi066.front.sepia.ceph.com': {'msg': "Failed to download metadata for repo 'apache-arrow': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", 'results': [], 'rc': 1, 'invocation': {'module_args': {'name': ['gpg'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'invocation')"}

All Details...

log_href: http://qa-proxy.ceph.com/teuthology/teuthology-2022-02-02_12:17:03-powercycle-pacific-distro-default-smithi/6655327/teuthology.log
archive_path: /home/teuthworker/archive/teuthology-2022-02-02_12:17:03-powercycle-pacific-distro-default-smithi/6655327
description: powercycle/osd/{clusters/3osd-1per-target objectstore/filestore-xfs powercycle/default supported-all-distro/rhel_8 tasks/cfuse_workunit_misc thrashosds-health whitelist_health}
duration: 0:05:30
email: ceph-qa@ceph.io
failure_reason: {'Failure object was': {'smithi066.front.sepia.ceph.com': {'msg': "Failed to download metadata for repo 'apache-arrow': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", 'results': [], 'rc': 1, 'invocation': {'module_args': {'name': ['gpg'], 'state': 'present', 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_cache': False, 'update_only': False, 'validate_certs': True, 'lock_timeout': 30, 'conf_file': 'None', 'disable_excludes': 'None', 'download_dir': 'None', 'list': 'None', 'releasever': 'None'}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_0f634dc1c0a64a93646e6d007882c69b3e1663e0/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'invocation')"}
flavor:
job_id: 6655327
kernel:
- kdb: True
- sha1: distro
last_in_suite: False
machine_type: smithi
name: teuthology-2022-02-02_12:17:03-powercycle-pacific-distro-default-smithi
nuke_on_error: True
os_type: rhel
os_version: 8.4
overrides:
- admin_socket:
  - branch: pacific
- ceph:
  - conf:
    - mgr:
      - debug mgr: 20
      - debug ms: 1
    - mon:
      - debug mon: 20
      - debug ms: 1
      - debug paxos: 20
    - osd:
      - debug ms: 1
      - debug osd: 20
      - osd max markdown count: 1000
      - osd objectstore: filestore
      - osd sloppy crc: True
  - flavor: default
  - fs: xfs
  - log-ignorelist:
    - $MDS_ALL_DOWN$
    - $MDS_UP_LESS_THAN_MAX$
    - overall HEALTH_
    - $OSDMAP_FLAGS$
    - $OSD_
    - \(PG_
    - \(POOL_
    - \(CACHE_POOL_
    - \(SMALLER_PGP_NUM$
    - $OBJECT_
    - \(SLOW_OPS$
    - $REQUEST_SLOW$
    - $TOO_FEW_PGS$
    - slow request
    - $MDS_TRIM$
    - $MDS_SLOW_REQUEST$
    - MDS_SLOW_METADATA_IO
    - Behind on trimming
  - log-whitelist:
    - $MDS_ALL_DOWN$
    - $MDS_UP_LESS_THAN_MAX$
  - sha1: 9e9316478f690f97f4b84b62cb8253f68f19c8db
- ceph-deploy:
  - conf:
    - client:
      - log file: /var/log/ceph/ceph-$name.$pid.log
    - mon:
      - osd default pool size: 2
    - osd:
      - osd objectstore: filestore
      - osd sloppy crc: True
  - filestore: True
  - fs: xfs
- install:
  - ceph:
    - flavor: default
    - sha1: 9e9316478f690f97f4b84b62cb8253f68f19c8db
- selinux:
  - whitelist:
    - scontext=system_u:system_r:logrotate_t:s0
- workunit:
  - branch: pacific
  - sha1: e0aaec1e167344c5e3171dbd45219c6997c9f81d
owner: scheduled_teuthology@teuthology
pid:
roles:
- ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'client.0']
- ['osd.0']
- ['osd.1']
- ['osd.2']
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=179218bf02c241b2a3b56ebfd4360b13
status: dead
success: False
branch: pacific
seed:
sha1: 9e9316478f690f97f4b84b62cb8253f68f19c8db
subset:
suite:
suite_branch: pacific
suite_path:
suite_relpath:
suite_repo:
suite_sha1: e0aaec1e167344c5e3171dbd45219c6997c9f81d
targets:
- smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDPgTfxnnynC6tZE4XJmgflcSPTORPC08fB20BD2BQ+Hh/3cGzv586WDyUnmrP2Igicn0l1ZmLZ1ILLVxO+G6aa0TzAKg5EvsxSNegGLxdQ0ztPCs1OtwBwcJiDckHMIEKShK2xFF2HBRX7WC62k8zuCy2bGvX4UwgtyvBZVVAkvkaob17dZ06cPG3RthiArjuUgJEepIkDbuMPK3Ov7Tu8EbU3gt4cIRig1Bfu8Y8SXl9LWbQIah0M58kO5rZj4k/ci/i3Qzv4M0y8sLg+gesDA4Vk8BSmBano1tgsStRDO0SmL1KRyd1zos7/0vX05J6lZHYyBcTjRm5BqbsRu1HtovUmh9qEliG1OOdqSWAhwZQ5o8k2aWEA/4t4YFaf+BQsarHx+x1MPD+2lJiEgu8ovSWrKG59SxXevEcSv7K6cDUIXU2R7lOMavNTCXYuNo2Wjhp7PdmxMj9on/Op4r5o82+lFG6/LniIYhvKjSWa2PoWdjms68OsTbE0Y7fOiR8=
- smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCcGjWZx3+HVyVO9pgkSTl28vUPC60LNqUVXSbXwNpfPZ+Rhv9pvd5sDym11XBlq5J2yJaqrgZwiSZYN7Hb/da5//C6lxoP8bytL8TfUiwrAsxqKWTg61LmUXfW5lNRousYOvNDJAVAM3PU739L8/I1eL6Lawb9x8Ul48MbkvMCBbjKyLm81kuSYwRbeNbwzlPFKad6wL6ZVAl02/V6QwT/GFqtQirLU5plzOBpyIzsdTZU/fUK+QSL6tto19+utz57X0VbP2IGnVoHG6UovmR0OkVJhzosQSKb81FIiFLVLI2YoKHkSZZY5soUEJVyc7GsAKwqJ0eAdd1hV+efqYuIhDfg76Bh5T5/0tJGjxsrYCG+TvE71McqAm2fqWOmk+EBsogwoV/VkiB5JOm6fW+RQDzBvz1XQkaGYaptstFw/l+96nbm3JgdOaLGAdvOPb76bDuajdHn110MOSYxcIsndiGngxRYdwNjPP+mzrEk7D2pnbqU5lI9vVVryJVmuUc=
- smithi150.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdt9dZIvKnBLUOE6wRcEkx8W4sBlP/LWzTsrrHKDazzLnxBjEcad3zr1bSusprjcRj8JRfavTT4dV+hdX+dAgDXDe78vcURU1lo5eRjHnsmoZmamBE3t1kY9w2e0RPR/pNHM/wPcuBxfE8V9/n6b2UbVD9pnIf79OaukPbn1F9jjw/aeBoyYUUaIZS/MaS+sCiVDR02Ixwx6l/pPVgzDrBbkUuEgq14MfaCIx7qTvH4vNvqjYq0NC3WQm7D+uz9uGOZ3cSrAD/CtGAZPiNhilMqoDMyN9C7NNbZ5Q+oDt0m4LUbkjE+1gTlRIYtbL/ICui3V7xZjGDzp/6IQyoxXUNCvY1etqDuEgFyLRoKm56IlwCSw7wSoUkR+Q/W+tkJETE5r+ju4kAkl0h0WY8t4Q4p+C9XEOWZWp0JDamFBVGBjJCygSccth5qPuIM9jj3W/pb553WCtM+YLIEItFzMbDItTc1pEfH/LbBByn9SkZ2W0zZz5ITmyJYS4FXwu0juk=
- smithi156.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwc9u/RE9pArbXoP61+laLnRjQYxpRdCsaW9oan3IGsO+LDLENjvWLoXD68YpmhR00Rg4rdGp3agZxlUXA740xN2x6XTswk+uLOy3UPh+T70KLFI02txik7dSgl1p1dqLu0rJym8LZbrt+tGEE6mC7E9d71Y1hbdnneiAnPC7Wlvhl6R4vdpHABkcnSXqfJfO96PdzXiABdvYnnRDjHe/uP1/szzbfgn3EQE8AcswgukjiWtNi4x87hlhTEgwaAmLFJCXeab7fhVXP8qyipFCP/CJ9At6RayDUKEYROq1dQejCLc0L7x76Lj6xIq0CrZ1M+u5E1kVOGn+Ogt2jioWHqybN8COaN5xRmNvkNiUHh870XCFn865O4QlUHOPoD9cZz0yj8yaZ+lN7Q9JnR6J9ERENxgWSLy2Gwj0EQhqf6VqCPRhAzTI8rGReUghFoM38/ctfZwRiKpcBAAQHIUIj+02a57DdrQBMHTVFT0/IETXoKVnBgjG9De3vE6UD2jU=
tasks:
- internal.check_packages:
- internal.buildpackages_prep:
- internal.save_config:
- internal.check_lock:
- internal.add_remotes:
- console_log:
- internal.connect:
- internal.push_inventory:
- internal.serialize_remote_roles:
- internal.check_conflict:
- internal.check_ceph_data:
- internal.vm_setup:
- kernel:
  - kdb: True
  - sha1: distro
- internal.base:
- internal.archive_upload:
- internal.archive:
- internal.coredump:
- internal.sudo:
- internal.syslog:
- internal.timer:
- pcp:
- selinux:
- ansible.cephlab:
- clock:
- install:
  - extra_system_packages:
    - deb:
      - bison
      - flex
      - libelf-dev
      - libssl-dev
      - libaio-dev
      - libtool-bin
      - uuid-dev
      - xfslibs-dev
    - rpm:
      - bison
      - flex
      - elfutils-libelf-devel
      - openssl-devel
      - libaio-devel
      - libtool
      - libuuid-devel
      - xfsprogs-devel
- ceph:
- thrashosds:
  - chance_down: 1.0
  - powercycle: True
  - timeout: 600
- ceph-fuse:
- workunit:
  - clients:
    - all:
      - fs/misc
  - timeout: 6h
teuthology_branch: master
verbose: True
pcp_grafana_url:
priority:
user:
queue:
posted: 2022-02-02 12:18:24
started: 2022-02-03 19:19:19
updated: 2022-02-03 19:31:58
status_class: danger
runtime: 0:12:39
wait_time: 0:07:09