Description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/volume-client}

Log: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-24_10:25:03-kcephfs-master-distro-basic-smithi/5465758/teuthology.log

Sentry event: https://sentry.ceph.com/sentry/sepia/?query=e700df0540c74ad7af199fe31fef13ac

Failure Reason:

Command failed on smithi124 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel'

  • log_href: http://qa-proxy.ceph.com/teuthology/teuthology-2020-09-24_10:25:03-kcephfs-master-distro-basic-smithi/5465758/teuthology.log
  • archive_path: /home/teuthworker/archive/teuthology-2020-09-24_10:25:03-kcephfs-master-distro-basic-smithi/5465758
  • description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/volume-client}
  • duration: 0:05:45
  • email: ceph-qa@ceph.io
  • failure_reason: Command failed on smithi124 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel'
  • flavor: basic
  • job_id: 5465758
  • kernel:
    • client:
      • branch: testing
    • kdb: True
    • sha1: distro
  • last_in_suite: False
  • machine_type: smithi
  • name: teuthology-2020-09-24_10:25:03-kcephfs-master-distro-basic-smithi
  • nuke_on_error: True
  • os_type: centos
  • os_version: 8.1
  • overrides:
    • ceph-deploy:
      • fs: xfs
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • mon osd full ratio: 0.9
          • mon osd backfillfull_ratio: 0.85
          • bluestore fsck on mount: True
          • mon osd nearfull ratio: 0.8
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • bluestore block size: 96636764160
          • debug rocksdb: 10
          • osd failsafe full ratio: 0.95
      • bluestore: True
    • selinux:
      • whitelist:
        • scontext=system_u:system_r:logrotate_t:s0
    • workunit:
      • sha1: e492447fde1dd189f90331681978e2b2973d54bc
      • branch: master
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • fs: xfs
      • cephfs_ec_profile:
        • m=2
        • k=2
        • crush-failure-domain=osd
      • sha1: e492447fde1dd189f90331681978e2b2973d54bc
      • conf:
        • global:
          • ms die on skipped message: False
        • mgr:
          • debug ms: 1
          • debug mgr: 20
        • client:
          • debug ms: 1
          • debug client: 20
          • client mount timeout: 600
        • mon:
          • debug paxos: 20
          • debug mon: 20
          • debug ms: 1
          • mon op complaint time: 120
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • osd op complaint time: 180
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds verify scatter: True
          • mds bal fragment size max: 10000
          • mds op complaint time: 180
          • mds debug scatterstat: True
          • mds debug frag: True
        • osd:
          • mon osd full ratio: 0.9
          • debug ms: 20
          • debug journal: 20
          • debug osd: 25
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • mon osd backfillfull_ratio: 0.85
          • osd op complaint time: 180
          • bluestore block size: 96636764160
          • osd shutdown pgref assert: True
          • debug filestore: 20
          • debug rocksdb: 10
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 0.95
          • bluestore fsck on mount: True
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(FS_INLINE_DATA_DEPRECATED\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • MON_DOWN
    • install:
      • ceph:
        • sha1: e492447fde1dd189f90331681978e2b2973d54bc
    • admin_socket:
      • branch: master
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_teuthology@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'mds.b', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event: https://sentry.ceph.com/sentry/sepia/?query=e700df0540c74ad7af199fe31fef13ac
  • status: fail
  • success: False
  • branch: master
  • seed:
  • sha1: e492447fde1dd189f90331681978e2b2973d54bc
  • subset:
  • suite:
  • suite_branch: master
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: e492447fde1dd189f90331681978e2b2973d54bc
  • targets:
    • smithi139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuhzOEKBtn4iaJ3len5BDTHB/+gmwxGWGq5YcaBOJYK4xUgPnewbSux6Xnc7A7V9gZ8ZR4ktS6drNZUENATJY2pPO96oPubBWFRpaJlMzZIL3vBC5vDneNwxXPIBIhptgWOkkV6slUklYGv1tcaoFDn7symzVdUE++MD3PmQccTSuJNCFQAFIQlFl0827aWglnHuMfNLnVY2oPCXm1V/UF6mdv8VtSYxHW7ihkthv6HYN4EssQ5Olib5QQ0CD+ntGJdTamDaEZlDE8mv1Kf9bKHhqjlKEffn3l3yAhgWImOtShKCYD1usJJvOL9PBK+aG4IxXqpRf7nqhKXnoyT6Cxh63ecjEDINhXPDrzonPiFbJtRVSDejTLk8yCRifRdGPpzW1vahcPD2qRNsBbkD9JmDSVVck3//MJorTStBLV68DLIsnWHEyNoe2GFgs/5Zeq4cibd3XThI9iVIWpfh3FX1lo9aoqOfbxsmSl2Jd0Oxj6d5M/9KmXKmL6ZLaE2LU=
    • smithi105.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDETOUngcbNoC3XeTFvDHp3lO75KWzEA0kcfrFUxz4ui5irM4UtZydhNx79+v1hnTHOppv5SSWkoCKs3MpknqIFLzJ1QEv26QZyvvB7CUxrSSE/Hyba2m5qyk3OsgvMiW5rg5E8cDc/B+n71Sx9Bb7GRy1dW6U6ucaMmrZQ91LyrEiSbK7XXrMuH5iEvbOmm4nfQ7qJeOqECUwOoYjuEbWVBIbsSxl+UtTjPnyU+JUcaM7LOQRQILpGnVvTM5Sl6roLh53YBIq8Oitz+1Wiw71aFPQ8p13E9usn27suPj9fyM7lpUBJvuma8pZzwgVxjC/4tLRWxkvfna9CLQY+Bz26gpELzkyZqqlDXjmywUmj/1/pn9495Ux5ZIvb6BIN1+nX653lDaZxLk4tErUtAOf5NqB31GdlXdiqaUiJ5/TAx/Oiy4OVXPHe5PBF1LpSMEU5BDtVAJytkDjRTb0uXgJGzd/g58K7QQ+NdtmTJ95AtKhLN3Xqs7RgiNRTRLUf7Rs=
    • smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj9qmmaH5qjeJLtH2ru1Lrb7ALFMRdGEt8ZNlUXz19HKkuDxafwalSMVAPJwHi/oKcTRS/70wiqa8DS5gVjjjuvQd78k/1mwgXBHA/WcsTJ3zgqxdjHq/TK7kcmunrNXGiK3+/HkU+XSR3ClXBdjIftwLDGmT2rDCPJISJJfWg0+yixKjWepuGFEO1nmSeNmNuUpry+ToeitAhdQAb9ERDto4AnNy9d4yCHP1/vKIgUxZt+HxKPnUAICZp6wfOTd0TcllAWf5yOL1sQHyiYq+FRa6qdo61zXutz1WMkY6KSGFLZMPfWL3155ivcNhh/M03dtgn5gTq+YSXP6RLtcPXwMpgoDj3LvqKayGr49uZEN2rSXEyTxvISEY7V/AD3qZblDpPE0E2QzOED1RhmHhiEuMzHtcu48+LEBYDP6x8DbNaPf2Ga7BBgqpeOERFNj7UGKMMS6JEwoB4yXzAlfnPbcgqrHJLvoVEtizRpE8qBn57nCN2aoeOLV3CsYBfgKM=
    • smithi184.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCYtI8NNmUk+SNkvMhdUPIamltGc1I5TF8QGULQPe7mqvLFXPbGSfNpf+9m6lLXL0oGiRjIhZmamOlVzx0w7P1ZRo+zr50arBWqjPn209Oon6JsfJ5q0BmkLd3l+bPhHHLi/BX3or8hLgMjhwgTqAVeeD9EjgsrVRE4XKB3UNOq3dqvkTRIuFjxcjokxL4rbJhaHMbmVrgPP4gwLegSc9NUFxVVSkol7OoDkL27EYffQJy9AXKi7vPqi7FNMfb3tD796b1eKJio3OOafuBrI+CLEzotAkaPklnlHmZFKDjdRt2W4WOwJQNxJ4qREPlWO2hf3Q5OYxqrExCgoMqS4UDQTiWwN/TuGV7ye60KVqFAvN6u3tcOCpMM1jP/qhIr2WyIUjk2E32sDMNbc/hYxD6xR2N1EXPpvdrqm2k/WATcCO1n/Xf4QWhYUaUI+D/MWgBI6hUne2k7izxbd+az21mDorqCwX2XtKFN7HILaD8xuNbhT5SE80X6s6pgIthvjrU=
    • smithi144.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmHiz/sdkPeKWoxIN8u5+/rfnBbjhe4RlWDe+n2CGpdKpqhqfJij+h8ZadA86N266ebW9FTzsn0eig8vjNmJugm7lI8tVyrqGB0uzpDGOhwisIRGYli15WvQtmO1VQY/DDV1jG5DaZBK+jpgU7H7tBygovVoEifxuz5GAOrBZJKLDhzrVuvFfSM3SfOgng4S1HWgSOm4w64k/Ug0UJ4/DyoCR5b6yodSKdqMJKg9nf8LbS8ylQ6vi6NJtm6ySW1zBojZD/ZfhnsFJkPyF0c3kwde1Osk0ayCG1jyFX0vQS6dJhJlYlA7lxcb2Z98+W2rDJMOKWRew05mv1UC3/agCWBxQwbvD/aLiEdYdvqL6MkqtAgRZYaSK6eJ84dFAUsDZRokfoMAqv5CUxPex+77bABtUE4iRMju79dJeYmCkaktMSRFW0/0rmP+Y9H5kUmPuIKHw8Si+Ij1MNCJzh6FYP3/oNt5cxPDqv5bIgu/7lT65vrkd8ad7ZutKLJ6UkwVE=
    • smithi124.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCtZ4ltwi5inihtEMcazWLzWR27PD+RMC/k3kVOt3iKmZrNHW8d4HHRLg6LLkQUmZ9dhsEVxPpmbQ0yI9FR7RDZVFk/8Uxel6sEPOvawBQBLVyymhhf1Kd+OZLJhBXGTSxHRex1ZfJ/y0T05vB1J0tgNjDZz9gMim1xCdhIAh6OJdtw8yKj3Cz2BIiQhqyqu5lJwAf+hg3fqQHY+RjEKW26NqBc/B/VbJuVTpC1ZwRITH+IsFrb3tY4kjazK5+QM1t8wjtViHv39yYiIynrMeCUmAzBElar7HvB5ZmnIsXqIbR0uIzoWK9RCMjSZDoyX4aMGB3tXHsS3xefD6L2McndRU2d5nMK0lPzFo0pCgpTnz8vXuNNKZV1pFwYJTRqT6LlYFKTc1OC+wmPwZEdvzj+pQCQukP3oAE5POvhEgjChGXAFByC8J7oWqYs/CCtc3vS6VV59ElWQQ38ViJqMGBVixKFolVs/IyfeLDDIXE1+aaLcykq1e2+4c7eGdVA6lU=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 6
      • smithi
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • client:
        • branch: testing
      • kdb: True
      • sha1: distro
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
      • extra_system_packages:
        • deb:
          • bison
          • flex
          • libelf-dev
          • libssl-dev
          • network-manager
          • iproute2
          • util-linux
          • dump
          • indent
          • libaio-dev
          • libtool-bin
          • uuid-dev
          • xfslibs-dev
        • rpm:
          • bison
          • flex
          • elfutils-libelf-devel
          • openssl-devel
          • NetworkManager
          • iproute
          • util-linux
          • libacl-devel
          • libaio-devel
          • libattr-devel
          • libtool
          • libuuid-devel
          • xfsdump
          • xfsprogs
          • xfsprogs-devel
          • libaio-devel
          • libtool
          • libuuid-devel
          • xfsprogs-devel
      • extra_packages:
        • deb:
          • python3-cephfs
          • cephfs-shell
        • rpm:
          • python3-cephfs
      • sha1: e492447fde1dd189f90331681978e2b2973d54bc
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_volume_client
      • fail_on_skip: False
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2020-09-24 10:26:22
  • started: 2020-09-24 10:46:17
  • updated: 2020-09-24 11:56:18
  • status_class: danger
  • runtime: 1:10:01
  • wait_time: 1:04:16