Pulpito

pdonnell-2020-07-28_23:57:49-kcephfs-wip-pdonnell-testing-20200728.221932-distro-basic-smithi/5265402

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2020-07-28 23:58:49 2020-07-29 02:08:42 2020-07-29 03:32:44 1:24:02 0:21:44 1:02:18 smithi master rhel 8.1 6 pass

Description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub}

Log: http://qa-proxy.ceph.com/teuthology/pdonnell-2020-07-28_23:57:49-kcephfs-wip-pdonnell-testing-20200728.221932-distro-basic-smithi/5265402/teuthology.log

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/pdonnell-2020-07-28_23:57:49-kcephfs-wip-pdonnell-testing-20200728.221932-distro-basic-smithi/5265402/teuthology.log
  • archive_path: /home/teuthworker/archive/pdonnell-2020-07-28_23:57:49-kcephfs-wip-pdonnell-testing-20200728.221932-distro-basic-smithi/5265402
  • description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub}
  • duration: 0:21:44
  • email: pdonnell@redhat.com
  • failure_reason:
  • flavor: basic
  • job_id: 5265402
  • kernel:
    • client:
      • sha1: distro
    • kdb: True
    • sha1: distro
  • last_in_suite: False
  • machine_type: smithi
  • name: pdonnell-2020-07-28_23:57:49-kcephfs-wip-pdonnell-testing-20200728.221932-distro-basic-smithi
  • nuke_on_error: True
  • os_type: rhel
  • os_version: 8.1
  • overrides:
    • ceph-deploy:
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
    • selinux:
      • whitelist:
        • scontext=system_u:system_r:logrotate_t:s0
    • workunit:
      • sha1: 30aa53049577e70968a728cf36a545a209e8d28b
      • branch: wip-pdonnell-testing-20200728.221932
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(FS_INLINE_DATA_DEPRECATED\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • inode wrongly marked free
        • bad backtrace on inode
        • inode table repaired for inode
        • Scrub error on inode
        • Metadata damage detected
      • sha1: 30aa53049577e70968a728cf36a545a209e8d28b
      • fs: xfs
      • conf:
        • global:
          • ms die on skipped message: False
        • mgr:
          • debug ms: 1
          • debug mgr: 20
        • client:
          • debug ms: 1
          • debug client: 20
          • client mount timeout: 600
        • mon:
          • debug paxos: 20
          • debug mon: 20
          • debug ms: 1
          • mon op complaint time: 120
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • osd op complaint time: 180
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds verify scatter: True
          • mds bal fragment size max: 10000
          • mds op complaint time: 180
          • mds debug scatterstat: True
          • mds debug frag: True
        • osd:
          • mon osd full ratio: 0.9
          • debug ms: 20
          • bluestore fsck on mount: True
          • debug osd: 25
          • bluestore compression mode: aggressive
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • mon osd backfillfull_ratio: 0.85
          • osd op complaint time: 180
          • bluestore block size: 96636764160
          • osd shutdown pgref assert: True
          • debug filestore: 20
          • debug rocksdb: 10
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 0.95
          • debug journal: 20
      • cephfs_ec_profile:
        • m=2
        • k=2
        • crush-failure-domain=osd
    • install:
      • ceph:
        • sha1: 30aa53049577e70968a728cf36a545a209e8d28b
    • admin_socket:
      • branch: wip-pdonnell-testing-20200728.221932
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_pdonnell@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'mds.b', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event:
  • status: pass
  • success: True
  • branch: wip-pdonnell-testing-20200728.221932
  • seed:
  • sha1: 30aa53049577e70968a728cf36a545a209e8d28b
  • subset:
  • suite:
  • suite_branch: wip-pdonnell-testing-20200728.221932
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 30aa53049577e70968a728cf36a545a209e8d28b
  • targets:
    • smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8VPQl9/IJUGANs7pem9ILk7hxR9bdYRQYqWVNonIYbu+vax513hbSE6wdsk5tES+EURKT15Z4iCuzO5Gzb+ofA9EDyH2l64Urxfh2/90GMUyX174BQJSaXdL+KGITmZ9fPumZayeq7HSBOcdzNmwo0LkNa1bkUfcOkkGRit4L1a2gaCkHFjkxSLtYaPlPw9bhCJsHYQd8ZvxMADT8oVK6FS9MNpmjSqJxKwS7wFP9GJrw0qa0UqrPWaRKsOR4lUmXKSsjQQRcy/s/LNhYSsADvUos5oO1g0hbe4rhjEVfInRormBBFgJ1RHsOQwZ0a0XPpmUM+F4eB/VK540NWDpmF4FQG1EBoHfTUSimMB9YG0H0xTDmFma1GvsGZ/7FtJsLBAXOVU2SL+SMCDIBTtVF6w7z3U0rvxhmpNBwrlAUjMtfBTfhulzjO+i5FbOdK+XisEYA7t7cehC/djNWosdIH2HRBfFVvMf0PDIV2nq4XY7hjWb9XGZtV5SL364irrs=
    • smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAgsbvxswxT8jZX5EUCSEsibkRSCTmp5nTOhcrgmf1KUl09i3+1XyOAxSQu1j9nboTqAQoh2IeuFVRWycHrcFwMqcytaA5as8K6TxgBSg/49cjlLVy7MiwxiSGzNiPbU1jfwMXZSKv+nUP0bL9qiJE+g1lZcng5rJjcSLWBM3YBBkJNuivMiY5nMbEDTXFNN6+tAO0nZ7V1v6yxQKOIdWPfxMD5Jeh+D9B2TvXLR6JeGu4PFIkONzODmSKHycO8FW8pAm4gYeq1fvixnj5z4m6qI2fJ0780jVxkJpoKZ0l+dLfQuVWvFE5zKex2pZD6fxGmcTV85w9u24lKcjCNbTgxaMipS7/ioU+RNRGP/kAxuIlHikN2anUQxgs6P/34v38aqF+VsJybRQ0w06jMypHbJLEjfT6N6bwRmuJaHdrMl3oOM1ZnctNqNFaY7Qeo/vElZvFOWWxbRA54bXPY8aC4kcg9OGCnE2jzgPsGiH6tXSVpM+shOXGzJ/yEgGn9K0=
    • smithi082.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCl9n+XCH+iaGF+MbTTUTSrM3FXR7Iegqe9SYVspH5Fl09J9FRRVoz8mGOjmGx9kJvNUeiGqYFco+Iuu0UJ6axEZAguz3MupL8Tqq9QyGTX3gRzn6gNJuNrsAqHA9dsmRpm083ieMuwy8TyWoMK+1UwwtRC508ASQgXXRKUpDDTLTOqvNKcd4I98L1ZV0JC9xzbE6WPH7NDBNbmxcPX9ghvfzmebJXrX9b/Wl4VxPK1wch/2FVihAzVxlnp44GpePVjO0Srv7ZVLaDJ/4s/jpwFMTYzCm24Y1c77X1IhCK0oLF1GVHwvYK6JqG6e14qh5VS2SEyCzF4hjCqsujfPAT5CKl08zA9Iu8EnrHbLiCj0BAuLPyduzVz5/kPp6Q5E90lTzpPzIDIFzIB2PoYEx/4Pbta8mcpmeRTUxlny5JQwgM9Hrabw0pZx8sSM1uahp9xQe5HbTL7cwH7nel5a2SdtLr1x26VosuDP+/cKs/tThvDvQ+K6DYVaTeQu+rAENc=
    • smithi112.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC53goy3ZZ+ApDUxs1s46r/DJ+E3NHHP4B34i7qrkad6tUbHCpdyu2qM3S6CqRm5S3FDBLXc1VIemCgN7BLwKzC3xNnOFtGyQ1Tg0XN1hVBYfyUzhXjF7KlEeEMxmaxDPkVgl5zxW/RL1EzZulDKd+tH1HQs9QiVeogn4fSORC1XBSApUWzFDITCQ/bC95PRjY9Pgct0+wMWY93vLjeOEI5R17vDTgIUVdKmJgnnPj7nbRx4p2rk3hwXGOY3hDzQ3VcAjO0CaJNa36PCq/tWv8h+FzLqHC1IHEMB+jI6/+C2fNjk2zsxolzyTssob7fZoNohouQWMsaAR/LdkSKLXu9QRsM3RQlPScwIDYoe6T60El7+1pmoOV+L08gW8klqwnINMyJASoXKxaggOW9PlrfuJQHsX1LMTIqPQMJfFSDZ9bHzn7utWGTyZEBN4uFmXraoais57rhL8Gaz519VRjcZVanq/70OKaaCqQfFa9LrZOwRL6X76gw0fjXVFaI5Yk=
    • smithi171.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDARE1EZM8ku2TA2phxJwqFpLz79ZoEjsIQXJ6B1KhB2gzqp6pYFPVpVXkka4tgbQNURncbfRox/7Iy2nwQVTJyIeNnd8rD2yg87Hapk0BYTetM8jKQt0QhUKTk9EfFhQ/waoxqCFODaFZxp/0RFUUSZ1+v+RlTBxQ0KxpEbZPb2pnbAWpY3n04tGd//myvk6nE3Ma6OqGWNVlrzEXKSYxwfbv1T2Z4LmyM0ID/1YRKArzUy5O492yn8kRd/soU+JDzxtoIM2CbpMwPDZ1QgqznlE3IE95zqjKhuPyMO0xrj2dx9vvuaLLg9wglbVztRvd/8Yxb/90Um6+iKKcvzOcisAqzhVHupZnRHmsbe+qbIhdgkpqxipJC4rawF3swctPq25Rw++sfATcgAl6j91/Flh6PsLRuzrZ/pEXJ5lFjTc7bfkI+3kE2pcKT78xABWam7sZjw8uvCiTdXAddaSiwWqc9F471zBkV5m5PVXNS7EBQV2TX38xWpeJdCdNyET8=
    • smithi132.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMjjqu05V6R7knMf5c07KbTuv+Eq+KSxsHaQdFfWdfYtwyOm3kJQimtb7+7q3z1lIy8X4XYap5oh937UeLFm7StX6bWQRyGWAyAwP13Os1X3cCPMp9HVz+meyrBa1xq/EVze09OwEKVP020ABquYbCGwkrxl1AY3TT0Wi82C4o+YYK8SCwz0BhThD98zQZ1k5C2EWYn7UVAFiigZsppPQw31km9tSRhA2DC09URNL5Cry4EnIwtKF5g1O3sM6Nwz7qI2EWZqVpgcM1daidj9MhrSjOgeIX6pHRiffMSXeUHnhbfpAcfvMaHC1oZ79YJ1XYbwCjEb/AIiHBlBw8hmG31as8oCQ/b65gQE8qDUA5Cppd0K5kJYiPnDpCRoZ0VS9nqf86PwKqdD4kBKcDHVoFmQPWRi4NxuJaYw3CJ1GtLhaqVo7OMgDbky17FxTdUk6YLZw6EuBn2BnuT+NdebCQQrampdX/6AcWlptDwODnzNiJ4ypmeoawGwOUP6KfSsU=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 6
      • smithi
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • client:
        • sha1: distro
      • kdb: True
      • sha1: distro
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
      • extra_system_packages:
        • deb:
          • bison
          • flex
          • libelf-dev
          • libssl-dev
          • network-manager
          • iproute2
          • util-linux
          • dump
          • indent
          • libaio-dev
          • libtool-bin
          • uuid-dev
          • xfslibs-dev
        • rpm:
          • bison
          • flex
          • elfutils-libelf-devel
          • openssl-devel
          • NetworkManager
          • iproute
          • util-linux
          • libacl-devel
          • libaio-devel
          • libattr-devel
          • libtool
          • libuuid-devel
          • xfsdump
          • xfsprogs
          • xfsprogs-devel
          • libaio-devel
          • libtool
          • libuuid-devel
          • xfsprogs-devel
      • extra_packages:
        • deb:
          • python3-cephfs
          • cephfs-shell
        • rpm:
          • python3-cephfs
      • sha1: 30aa53049577e70968a728cf36a545a209e8d28b
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_forward_scrub
  • teuthology_branch: master
  • verbose: False
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2020-07-28 23:58:49
  • started: 2020-07-29 02:08:42
  • updated: 2020-07-29 03:32:44
  • status_class: success
  • runtime: 1:24:02
  • wait_time: 1:02:18