Pulpito

yuriw-2020-06-15_18:59:46-kcephfs-wip-yuri-testing-2020-06-15-1645-octopus-distro-basic-smithi/5151262

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2020-06-15 19:00:38 2020-06-15 23:35:51 2020-06-16 00:27:51 0:52:00 0:13:29 0:38:31 smithi master centos 8.1 6 fail

Description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/damage}

Log: http://qa-proxy.ceph.com/teuthology/yuriw-2020-06-15_18:59:46-kcephfs-wip-yuri-testing-2020-06-15-1645-octopus-distro-basic-smithi/5151262/teuthology.log

Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=de531dd85cec4e77bde5e1e9d1ee695e

Failure Reason:

No module named 'cephfs.kernel_mount'

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/yuriw-2020-06-15_18:59:46-kcephfs-wip-yuri-testing-2020-06-15-1645-octopus-distro-basic-smithi/5151262/teuthology.log
  • archive_path: /home/teuthworker/archive/yuriw-2020-06-15_18:59:46-kcephfs-wip-yuri-testing-2020-06-15-1645-octopus-distro-basic-smithi/5151262
  • description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/damage}
  • duration: 0:13:29
  • email: ceph-qa@ceph.io
  • failure_reason: No module named 'cephfs.kernel_mount'
  • flavor: basic
  • job_id: 5151262
  • kernel:
    • client:
      • branch: testing
    • kdb: True
    • sha1: distro
  • last_in_suite: False
  • machine_type: smithi
  • name: yuriw-2020-06-15_18:59:46-kcephfs-wip-yuri-testing-2020-06-15-1645-octopus-distro-basic-smithi
  • nuke_on_error: True
  • os_type: centos
  • os_version: 8.1
  • overrides:
    • ceph-deploy:
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
    • selinux:
      • whitelist:
        • scontext=system_u:system_r:logrotate_t:s0
    • workunit:
      • sha1: f2a1445fea7d472ddf5bd826443bfb7ad4686b08
      • branch: wip-yuri-testing-2020-06-15-1645-octopus
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(FS_INLINE_DATA_DEPRECATED\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • bad backtrace
        • object missing on disk
        • error reading table object
        • error reading sessionmap
        • Error loading MDS rank
        • missing journal object
        • Error recovering journal
        • error decoding table object
        • failed to read JournalPointer
        • Corrupt directory entry
        • Corrupt fnode header
        • corrupt sessionmap header
        • Corrupt dentry
        • Scrub error on inode
        • Metadata damage detected
        • MDS_READ_ONLY
        • force file system read-only
      • fs: xfs
      • conf:
        • global:
          • ms die on skipped message: False
        • mgr:
          • debug ms: 1
          • debug mgr: 20
        • client:
          • debug ms: 1
          • debug client: 20
          • client mount timeout: 600
        • mon:
          • debug paxos: 20
          • debug mon: 20
          • debug ms: 1
          • mon op complaint time: 120
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • osd op complaint time: 180
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds verify scatter: True
          • mds bal fragment size max: 10000
          • mds op complaint time: 180
          • mds debug scatterstat: True
          • mds debug frag: True
        • osd:
          • mon osd full ratio: 0.9
          • debug ms: 20
          • bluestore fsck on mount: True
          • debug osd: 25
          • bluestore compression mode: aggressive
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • mon osd backfillfull_ratio: 0.85
          • osd op complaint time: 180
          • bluestore block size: 96636764160
          • osd shutdown pgref assert: True
          • debug filestore: 20
          • debug rocksdb: 10
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 0.95
          • debug journal: 20
      • sha1: f2a1445fea7d472ddf5bd826443bfb7ad4686b08
    • install:
      • ceph:
        • sha1: f2a1445fea7d472ddf5bd826443bfb7ad4686b08
    • admin_socket:
      • branch: wip-yuri-testing-2020-06-15-1645-octopus
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_yuriw@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'mds.b', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event: http://sentry.ceph.com/sepia/teuthology/?q=de531dd85cec4e77bde5e1e9d1ee695e
  • status: fail
  • success: False
  • branch: wip-yuri-testing-2020-06-15-1645-octopus
  • seed:
  • sha1: f2a1445fea7d472ddf5bd826443bfb7ad4686b08
  • subset:
  • suite:
  • suite_branch: wip-yuri-testing-2020-06-15-1645-octopus
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: f2a1445fea7d472ddf5bd826443bfb7ad4686b08
  • targets:
    • smithi176.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqYhU+R8YHXPPNhmpJWHS3KlKm6Plmh7e9bGQ2TPtJaD/6LlMU747omoFAjKIqjGHwRd6ljSlQUbPFA1npW1Fsmo+DyiCtrvIMq/UBI5HoKP22XqbYiNkLo6d/dBcndAHY2v1PIGnKYREJHl2jIWLov3XnXH3CWAjijVoNg1THiQZToq4NEzb7tMC0KPK+3IGJ7KCdNa/c2G1I9m7tmIXWJLitf2rON4hDNQ+peHTyJAH9Bb3ugxx4PGam9hy99ZH7kOBQO21iNBGiPFMTpOLVTcz66eVlRM+498zhIYM/mv7mnek2ClxkrRn3fOSnR1vPVCXRrzEpq6YieTFa88MfN323sM2eiS7o/iJe6u/ARF46GLwRV5EvPgQ6seossEZTH+MOi9MP1JF/SVYBAREiT+yTLP+IVIsY4SaKCN/JgxFbTt2TJroI22PRCBKdND3YBUmMTxvCWoPNy0Ntqt6j6B6u859G+BfoImL8Tq72B/YL3xpimRFu6UFNz5Knpvs=
    • smithi187.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4J+hofou9dM47Spp5MY6b2lDPWPwC5LYivfTvv/Xoomt6EdcOVdgCmMGtyVodTkiLkhXAfORDGbxLlnROVkaIDXw3swJCDyAU3kPQtb52CMz+xQ7/rZX9/damNIvY1As6f8wO0qZuYutnZePO3FvB81yUBoh0MSQsq7KJh7I0cnVlUI1wsDJPy9LZxe0Z/CzSoq9Zio+tGB5W2hStxy7GDgVQxmvqSC38DSn/acbWZrE7phuWJOU1uGTnWHjKgo5ieYhCAvmnD2hO5KEiAAqPpbq1V4Kzgtj9YK3ajWIUP3kjo/HK313VPTFCrdQlz98tUV71IJ51DEJKSeM44q2k+N2vXXE08r0EpXBfF+6YEgY+QkgquweMG/Pe8eKYAIavTed3kSg1Cs2aNYAGHArc36zlcCSTSm4ZUna1/ebyHr17Zw0QltNdPTaa7knkib6x5p3rv6J/o27WDgheZGPvmEP3mOUw8upCo0QVozJo+LECOXXYg8YYIn25jNlNyDU=
    • smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDY6Gd91KOxZ9UGg1ngNC8XhLHqjt/OABcUD9Ygso25Ptlzs/mRTJo+R4zhbJU8YZgVHaK8u1Fll1VxAi4KABfzKxb/Ecz/LMEe1Rwy1oNtSdSRt+FntfMdLOWFyDieZQEvdmb1YVNvgY9JCmenr4scfzxWwGLSkw+hzm6fHXbbpnEcBvv0eN9do4Wpm1V0aeMRcxeFQJjLsm0fEGe0zD7wBqcEG9ZYcfFrV59yjf2rxgxlQQxNxkPb8I52mGwRa664W+COaszeezFiFYpCHcLtvEy6zKIFmCxUKuJ4zs3U8J77kQuvrvBne8K1D/Ppdo5N0R3DJuSqyY8ptntYHkNqOrYylAXqDTdxrgtU9Q3jyw8BmH+KrAsO83h/zNrs2Z1F+Ex5Dc74uOLD8MRy8rqCwPc14UwagwEtMNCjlPYXqhGeJPQ/5B6lNMUFsGa7D3UCHfd5SF7qN/9V3FkSoAR2svSMGDjVa86xvUyOTcKJ5gvbTFZ2xV+NVUS9nt8+LRk=
    • smithi005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC26aXYe9IZGpkZeOm3snv8G5TeeGm5Urd3Z65bd2UoK5fAKa2vWC3mGUB9vfQJb+HZdgWK+f912sXWwUiq8nrUZpiJXWKy1DJMvvNvEp4639AH9s4QGVA7y+KnFGe4vQQfWpD41HNVpZq8Lfs4hmctXm6D0PxaoGR/a85GK3xPQs61n5u6GltNzLe9Z521KCgutiDC1+fRWYA0ZFy8qZtU+i/iwMoVrZ7L/fZdTdanJwf8KLUCmEb1fO5/GodtSQBQeEWShIqoEjp9kvnFB+01eQGDpd2m26Xvd9iQ0Su5X01tteopFEFSysVAEkTZ7W4bAEGK4App3F5n09QHWURAGniwtAg+V/73qG5JCH6rnhjnZrqKcx/LO023e7RaoPbVqYn4HGSfJZYt/F1/RlJPAjxWAxscMjnGnmHWxcu8bcUAwpNnUADc/OLRhieQowFxkUTPpD+O9HnouoZk8xACeHWaFmdz6Jg2jsKldCW9H8GAI903IeCCld9yQ+gaOrM=
    • smithi028.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/T0KzHOe6G5RuTchNiuyBbmYN1wwi444ZazcL4AAcp6HbQ7D4ekYi4TbnHw3uli+ULp8goZS7yek+mAIrLw1OPso1JDXfKaYUOFQT8YnyN0nbHTeLP9tCNjl0V8y7eh95JeEeuvmsRGrQ5620O1TIpuCiSDqb21nPYBN+of3nA/wxmjjb8czZWuLLeb0FBUE1riPSz/cniVy2IO6bcscw1MweGRormZoBVeXGkSy69XHAGIFa0GJwnjia9owdWKTO27Q1W0nIhrajbQYtJIWTlgwEdRZIsXVNk+S2g4qqs6AOoJQE3UG29WpzCCpG7Z9SbqtEduXXp7ZeGwBEX1Rxbl+wQmg9pTQGVP20usli1gahmwOOjPcRw3ATwrxSbLt54inWg1rWZsx+z8XwM+dlJla3qwDJT7obEIjufCv7rwrX8h2CNiqH8AVwU6Cx+WnkTYKlYF8lawAMtTG73xqUHEjQWvmSwmLTR6tvIG744fwz5cLgHuqNINZgZ1WaJX8=
    • smithi070.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDk93xVzQuRbR5JvLDn6PDpa6f9dc+GfxLc1gl63kHsSAigp0BnzmkCNgfCoM3pq2bfjeD8UCMvZ6nxYEZ+qgOIrPsHOeicrjxy7Nhr7lOjodwXb2v4JWcb4wa2CiQJverQKvb90u+Lb9kZUnCyeoRNaRrbFlFKRiQi5SLdX2T91NC08EsJ75oP2ZpybqGlKQO7YPnZKLd06sHrzaOniUzUlehgKmy4z0JHl17MIAaltvngNZAxFSmQUshToL7lGz8WaxRMzYXiqGKSLfyXnw9V6oD5eOaSL/ltYdX3JsWD1dd/aHnbkF/NccFtlnZyJRvWsml5KTmkKbv+E1pKOqemF0YrbokuKEl/3RNHEYvpUSBi2Xe/Zt064KkyvG6dM3VRvJz4yuLL+nS0tQ9b6qfiz+HTIYM61099wiNcariVeg/AzsSEWFafUfep/H1DEGc1eB0Y5d6DP7eZe0DNbs90BrVI9Ecxp/+aaQ7tQIACmeJQvP5Osft9J2ILRrZxnN0=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 6
      • smithi
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • client:
        • branch: testing
      • kdb: True
      • sha1: distro
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
      • extra_system_packages:
        • deb:
          • bison
          • flex
          • libelf-dev
          • libssl-dev
          • dump
          • indent
          • libaio-dev
          • libtool-bin
          • uuid-dev
          • xfslibs-dev
        • rpm:
          • bison
          • flex
          • elfutils-libelf-devel
          • openssl-devel
          • libacl-devel
          • libaio-devel
          • libattr-devel
          • libtool
          • libuuid-devel
          • xfsdump
          • xfsprogs
          • xfsprogs-devel
          • libaio-devel
          • libtool
          • libuuid-devel
          • xfsprogs-devel
      • extra_packages:
        • deb:
          • python3-cephfs
          • cephfs-shell
        • rpm:
          • python3-cephfs
      • sha1: f2a1445fea7d472ddf5bd826443bfb7ad4686b08
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_damage
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2020-06-15 19:00:38
  • started: 2020-06-15 23:35:51
  • updated: 2020-06-16 00:27:51
  • status_class: danger
  • runtime: 0:52:00
  • wait_time: 0:38:31