Description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}

Log: http://qa-proxy.ceph.com/teuthology/pdonnell-2020-10-24_06:26:58-kcephfs-wip-pdonnell-testing-20201024.032205-distro-basic-smithi/5553612/teuthology.log

Sentry event: https://sentry.ceph.com/sentry/sepia/?query=48a16fa5658347e3bfde3bb3bcdac0fa

Failure Reason:

Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)

  • kernel:
    • client:
      • branch: testing
    • kdb: True
    • sha1: distro
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 6
      • smithi
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • client:
        • branch: testing
      • kdb: True
      • sha1: distro
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
      • extra_system_packages:
        • deb:
          • bison
          • flex
          • libelf-dev
          • libssl-dev
          • network-manager
          • iproute2
          • util-linux
          • dump
          • indent
          • libaio-dev
          • libtool-bin
          • uuid-dev
          • xfslibs-dev
        • rpm:
          • bison
          • flex
          • elfutils-libelf-devel
          • openssl-devel
          • NetworkManager
          • iproute
          • util-linux
          • libacl-devel
          • libaio-devel
          • libattr-devel
          • libtool
          • libuuid-devel
          • xfsdump
          • xfsprogs
          • xfsprogs-devel
          • libaio-devel
          • libtool
          • libuuid-devel
          • xfsprogs-devel
      • extra_packages:
        • deb:
          • python3-cephfs
          • cephfs-shell
        • rpm:
          • python3-cephfs
      • sha1: 9926edf508fe4dc465539643dd9f95b0d0831e83
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_full
  • verbose: False
  • pid:
  • duration: 0:15:01
  • owner: scheduled_pdonnell@teuthology
  • flavor: basic
  • status_class: danger
  • targets:
    • smithi045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDI7AnkL9v4SqxdwUgC3rJiD1CeCq+eCaRxTLIfj35dvp5F3XSn+k9IrzLcVB5SxM++avwJEjs3/OVwAydu/1qH0Yhy+sTVMxkvXu+an8yt4UF8TkBtvgwLJVDxk5jyy7LgVyUtOsQtAYrfLuiwJG88OM0YtjpPc9mxrKUYmxHiJlW3iAi1tXEmaSuqTTCUiR9yJeFcioCtDXobHb6NAzeULj5zXm6OTycTCYsB5N6GFjNlOsZirDssycUpE6Iti81uaRU65JyijNhhW7wOKxkDDO2ZvZAU0/sadO/JeTP6Yoj7hT0PEXatVznv4fwJqUoJuQiOElnspADfy4s/8ETdBGDkLXa8Pb7Cz86z2Z3Ym4fEnv3lKOl61SbVLhxznPtITXGP/neYVUw5BXlrxhE2WPJ8DOzISSraXpKQOiV8oJbTqltFNt9RuP+B+DJognTNTRWz9iv+chh7dUa069hMwMVOeCeU7AWBbfa07iomzd8wHqWG+Hnz1+2Lw18ZKDc=
    • smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCe14OmAG4FWGPG3SvrmBxTkFyaSdQtt4fsbO0NdR7vGIssOlQ/FkYiLyycVo3w6wbDdNpVqphkNsTJb0AhMC1SNehseWq+FAj9rwDldu+O+k26TFD20ACIXW59uuAPYFNRgiy4TxUII46vsm+Qi9FMjeTyeUaofL0ffjpVKrcohgOI3a9yhPNGObPHwD4+KRQP99wkCWIM7/NAhdx42KpUoj27qcCKoPVRE5Z//VGiem8IzmMic/y63NXe9WZ2chnO6311RgNblVLOBIUjSs53akMR6FyIKIZwSBgc9igqBJ4CdHD4KKm/+t/C9AscdwQ12mQDX8j8+4F+VShiDU1XJQNamkClSF4OKbi/LAeakO9GJ8gIcT9+h0z5fo0jQt+HUKCKilxfxUnM8KdkC9+UAu5YO2AIwxZPonWTBSc/Ohs5Yms7B7GPWMH+BlW6MhDUJGSdpm0w6cpECyIhqTqTLSqGAPoZeC4wce0hOgllXViJJVCTV0CgQg94HtQPQ0E=
    • smithi002.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoyIjeejd0nhuYOjIG3sONILc3Bf8yE2mTttOF+p/y+hcLUHq7IkPFeMllY9g8VnrzM1+vzfUnMXcaABJt81GK9lCbQVvLEFV8TBBL+A2hrB8J4XKI+1R9XuWkHlcpiBeOrsL1ybForZbOhTj4xtgh9sN8Ox5ahx5Pof31pjeZyfN7DSd2u+K68nh6s0pXVnIGtd+k83jRSscn3SmfiZaiWqn07J0aEdjT1L7/EzY7K2SZ1rii7BIO8wStx2TCCVFJvzKBzZkoRhN5sISbn8rpin6sXI1VF+XXRVRNTT1QL2oVbujPxOeoqkbG2Ua2FYEqvmBt/kltqQ5FBlW3H2YgHqaHQ3WVEuvNTzJy7bNPilb4OCmtK+OzYQ1yHc3IVSst+6ZJW8SK0lhGZhIEB9IEQ+K+PK8vgR55js69rIypvmJnO6H50EFcZjEgfwNpjUJljTIFRDJoMTu7LCOUa2NOzGEJbcVaEEWEboDNyJ/aqyvQAAtMFsqw9dQcui4b2Xs=
    • smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMy1RmUOJmzV+yNiX6RP8xgVcXlx2GN/CZL/aFQbCIfhjG++r19DznMG7szCnODZ5Yp1oH0SBIEZ6j++cXyCbdtKsC1/0IVI6v7tYaQnmgHG0dKzCZRCtIypsUlw2jSRAn2M+if8OdJ9rM+XP2jMVTay7FreEp/kT7RuVc+WJcSa/bxcJCL75R7OKXVYe/xGdWxItkzOCakeKB0uLhTTQn0qh20Xih1gT5rfEScCYj9FYdapWM13KgOoTn9it6TaXvDE2ysvxvdflou5n3ii24I3A3UCnpDr0sfP1abjV3ENBfd8fPObEe5UERqPVDf64EhFb5ucLfVaVW2Lz7BVdfrwaMmbbXjKTz+T5SNeCmoHZwKjwsHcsJxrcTUs41NBKL4aULOpsdcdkXJ+fPJy1hu0vcZ02qIcZyR5wbvpFGNDwv6USp1wOEJg93gjan7XqOBJ01ahi8h9RionQcV+/QikV0TRAYhCzEKTGdMiqjxETwVE2HfUCfneiiXmUluB0=
    • smithi038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyAiyHYhTkVz7utcOcMzPmAOMQlF2tZb5kR2/lg763T/gHBkZgty6HvVpJ2t4I/ecrCflIriwYUjlVSU1mYqJsdK+mOVDHKuOsxk7xnDjnGnyWRj7r2QBJfSjWUSepDxNG6tSMxXMf37tH/qG/ZxkKI1Hs+sjUtsjSv0ve3Gqq7ZWANapNrBg0RTKvD9ZvWlD2fJWHxS1ybNKf03obe9NRQVLy1rr/WANaHROtVFXGpASDuPWQtv3XEGM5jgAz+YZRwQJYu0wdYkfpilfQncooNxaCXFWVJGqepbrHnXFj+UE2al6oOLjGoG05P1mlg4Q18gyJj189c6yoK43ktZVx19gpyP8aoGUaovsQ2mXUDHDyNvF5ZR0fY7SGAeLErUZZtTd3/g8IPXiXWq+fGjl/SMEYRNB7pFf4ArtAdlrj60AnvUb/gTyu5dlk+Q3ZctnY6Id8gAHiV/lYai5umn9jSTvJjOv31NAJzef6dVOf6sZWd37Q20M12A9v8NSYB4E=
    • smithi089.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+dl22xssi8EVF2W0Q+oAEskMQJ9BNmLbogA03QLn5ih2Zomts+6Am788vNco/100dQaSS5pl3/8nC7TR+s2UpVXzJKQgLs2bWo15yf09o1pgCVwoM514eow9gggrxj4MgoYXCUEMRq8597GiK9XCK3Rze9t5oOyVX6osIIsKNvaLFN3IkvVy4/0oQia04NXAot/DeCGs4HNUj5WYyvhSCIHJVd+YHnj5wHBYDwb0LGMinUruhKGM4q1zBB9bgJ7qwqgYSoWKUwVLMv+HxRqu1tvx5XsnsiWySUm1nuTCFtPFI27bEqauqKhqF3fW1uNJmJ+1GRKGSRpOvablVpfigTTc+4+nh622lwJLjxt9yE18nw2wbCJVHdPuSlfWL0UlvsoyE3PMLA/JVdLgEXWKyQfWeEEpqIOl/NAQOuIvSWiA0oiUkntoHGyxbc4HEiByozg7uwCguCmmh2oiy/TNpceOVu49B1sN5tR3ShLfDXJ/65nL4LFoB7WDmJYs5tZE=
  • job_id: 5553612
  • log_href: http://qa-proxy.ceph.com/teuthology/pdonnell-2020-10-24_06:26:58-kcephfs-wip-pdonnell-testing-20201024.032205-distro-basic-smithi/5553612/teuthology.log
  • suite_branch: wip-pdonnell-testing-20201024.032205
  • wait_time: 2:13:01
  • os_version: 8.1
  • branch: wip-pdonnell-testing-20201024.032205
  • pcp_grafana_url:
  • email: pdonnell@redhat.com
  • archive_path: /home/teuthworker/archive/pdonnell-2020-10-24_06:26:58-kcephfs-wip-pdonnell-testing-20201024.032205-distro-basic-smithi/5553612
  • updated: 2020-10-24 15:55:00
  • description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/filestore-xfs overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}
  • started: 2020-10-24 13:26:58
  • last_in_suite: False
  • machine_type: smithi
  • sentry_event: https://sentry.ceph.com/sentry/sepia/?query=48a16fa5658347e3bfde3bb3bcdac0fa
  • posted: 2020-10-24 06:28:17
  • teuthology_branch: master
  • sha1: 9926edf508fe4dc465539643dd9f95b0d0831e83
  • name: pdonnell-2020-10-24_06:26:58-kcephfs-wip-pdonnell-testing-20201024.032205-distro-basic-smithi
  • roles:
    • [u'mon.a', u'mgr.y', u'mds.a', u'mds.b', u'osd.0', u'osd.1', u'osd.2', u'osd.3']
    • [u'mon.b', u'mon.c', u'mgr.x', u'mds.c', u'osd.4', u'osd.5', u'osd.6', u'osd.7']
    • [u'client.0']
    • [u'client.1']
    • [u'client.2']
    • [u'client.3']
  • overrides:
    • ceph-deploy:
      • fs: xfs
      • filestore: True
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • osd sloppy crc: True
          • osd objectstore: filestore
    • selinux:
      • whitelist:
        • scontext=system_u:system_r:logrotate_t:s0
    • workunit:
      • sha1: 9926edf508fe4dc465539643dd9f95b0d0831e83
      • branch: wip-pdonnell-testing-20201024.032205
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • fs: xfs
      • cephfs_ec_profile:
        • disabled
      • sha1: 9926edf508fe4dc465539643dd9f95b0d0831e83
      • conf:
        • global:
          • ms die on skipped message: False
        • mgr:
          • debug ms: 1
          • debug mgr: 20
        • client:
          • rados osd op timeout: 15m
          • debug ms: 1
          • rados mon op timeout: 15m
          • debug client: 20
          • client mount timeout: 600
        • mon:
          • mon osd full ratio: 0.7
          • debug mon: 20
          • debug ms: 1
          • mon op complaint time: 120
          • mon osd nearfull ratio: 0.6
          • debug paxos: 20
          • mon osd backfillfull ratio: 0.6
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • mds bal fragment size max: 10000
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds verify scatter: True
          • rados osd op timeout: 15m
          • osd op complaint time: 180
          • mds op complaint time: 180
          • rados mon op timeout: 15m
          • mds debug scatterstat: True
          • mds debug frag: True
        • osd:
          • debug ms: 20
          • debug journal: 20
          • debug osd: 25
          • osd objectstore: memstore
          • osd mon report interval: 5
          • osd sloppy crc: True
          • osd op complaint time: 180
          • debug filestore: 20
          • memstore device bytes: 200000000
          • osd shutdown pgref assert: True
          • osd failsafe full ratio: 1.0
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(FS_INLINE_DATA_DEPRECATED\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • OSD full dropping all updates
        • OSD near full
        • pausewr flag
        • failsafe engaged, dropping updates
        • failsafe disengaged, no longer dropping
        • is full \(reached quota
        • POOL_FULL
        • POOL_BACKFILLFULL
    • install:
      • ceph:
        • sha1: 9926edf508fe4dc465539643dd9f95b0d0831e83
    • admin_socket:
      • branch: wip-pdonnell-testing-20201024.032205
  • success: False
  • failure_reason: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)
  • status: fail
  • nuke_on_error: True
  • os_type: centos
  • runtime: 2:28:02
  • suite_sha1: 9926edf508fe4dc465539643dd9f95b0d0831e83