Pulpito

yuriw-2022-07-20_14:56:20-kcephfs-octopus-release-testing-default-smithi/6940827

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2022-07-20 14:57:28 2022-07-20 23:11:11 2022-07-20 23:47:29 0:36:18 0:19:22 0:16:56 smithi main ubuntu 20.04 6 fail

Description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}

Log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-07-20_14:56:20-kcephfs-octopus-release-testing-default-smithi/6940827/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8de45cc211a44036ac85f50e2bd80904

Failure Reason:

Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/yuriw-2022-07-20_14:56:20-kcephfs-octopus-release-testing-default-smithi/6940827/teuthology.log
  • archive_path: /home/teuthworker/archive/yuriw-2022-07-20_14:56:20-kcephfs-octopus-release-testing-default-smithi/6940827
  • description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}
  • duration: 0:19:22
  • email: ceph-qa@ceph.io
  • failure_reason: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)
  • flavor: default
  • job_id: 6940827
  • kernel:
    • client:
      • branch: testing
    • flavor: default
    • kdb: True
    • sha1: 7515a529973fc26f43af978722a22367693c4342
  • last_in_suite: False
  • machine_type: smithi
  • name: yuriw-2022-07-20_14:56:20-kcephfs-octopus-release-testing-default-smithi
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version: 20.04
  • overrides:
    • admin_socket:
      • branch: octopus-release
    • ceph:
      • cephfs_ec_profile:
        • m=2
        • k=2
        • crush-failure-domain=osd
        • disabled
      • conf:
        • client:
          • client mount timeout: 600
          • debug client: 20
          • debug ms: 1
          • rados mon op timeout: 15m
          • rados osd op timeout: 15m
        • global:
          • ms die on skipped message: False
        • mds:
          • debug mds: 20
          • debug ms: 1
          • mds bal frag: True
          • mds bal fragment size max: 10000
          • mds bal merge size: 5
          • mds bal split bits: 3
          • mds bal split size: 100
          • mds debug frag: True
          • mds debug scatterstat: True
          • mds op complaint time: 180
          • mds verify scatter: True
          • osd op complaint time: 180
          • rados mon op timeout: 15m
          • rados osd op timeout: 15m
        • mgr:
          • debug mgr: 20
          • debug ms: 1
        • mon:
          • debug mon: 20
          • debug ms: 1
          • debug paxos: 20
          • mon op complaint time: 120
          • mon osd backfillfull ratio: 0.6
          • mon osd full ratio: 0.7
          • mon osd nearfull ratio: 0.6
        • osd:
          • bluestore block size: 96636764160
          • bluestore compression mode: aggressive
          • bluestore fsck on mount: True
          • debug bluefs: 20
          • debug bluestore: 20
          • debug ms: 1
          • debug osd: 20
          • debug rocksdb: 10
          • memstore device bytes: 200000000
          • mon osd backfillfull_ratio: 0.85
          • mon osd full ratio: 0.9
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 1.0
          • osd mon report interval: 5
          • osd objectstore: memstore
          • osd op complaint time: 180
          • osd shutdown pgref assert: True
      • flavor: default
      • fs: xfs
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(FS_INLINE_DATA_DEPRECATED\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • OSD full dropping all updates
        • OSD near full
        • pausewr flag
        • failsafe engaged, dropping updates
        • failsafe disengaged, no longer dropping
        • is full \(reached quota
        • POOL_FULL
        • POOL_BACKFILLFULL
      • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
    • ceph-deploy:
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
    • install:
      • ceph:
        • flavor: default
        • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
    • thrashosds:
      • bdev_inject_crash: 2
      • bdev_inject_crash_probability: 0.5
    • workunit:
      • branch: octopus-release
      • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
  • owner: scheduled_yuriw@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'mds.b', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8de45cc211a44036ac85f50e2bd80904
  • status: fail
  • success: False
  • branch: octopus-release
  • seed:
  • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
  • subset:
  • suite:
  • suite_branch: octopus-release
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
  • targets:
    • smithi006.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9vcUMZ6dz+Nw6WqFlQTXPYwekaNHyS9YPDeYA6I9XXlTQKOE6VJTQ+uj+NNdAmKsj60HRIlEf/HuNY7SiFsygl2OwTrLlObpvMTvSr6Xh7v1ufXpGCk6PV8LwlQZrwdOrmz47f+2hCRzqpM9NTkkUyzWo1qS0c0PmPB0NCjJ9YdhaQsON7tliFwc0cmc3b08jR3jwwBi32IDQ9IQGV6ZATS1hXKol6E/HTu42Gqdx1g4bA/9gtocCHZTyM4oC87c3nLG9iJNuVqWwEHSWboy9N2a9MWsSM5ZKl/GK4Pa8BvJT2CZceeTcaPtdqyhiKXJbef31co+mTcEaheVcb7Krergl6y3muGjkl3pdks/dJW1GT1wGYS1djng3hBI0Rs7JAbrECREuV2qJnuX2UXdl13IFNFnZALlHT/UXDXiH7g92GwjuNXJdpi7arCj3dsRHOI00+gs1xr/gtncZAb2VLQqsT/x6ToUhOVBIIVnh3yfKByViQ3E/kCXJPRygnN0=
    • smithi007.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDW4VfOc+KXO49Z4YZjZsrP532puEtzX2uhSFq92kmTCxz3RydhIUj9a0DnGDgf82SCj1S0uc5JTKmfhQzsIfDvT6E/cK+W8apvOKpvqyXDaYQg0lyNIQnJp7qel1G/FiDWogmWU6rhzeJnJaoFz+qkEd/ZtdFo3IZP4Qpmllnd2b9dyyEKk+yu2R+cMLRlW1As1n9O5O3KTGd2hpX5fFx2GCGwF9BUhM/voQAenhehGXKFfVLbUATm+JJ1iqFSffJmPZAONfYh6HHyH94iC8IFhrm+xv5N+MPM9yC4mYpihEv2ZSO0sthVkPMLq6WyZo/FWJTWGa80VBbE7rJPtDqjQ3RZ+DWdREK5wLdGn5qE8bDTyx+3FNdsicNbxVeSrqOHHtg3yzC983BdpJ6A5kqy62146fP/WPX06I7fnBtsNHHVZpczgHxv8YVJIsv2zY/teCYXisTIYH42/xmsjNkVFlG8Fj6qGdaYW/prAgpv9HkyXNf+5fXZRCECpzUgSck=
    • smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8rKKWUZfTeT9YJQRcUaBoCo28quhH8V+a4gC/PD/7pnFfHyXcrt6IP+I0ygdRU/G1QPX++tGDd9qF8yo1IVy1VKmo0KOvJjm3ecDT7zBGZe3NUL1NUSEkJEx7HSo70FuB/ATlAl3pjfDH0jJe1xNffTo5oLqh5iuy7hd68OjxSs2v5eL3Uu+qIIF02Z+Oi5fHnpINpbWr2OtjAbSOCA/3DyVkzqoIjft4hKBdkWYtJOfTvAm3r6wnDh72eUDxd2ZBGnC4wnrm2cEuWsjLZGlqI6oToMQZxaRc8FqsZ+pG/bx+5U1eOA65tPIqQIjh732gAFZpyuTjGLwcJnXg5WGItzrbPazmBQd93/1+6810gKsffYvvfCKnmMT/3pwEWo+9+GrFMXCGfO2lI4QvkcXJ6rAiqx1BCCbuufwFpa637faM1DBt6w5qITj2kbBPGZo6EVsdvuv3UC/4My1ONht4gyBEoU3FFmAEBsgoKQVGB4hCuhMF5G80ga0fiEOiKVE=
    • smithi071.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+Dr1T3iggcAArluEbOr9LW1+rou691gt/R7IBmifjJ4bmkcnUwS6Ho3wUiQtHSWiaoVR2MnY0g8U3tfkZ7rGdL/cDulW81km/Us3grefNbcyVtRuy4UGIQ8CKPzXbW/1gy4URj3njlJTNIGTMQ5aE2dICMIr23tqp3YYFMtIVb13IiOW5jwuhFAzDRXELSFquyjb6/8qxALXtsgT6XRZ+anw/lHDtJ2tTnHt/VaxhDfYqCxGqA5+FH7NuPELJ5dXV80PdtpHFL7Crj6OO8/cbWk8szsRDO7dWmZydO44PK1q/shaP7ghzCGjumNa7B3ErbFyDQ14yhb2ujU5O58jJDKW4co5pzzERjahNO3Pl5coVU7l+5Y/0hSDIp9I324DeIt2gjWzV1l+eaGIpQFeEYR/EV84Pl5qnpXnBkbi93fFnqs5Gf8CdhFdM0xNBCK0ZjOVKULr/fM1hYLKTBonAwaXOqB+OUBojPx8zb8QdAENKCQGRSJNoS/CSnAfe1rc=
    • smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGCU8jiUQ8MTU5FRCcLVrnER6h5cswDaHqT4MWX+7Hnw8Eq2/dU4CMQGh0jYeb2ONWdITmYBNq5//hO6oilAncMk92Kz9n9cqfn/o0VFGxXdu7dodvRyoV6UHAH4lHv9a7yL3mwVfZuc308/cxOp6Uyv0e2rqRctBvZGnYic24oZOw3mO2L1v1CrKJl84Ni9hBZK7Wd06Z0Rup6pjMJ73GrnwNhh8OzJTN8cjyhX87Ix76z1xehwqw8Vd9LqiBqf+fRiuo/LpaBshtW6L8POJnynHdBgSIDXjxCHsnsdmPKAydGGXg3PceGwmnX9b+ddoI/u5EIDh8/w/DiDcdZq2JVuOousUkHpf9wqdNHIsYLLhaJnr+C1pKVBxMCUWIjps2IFan3owUMYLHVbArUH110AMfkDZDj/BwcF1iY60cj5HJBz4ogKB66gYId4u22HuoNUlIugdcs/HJScPEPAbnMArlBSkxUqgQa9gbL0xZ4FEHyf3tIgBFyD1prVco80M=
    • smithi094.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiMX40ZoK89hzpJM+w9V4wnlUEHLcpfDPtR0Tyli6VGXYD+kH4oR51lnJsVNLMFOrC6Q8s+om75BQrNqTuftQ/JNgwr/JPxpNgADlEb/IQV7gTyjQh6bJm7C6IgL2xdDnCy7MnLdHQjrc7jC1+mMKlX12ke7ci4WSHBFElmdDLWb/lgXjo78M4QGx+ZN9h/plV0zymjmHH5xJYgwLqNn4cLZQwsw+xJEWNoe/f4sXd5rfbGLdRfatbGbpb8ji5tiOK5zIUtSDgd6zpGNnCMxa+LMmYHp2R4OgWy+CtnyG1s4hhfvrgaVKeCA4+Tj0EqNgZMU3OROLCF+UnubmAqXyyFpkTaDCsEm82Jz6DaoQRtoaWEehYWBTsoGUghY8twiPGTq57i+38qJoLuXVKi/ydusTS1DWjtkaIY60xpxDfaRPHjBJN5YhLqOem6PQ9f1fYQyg45JuaDHrx2H1av86oGwGoUtP34CP7Ylh7Y3JsV4XasR9fvQbs3OBZptpfjkk=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • client:
        • branch: testing
      • flavor: default
      • kdb: True
      • sha1: 7515a529973fc26f43af978722a22367693c4342
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
      • extra_packages:
        • deb:
          • python3-cephfs
          • cephfs-shell
        • rpm:
          • python3-cephfs
      • extra_system_packages:
        • deb:
          • bison
          • flex
          • libelf-dev
          • libssl-dev
          • dump
          • indent
          • libaio-dev
          • libtool-bin
          • uuid-dev
          • xfslibs-dev
        • rpm:
          • bison
          • flex
          • elfutils-libelf-devel
          • openssl-devel
          • libacl-devel
          • libaio-devel
          • libattr-devel
          • libtool
          • libuuid-devel
          • xfsdump
          • xfsprogs
          • xfsprogs-devel
          • libaio-devel
          • libtool
          • libuuid-devel
          • xfsprogs-devel
      • flavor: default
      • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_full
  • teuthology_branch: main
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2022-07-20 14:57:28
  • started: 2022-07-20 23:11:11
  • updated: 2022-07-20 23:47:29
  • status_class: danger
  • runtime: 0:36:18
  • wait_time: 0:16:56