Pulpito

yuriw-2022-02-14_16:52:17-kcephfs-octopus-testing-default-smithi/6682171

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2022-02-14 16:52:51 2022-02-14 17:32:22 2022-02-14 18:01:47 0:29:25 0:17:02 0:12:23 smithi master ubuntu 20.04 6 fail

Description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}

Log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-14_16:52:17-kcephfs-octopus-testing-default-smithi/6682171/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=55b7379aa0694126b900a7f737e9c107

Failure Reason:

Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/yuriw-2022-02-14_16:52:17-kcephfs-octopus-testing-default-smithi/6682171/teuthology.log
  • archive_path: /home/teuthworker/archive/yuriw-2022-02-14_16:52:17-kcephfs-octopus-testing-default-smithi/6682171
  • description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}
  • duration: 0:17:02
  • email:
  • failure_reason: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)
  • flavor: default
  • job_id: 6682171
  • kernel:
    • client:
      • branch: testing
    • flavor: default
    • kdb: True
    • sha1: 4c199e30549fb37d1dea5b2ed8d72a17b49671aa
  • last_in_suite: False
  • machine_type: smithi
  • name: yuriw-2022-02-14_16:52:17-kcephfs-octopus-testing-default-smithi
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version: 20.04
  • overrides:
    • admin_socket:
      • branch: octopus
    • ceph:
      • cephfs_ec_profile:
        • m=2
        • k=2
        • crush-failure-domain=osd
        • disabled
      • conf:
        • client:
          • client mount timeout: 600
          • debug client: 20
          • debug ms: 1
          • rados mon op timeout: 15m
          • rados osd op timeout: 15m
        • global:
          • ms die on skipped message: False
        • mds:
          • debug mds: 20
          • debug ms: 1
          • mds bal frag: True
          • mds bal fragment size max: 10000
          • mds bal merge size: 5
          • mds bal split bits: 3
          • mds bal split size: 100
          • mds debug frag: True
          • mds debug scatterstat: True
          • mds op complaint time: 180
          • mds verify scatter: True
          • osd op complaint time: 180
          • rados mon op timeout: 15m
          • rados osd op timeout: 15m
        • mgr:
          • debug mgr: 20
          • debug ms: 1
        • mon:
          • debug mon: 20
          • debug ms: 1
          • debug paxos: 20
          • mon op complaint time: 120
          • mon osd backfillfull ratio: 0.6
          • mon osd full ratio: 0.7
          • mon osd nearfull ratio: 0.6
        • osd:
          • bluestore block size: 96636764160
          • bluestore compression mode: aggressive
          • bluestore fsck on mount: True
          • debug bluefs: 20
          • debug bluestore: 20
          • debug ms: 1
          • debug osd: 20
          • debug rocksdb: 10
          • memstore device bytes: 200000000
          • mon osd backfillfull_ratio: 0.85
          • mon osd full ratio: 0.9
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 1.0
          • osd mon report interval: 5
          • osd objectstore: memstore
          • osd op complaint time: 180
          • osd shutdown pgref assert: True
      • flavor: default
      • fs: xfs
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(FS_INLINE_DATA_DEPRECATED\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • OSD full dropping all updates
        • OSD near full
        • pausewr flag
        • failsafe engaged, dropping updates
        • failsafe disengaged, no longer dropping
        • is full \(reached quota
        • POOL_FULL
        • POOL_BACKFILLFULL
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • sha1: 91f471b19a8436443eb4f75ebb3d4414ee623c6e
    • ceph-deploy:
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
    • install:
      • ceph:
        • flavor: default
        • sha1: 91f471b19a8436443eb4f75ebb3d4414ee623c6e
    • thrashosds:
      • bdev_inject_crash: 2
      • bdev_inject_crash_probability: 0.5
    • workunit:
      • branch: octopus
      • sha1: ace4ea5047df4a17fc495b72a8b6489941894e94
  • owner: scheduled_yuriw@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'mds.b', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event: https://sentry.ceph.com/organizations/ceph/?query=55b7379aa0694126b900a7f737e9c107
  • status: fail
  • success: False
  • branch: octopus
  • seed:
  • sha1: 91f471b19a8436443eb4f75ebb3d4414ee623c6e
  • subset:
  • suite:
  • suite_branch: octopus
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: ace4ea5047df4a17fc495b72a8b6489941894e94
  • targets:
    • smithi006.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVQjUfVvIyI4SAXRauqrCrAQzHT+d9sLbiXEIG853o2c5LpTUfUp7aztW2KkKTFFtYIXx19Ox2IUzAhy3N4xu58sFZ6Ae36SY03dzCR/1kPaI3qBHC5T/+qKul9Rb1o+lGEgXYf2Ar2FiD03SrBy4EnNKwUwODDMRwB7qusFVj5rpDgh4E29t07zC8xx3Gkp1LQfBySKIcPhJBe2nVNtpNJia9dgDtvNK9n25208uYb8h10xYIPA577k6iZmD4OGPyxIRoso6AN1xzUmelYz7d0Gim4KeM5Cvip46xqEsFxHhXQ9bZag0MJLC8kKZvHYIACGGalhMFn2XQfCbx2Z9qgcwZiCu6Ncgdsj7swehSR+9Rjq8ajluzNzS/0plTCo9Qg7G8v86mGsMhmgeoinFK8U+2DH8wQjvnFrjizir7X8P/8xvUOF/63gOx+DSSnJic9AwRiWcC6Eus0MbiVExHroELuwNU4LcyeoCILdQZDRmXIc4xNZgqLUK07HCgk4U=
    • smithi027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgIFtryfsy/SB4lN8miP02nlbgny24a7fNFIf4aXBsR2uzDEHumeoIGoE+OI+vXTzKZyEo01Cvmmkue+KrC1lA+jOIQpKOPfLxRJwYikH0A1l/3YU6pEb6WXiDQkxOPubMBVi3lAF341SalVOD1C9aFDs0wR5Ro3Z9J2MyIOl4x3EkLHxqIZW390NJ4Kd4uLQF9chfYEJUaWb6k1XKclDw9X4OnZftCLTWEfSOqgf38zpPW1wetck1eWwT9mwVebSBEk9cPGsWn3YBnA7xOb7kIMeB5Us+ZWQD58ZD/OxNpB7BCI6P5CLeES8LiBtYips1sBUKT6svrIudAQu5hRoSBgifTJZwFtpFuqYoZ/mQ4W0dSRFmp6rD2ciVdfVNIPOkp3UZ6UH4V0pZJu+p8FBUICzxnjxzZeeumW0PeGQ6u9lQw28PnwqbT06tiKn8S9od9id2pyJ3j8dZefYdo4M41vC4r95xvDFyzqJCwchClz94u4S3ESKGpGGZwtPZPdM=
    • smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOrp1BGxER8RI6LDBuqD/6upGfQIaNbNFr3e5j0PLRBjwHWr1Im4TRMpk2bKUdxj0c8BZ6sSBg5aouzfLfaI9+aOLbjEmT8lWh2RrPz0aWPZF44NFwidSgGOQ8GWJ8mkiOafBsE7SLeK4hFaQ3PVOW0MOcrmhdY9Uz7StuwmpOgmgxnzuPVOL3dVTanYdEY5iP/FxbLP46mDRH/J8llwAceBa8IxI1f2LNyjsicB4Jv7rEGTDIuVMUVeVA7QgRZCl25PSCVvsVC9KTBZ/vxxWRraVaKniIGi6i820wctfu1Ja0rpq5XX7eVQWijKPYv+dQu2S4CP49qPlMSKLtQM/cq28pgbP5f7Gfdief6CAKDnLknnIK3uJuB8fiXFjkE8JwsKwvPwNcrpR5SWBpiO+mvWBFwAKvQAWarGR1Fy/sLjEEoTrY6yGa3B9a2jzdCg9XJUOaAjxTPv3ravgCeyNueEwW/nDTVtQH5OAoe9abL7IyAN3MWJlbzcn0AGo62fs=
    • smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCobTi68sBSfzaL7nc3wRbD11BIl10O+Ttd9Tr4oYf4qmZ6a8Lf6kBYOwxbyClDL3mNJtOaR/bikEc2SyZ+1wqg7qKKilDt4PBWgGYxv09I5jYuGjy+1XImO9mWfwIV9yGO6q0Jo/1aAnwH54vyR5Gz8LyoT4wjLSK+W7qpTYa33o+BYETak9ZjoPL10Jvh7WKVGMxlVkDXur//uJc/d/CufM4UPieHXrEQJnd/8R7jDyrH2QFoVMIOs82SFNtqDstAd2Jy95s9dsCjQTAOdACnkxvTjMdHI+j3pIREmbBstQTtjekbVy4Vyqpmk4huDwWt+ayABf1dPjInAFfQA5KeufDBLE2oUiQNRXIPnUBlUnxNqQQo4gLEShNVyulP3orfbj6ErdMfyVLHS4q7GLr36GLu0Nx0ugMEYxMz6iGL2geNOv7uU69sxshvtZRzgaAAefEqVZ9G27XJFgXAX86Mut0FcrSIDOylIhxT9l5iz1Vp009rsjFvjXQh2KrYUmk=
    • smithi184.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkKpIx+vnqAS91bso+ktaEwsuqPq7Hqdhz4/KTs0o3hrVSHxkaLwfBCHoS5nHiuJ0Tb4lceZ7zN0ela994mT6s4DlquKHDIyKorgcIHdFKSiEPzQEVMzxqDdIFJO/WsaDZp1peVYtuwsw6yIZlKHz6EGrt14IgDRH/v0az4dIKTKnGeely5w9A5w3QYe5i2NN5wUOkzWIsy6fFJExGN0gr2l91hCpVSu7iApNObdqT+WKBOWyZeol49T7OBSXu6A8F1u3rZuo+6/06WqEhhUyRfYhHc4YmxEjO2EiCvND5MdaMHnL9bVYLjxUFMqGXDqeNg8Zp/iRsPPDoSvfKho9WFYY/QFk04bWZlBOsNE+vYOHrU9FsP61fP0CVctOp3HlUZZAT8B3s6Gpt5jZx7wpv7dRZIPkAq7Ji2yCVTQr0wS4//Ds6S2hCDbAKvYaDJ/ZKeFHkjMkvAc3TMEwD9aSFFgTzLy6vOj96nXG0U0oJtjOCDsbIRCdnvJKnhz1EeJM=
    • smithi190.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfTe5+jxRGVmFJ1vCo2qvPCXXWR9D/k+PHQC36bHjHiviyBENhMDCnpvcji9D4d5uMRh/3dQhvJTb9FmUhnMF3vnSWwyzTZE13VLgNrVU0bIL6UCr7BJhehtldlKeViob134VX/czvMHfzZi2ze0kbO63E9gO55pEN+B8wb+x1AayKsMYQV3A7kbs3qEr9yBaT2x2CF6hVqxYXMMBNMlSfiL8CzacrrSHZUJ7EJzQ9QqvZrtXYRML9XgYLOQxwEliSVSapsJJLiHGArlwNbPAUXprsN/i6AkXl1vWe+tAKi+bWMFu7pFrWslC69gdQAGNDiM9MVzdK6GeIOpNCAp4DfV1t8TofARKkGI1SHoVFuH2vDU8YqlUzaLNNdguD5IrEQVOLLNQeXC3CxdURQwsnYRk8o/gElCjXnrAJLi6g7eEaapblnn5fDJG4Gikrq6hb0yaz+r1N03dWm2lsAIyNDlR9B497siTKJfL8GH7nd+VeDFRq41P4IX4Nn8QwLG8=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • client:
        • branch: testing
      • flavor: default
      • kdb: True
      • sha1: 4c199e30549fb37d1dea5b2ed8d72a17b49671aa
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
      • extra_packages:
        • deb:
          • python3-cephfs
          • cephfs-shell
        • rpm:
          • python3-cephfs
      • extra_system_packages:
        • deb:
          • bison
          • flex
          • libelf-dev
          • libssl-dev
          • dump
          • indent
          • libaio-dev
          • libtool-bin
          • uuid-dev
          • xfslibs-dev
        • rpm:
          • bison
          • flex
          • elfutils-libelf-devel
          • openssl-devel
          • libacl-devel
          • libaio-devel
          • libattr-devel
          • libtool
          • libuuid-devel
          • xfsdump
          • xfsprogs
          • xfsprogs-devel
          • libaio-devel
          • libtool
          • libuuid-devel
          • xfsprogs-devel
      • flavor: default
      • sha1: 91f471b19a8436443eb4f75ebb3d4414ee623c6e
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_full
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2022-02-14 16:52:51
  • started: 2022-02-14 17:32:22
  • updated: 2022-02-14 18:01:47
  • status_class: danger
  • runtime: 0:29:25
  • wait_time: 0:12:23