Pulpito

yuriw-2022-07-21_14:04:31-kcephfs-octopus-release-testing-default-smithi/6942540

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2022-07-21 14:04:56 2022-07-21 14:39:42 2022-07-21 15:15:55 0:36:13 0:22:26 0:13:47 smithi main ubuntu 20.04 6 fail

Description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}

Log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-07-21_14:04:31-kcephfs-octopus-release-testing-default-smithi/6942540/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7a8fec3ff8d143e4adbce9761e306e6f

Failure Reason:

Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/yuriw-2022-07-21_14:04:31-kcephfs-octopus-release-testing-default-smithi/6942540/teuthology.log
  • archive_path: /home/teuthworker/archive/yuriw-2022-07-21_14:04:31-kcephfs-octopus-release-testing-default-smithi/6942540
  • description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/mds-full}
  • duration: 0:22:26
  • email:
  • failure_reason: Test failure: test_full_fsync (tasks.cephfs.test_full.TestClusterFull)
  • flavor: default
  • job_id: 6942540
  • kernel:
    • client:
      • branch: testing
    • flavor: default
    • kdb: True
    • sha1: 7515a529973fc26f43af978722a22367693c4342
  • last_in_suite: False
  • machine_type: smithi
  • name: yuriw-2022-07-21_14:04:31-kcephfs-octopus-release-testing-default-smithi
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version: 20.04
  • overrides:
    • admin_socket:
      • branch: octopus-release
    • ceph:
      • cephfs_ec_profile:
        • m=2
        • k=2
        • crush-failure-domain=osd
        • disabled
      • conf:
        • client:
          • client mount timeout: 600
          • debug client: 20
          • debug ms: 1
          • rados mon op timeout: 15m
          • rados osd op timeout: 15m
        • global:
          • ms die on skipped message: False
        • mds:
          • debug mds: 20
          • debug ms: 1
          • mds bal frag: True
          • mds bal fragment size max: 10000
          • mds bal merge size: 5
          • mds bal split bits: 3
          • mds bal split size: 100
          • mds debug frag: True
          • mds debug scatterstat: True
          • mds op complaint time: 180
          • mds verify scatter: True
          • osd op complaint time: 180
          • rados mon op timeout: 15m
          • rados osd op timeout: 15m
        • mgr:
          • debug mgr: 20
          • debug ms: 1
        • mon:
          • debug mon: 20
          • debug ms: 1
          • debug paxos: 20
          • mon op complaint time: 120
          • mon osd backfillfull ratio: 0.6
          • mon osd full ratio: 0.7
          • mon osd nearfull ratio: 0.6
        • osd:
          • bluestore block size: 96636764160
          • bluestore compression mode: aggressive
          • bluestore fsck on mount: True
          • debug bluefs: 20
          • debug bluestore: 20
          • debug ms: 1
          • debug osd: 20
          • debug rocksdb: 10
          • memstore device bytes: 200000000
          • mon osd backfillfull_ratio: 0.85
          • mon osd full ratio: 0.9
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 1.0
          • osd mon report interval: 5
          • osd objectstore: memstore
          • osd op complaint time: 180
          • osd shutdown pgref assert: True
      • flavor: default
      • fs: xfs
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(FS_INLINE_DATA_DEPRECATED\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • OSD full dropping all updates
        • OSD near full
        • pausewr flag
        • failsafe engaged, dropping updates
        • failsafe disengaged, no longer dropping
        • is full \(reached quota
        • POOL_FULL
        • POOL_BACKFILLFULL
      • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
    • ceph-deploy:
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
    • install:
      • ceph:
        • flavor: default
        • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
    • thrashosds:
      • bdev_inject_crash: 2
      • bdev_inject_crash_probability: 0.5
    • workunit:
      • branch: octopus-release
      • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
  • owner: scheduled_yuriw@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'mds.b', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7a8fec3ff8d143e4adbce9761e306e6f
  • status: fail
  • success: False
  • branch: octopus-release
  • seed:
  • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
  • subset:
  • suite:
  • suite_branch: octopus-release
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
  • targets:
    • smithi063.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkWkaS9J8ypSQo2FdyYXffPMToa80pZsDMZHa3Qif3ec68OZXJFToLnxRoNXFgyxUPDtorxCeIi6spNQ0NC/S6i0q0IcdZggEncSH53qG3AoWzHkrlEDWL8W5/4GXSMN4/MWgx2dTOdvYocSo+mAvaGcKbXDcmWuwXrbkrWpuS64+KQ/OAqtw4alTn3C5wu0RBmZqRbW06nspGuXv6pU1LpDPDYhZZ7Ebjktl0imwqov/LxRg7Nx0Wpevks6iA6kFZK9VW2GqujB4f1MAtxvlrpUGLTjB7fwglQG1FhLYNUDk5Nq5uaGrol3qwSS+QLtmRiohHqHW2MWR3q2pzbX6xybZy1gEOek51bhI/4P13MVRCdvVXgxh6WPQREnVA07fbuba5TqNfVhwpzHiX28RAw1Q2HlozEunVBsxRSRt3axur2bdBX7kg8aCjom+Uf7LLNiMqpyK0fO1/uLxJx9PP9gknmFFHdpbs9DFxg0RK+G/L+slol0yflR8Vvgki2uc=
    • smithi103.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpiu6ZEB++MvPYXKHcrxMqACQ1LwdQ3HN2iQo2FClj7PdK9A2qJHPlSwSBpCrhY/ou77ibpiVB7sk9wLrHj2FJZOIAzad/KmNorgRJ3t49c/Gs3qPb+Wi3jM7CbzDkezLU1vGFu94DV6+VClCyIdbAgXjjWxL8tnLLVSWiW54NAzcskE+q8ADSkp/oLdG1o2pwh/S4wXCdYuqCGBU0Fo1/LwUpRdGT/np7YxcZHUJAOJkUrQUE1iC4PnHbnN+g4IoDZy204pRFLY6tBFzdguou/RMn/f9CQL22WxO0JAudM0KicX/f28iT1qVLYl7K0RhxY6wntmZCU24tGI8+04l9tMTTDGjXg726DCPHlPcGyVPT0kFuj7+DruKWMc4F/KF4THASXmHHHct4YIbJJaRfdfWcSO3Imf26GdxjnpC1paQ0Taa/kozpsY94soX0mDkYr3cq1QsIJvS/pB1fbAmt3Q0lYiYpP1Ad/kcVIGmFWRajUg9eBDIQRzPxz+PVXec=
    • smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCid3IKv+B/oN3TdAxO9PBFV00qtW8x3sO9IxGrsi8O8silH0qnVLzpRMs9tDz1Cicb7P9qmRv+vuQSVayoX9W+fniuDMOJMLSrQipJxqjvyur18KYzVQ81/rDaPSz8tUxXv6YFp87s0Y8BpwqLHVJCJrWLK13uklkVIZcxawdw53gcOfjqwQzrxAhQ7NQgPz3h9QKyZbv4Z6TXdXAP5G5yozGNM+08ZnP90k9m7zIYKA2MBjr5tNFSwkYYdZi48IT6Vyfkeub8VqPgf4dr7IVNDumbw8md3tGgu77lzxl4Vo4xOVArip7y0pG/3uIijD+joMPjd/mIX0yJkKpHlNOrNjM6Oj4RZFQSmRV5B/q4dVtZPPnsVoB9oKB/3xTTNzjTMDzyH9UcZtQ73cWYMl8XgUtxweenhLGzXqD1xJVcEipgncDlE58X5WtutNylSzWG7mWPa0W9U6gX9EIDPSmLB1XjYHRLe3f+fLq5LVHJSRqw1yUjDvoRKlu2N2fVoss=
    • smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMJMflcpLZTvOhYinhsEJcDuzhnqKojdF7Vzde4LTlow+vxqD4ZwggQvkZ1wCZIOWRUKktkYo2wqeJkaOdcpcpt4M8lpbXLmJ/SqusEDb+YQ7fmCuVNuy8nqQP2G2CHdCOsMyNGHizCy47H7fwoxzHDWLXxodh/4Pbz9FMxua4EoNlzTdacH0Zjez8eeUe7dxGRRGuClvR6ZGDwcHw1kqNf4oiWZO+I08ZVzuOulrXY/afV4doV2Hl9WzY3c163SFypa89H90/DPeI2Hre6jsRJFt81+ftC1OKLfqaQTDEpvNpKw90Qfd2G0Hs321SoBS6e+xnqLKUqHXSbxMp/Zq7MK53duLbjYJLGrmTeyR8iXSinQOhSDkC+asKfQqTLb/qYZFHYC9xAc+9QujXrRaak5Ci53sH0e+lMKVVkvd2TWrEm2JXrYjn/R758M4M1bLHFRyBIlZNvX9mHLpk7ioW5C5EeQvZyw+FSvEjcLBRMIqzCc1x9mkrfWfQGuaW840=
    • smithi167.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1LdiJMkte0ZB5mmpqW3GKzPlbc6/lBs/Njf6eishFnBYUSkID1UOPYVK1zKx805tLSLnO/HJfDZyVg+ODhHXWxO39HmpgTAedmZO+E4TGzxBVugIRWgFD4lsM29sCwvSSISoDCY2nA6VPVxnMCBwz59S9F4VzIfmwTkk44iB/SlT+ltNbWPmWR0sZxnc/1k+MTAY1Kf05d9OHIw59QPgAU0aKiM/m6Vz9/I+4DW9Pojz3lcoM5YQtQEs81ra1hktX/F6Hyb/iknw4uGF9a5veaUO4moWy36XOuGyO6U3oyUWCQxxCrfqWg4aFFo9q73GG4qVPbp4yoWlgZwDyHwfHric3MyE5O2uclpfCbPQg6pPZilPJ326915IMB+M20lDU/BwJ+hqyVgMPe8cK3IZOf0Gdw52m/oCTXwVqreKfJXI+qi+/iFE97GmNUE/H1L1la77vc/0QLhdWKV2VOza2fwgo1+ey99wOZd3Bq9mrf11u+vtRZWHixXmGLqwY9z8=
    • smithi175.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQfh1QJVWNAAbcOrKk8QF8Ho0PTdUPmHJp988HX9PRCxaQO7irzs2LDb68hTaPeUMv9PTXefMIjZtzfmkqymUmGydTM1sD/qLVHAhixHOQYpjAS2Bq7cA0CEPcuDvH0qD6sg0KEnAViO5EathoYhsE3lK/o9YGNtLUxsCKdOlADLWUDZtupn+zXVgF/RGVOXXMqxtJUFgSfwW3O014+dlE9YGOeYFFxHwhbl+HPpxU3g48A+b20/Jcs+0YYApZwDVPXaagNU90kswakkKKzp7lyB84AEFaGEvmshCgKf4HmVVDqNBA5cFH+7Cxnt5mJneBd1pzET+IjsfRyyZhcAYmxHA1Kay2Tumyo6ByGdb0u+WE1w3GyDTM9rYal7si9wg6w1KJrHEJYVRUcZjPbdCDfmkFbxGpdeDB96qRivW/f9bKluM2Ee1E8PMwza/gd3BBwWprFMcWdBSGekoG/thX/rGSbAqr3dYvw5555gsVQmmKLfYwJIPoumOUJeg/84k=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • client:
        • branch: testing
      • flavor: default
      • kdb: True
      • sha1: 7515a529973fc26f43af978722a22367693c4342
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
      • extra_packages:
        • deb:
          • python3-cephfs
          • cephfs-shell
        • rpm:
          • python3-cephfs
      • extra_system_packages:
        • deb:
          • bison
          • flex
          • libelf-dev
          • libssl-dev
          • dump
          • indent
          • libaio-dev
          • libtool-bin
          • uuid-dev
          • xfslibs-dev
        • rpm:
          • bison
          • flex
          • elfutils-libelf-devel
          • openssl-devel
          • libacl-devel
          • libaio-devel
          • libattr-devel
          • libtool
          • libuuid-devel
          • xfsdump
          • xfsprogs
          • xfsprogs-devel
          • libaio-devel
          • libtool
          • libuuid-devel
          • xfsprogs-devel
      • flavor: default
      • sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_full
  • teuthology_branch: main
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2022-07-21 14:04:56
  • started: 2022-07-21 14:39:42
  • updated: 2022-07-21 15:15:55
  • status_class: danger
  • runtime: 0:36:13
  • wait_time: 0:13:47