Pulpito

teuthology-2021-11-22_02:01:03-rbd-master-distro-default-gibba/6519166

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2021-11-22 02:03:37 2021-11-23 16:55:19 2021-11-23 17:30:44 0:35:25 0:29:14 0:06:11 gibba master rhel 8.4 3 fail

Description: rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/ec-cache-pool supported-random-distro$/{rhel_8} workloads/qemu_bonnie}

Log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-22_02:01:03-rbd-master-distro-default-gibba/6519166/teuthology.log

Failure Reason:

"2021-11-23T17:24:47.110473+0000 osd.0 (osd.0) 3 : cluster [WRN] slow request osd_op(client.4430.0:117149 4.3 4:f1889b26:::rbd_data.11442aaf1928.0000000000000135:head [stat,write 3047424~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) initiated 2021-11-23T17:24:16.923658+0000 currently delayed" in cluster log

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-22_02:01:03-rbd-master-distro-default-gibba/6519166/teuthology.log
  • archive_path: /home/teuthworker/archive/teuthology-2021-11-22_02:01:03-rbd-master-distro-default-gibba/6519166
  • description: rbd/qemu/{cache/writearound clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/filestore-xfs pool/ec-cache-pool supported-random-distro$/{rhel_8} workloads/qemu_bonnie}
  • duration: 0:29:14
  • email: ceph-qa@ceph.io
  • failure_reason: "2021-11-23T17:24:47.110473+0000 osd.0 (osd.0) 3 : cluster [WRN] slow request osd_op(client.4430.0:117149 4.3 4:f1889b26:::rbd_data.11442aaf1928.0000000000000135:head [stat,write 3047424~8192 in=8192b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39) initiated 2021-11-23T17:24:16.923658+0000 currently delayed" in cluster log
  • flavor: default
  • job_id: 6519166
  • kernel:
    • kdb: True
    • sha1: distro
  • last_in_suite: False
  • machine_type: gibba
  • name: teuthology-2021-11-22_02:01:03-rbd-master-distro-default-gibba
  • nuke_on_error: True
  • os_type: rhel
  • os_version: 8.4
  • overrides:
    • admin_socket:
      • branch: master
    • ceph:
      • conf:
        • client:
          • rbd default features: 125
        • global:
          • mon client directed command retry: 5
          • ms inject socket failures: 5000
        • mgr:
          • debug mgr: 20
          • debug ms: 1
        • mon:
          • debug mon: 20
          • debug ms: 1
          • debug paxos: 20
        • osd:
          • debug ms: 1
          • debug osd: 20
          • osd objectstore: filestore
          • osd shutdown pgref assert: True
          • osd sloppy crc: True
      • flavor: default
      • fs: xfs
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • but it is still running
        • \(OSD_SLOW_PING_TIME
        • overall HEALTH_
        • \(CACHE_POOL_NEAR_FULL\)
        • \(CACHE_POOL_NO_HIT_SET\)
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • sha1: 17f4da9df2ac8354b028ed986616b2b7ae56f05d
    • ceph-deploy:
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • osd objectstore: filestore
          • osd sloppy crc: True
      • filestore: True
      • fs: xfs
    • install:
      • ceph:
        • flavor: default
        • sha1: 17f4da9df2ac8354b028ed986616b2b7ae56f05d
    • selinux:
      • whitelist:
        • scontext=system_u:system_r:logrotate_t:s0
    • workunit:
      • branch: master
      • sha1: 5b1eae6953701f0f06da5da195662682789b1eb8
  • owner: scheduled_teuthology@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mgr.y', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
  • sentry_event:
  • status: fail
  • success: False
  • branch: master
  • seed:
  • sha1: 17f4da9df2ac8354b028ed986616b2b7ae56f05d
  • subset:
  • suite:
  • suite_branch: master
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 5b1eae6953701f0f06da5da195662682789b1eb8
  • targets:
    • gibba016.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDRSO0kk4LEO6YkY1MXvCEgb8UyJW1Wj37kPLJ6826SsiejIax/bf9eAJ1fWSZdPxQlThf2PFKIwQNiaj5eLOH2LXvW8gxQukAuLZGk3bo6UeUt4aRm9/HUBd2SgqEJiZpJCpmVsGqTf7YL1B/UGtXHdh1Vg3Jh2sodI9rrzstNp43679L7mVHHpTX9j+PcQLrGDplXPkm6jQ0usk8mNe+Eh6BS9tYtuLW2MNzMvvv+2YPwxM+TGECQDFY+4cyLvfDca84BTtICGN0tsp2RWWg/H6iZlqwqt5W5ToaSwDTvoEfvScM3D479uqKLw7KfHhq2xJMHfOPcj41W4/eo9MYLT2+zERYSvh2NDLHruM9hrsYDJe4YXSJHumtTQkxJ5gf79V/blICfkW4dcWI17W435GY4sSMRFQdN3C45jAPnpYkRsajSR2xHClOx/L7Bq9c0vdY3LHaPwoK9aT9OOT6s1dBKbekPi0rFAQ5Ddn8t1hHH8h/MljZd/4qs06XF4m0=
    • gibba017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrRtQvtuf2EiAB8iIHjNcpBTTGD0kjsRzMP6ccNS8eC5Sv6n3MLvgsQmjr0I2d7NVCNbrQ7/mnevKJmjawtv9QP//MGypAkJ3FlxLUyrHj5W+QRN0uG+kwlmD84LIaXBJHjK4OvV7radOqVfGB2Ozn8IIucMV2xZ7wIgmeSnTmP5SB9ifznmanxpI4f5huuEJskdd0GVdbut911sx+9EoEZjDNR3w8qnQnrfrpKBDiXlHAkA/x4AtSQH4qSGwZr/cKj4uoBRMiXoizwlhU8bMdLSov9WHhmkkWRUbQrPd4+NDXufNsxexzonODgkBXdSjdSyE1VfBPf5/1tQwL43bsHcPO0Y5ll+KbjMP3dZWgfeFzncrVnpNIfNamn8hyFPSHT9Y5AGxbfpvjN/JFJDcGrX/G1beYB3CsADpjrf0SVJAX3X944kbavYpQRTO0DET/Fp9fuHtqrjSOG431LjMWqV42SS/u6I93PJ4bFhvk58mwyf+seApDa4msOynzabk=
    • gibba040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKD8DOX3EV/crVdeu1Bq8qLna3fjaoZ8raT8uiy12rE7PF4yADN+upWO+1EsNMlZP3GwK2+oLLdTk8bUtARnVUNTNbzy2RgXuooDk9RkGY0yH7glHqHVN/pUtQkQiGGsMNmdK2E1djeFVWxSWTINsDy8LW2wU2BVYUWxyoF1auT+4GojoeT+WBbB8UNtZY9lC47ZiXsLmpJrOocVVYvJVIJBmmofsnmJEdFFASTaNui+Qj0S1yiRxzsv4s713u1Pa+lW+urc61XiLRbFG6z3t5TWIgGRRp0rNyD22UJk3eG3WhJ6MplhFAqcrAeCdtitd0/X4so8NJVFTYNx5vgEtAka95QDOlCNlXUi3Up2q/RkH3BEDWrmlrxQ6Hzw+cmGhgco7DLkATt+HOjBbqXfOPn5rX49NH66qGL6sOviGRo638lDXP98fRx+pGA7H41Q6OdfrA89TnUuLMXsr6fsrLzFuJ4PsdVIJPCzEXHYh1+JXBpRBcJPsID/XEGjatPgM=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • kdb: True
      • sha1: distro
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
    • ceph:
      • conf:
        • client:
          • rbd cache: True
          • rbd cache policy: writearound
          • rbd default features: 125
        • global:
          • mon client directed command retry: 5
          • ms inject socket failures: 5000
        • mgr:
          • debug mgr: 20
          • debug ms: 1
        • mon:
          • debug mon: 20
          • debug ms: 1
          • debug paxos: 20
        • osd:
          • debug ms: 1
          • debug osd: 20
          • osd objectstore: filestore
          • osd shutdown pgref assert: True
          • osd sloppy crc: True
      • flavor: default
      • fs: xfs
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • but it is still running
        • \(OSD_SLOW_PING_TIME
        • overall HEALTH_
        • \(CACHE_POOL_NEAR_FULL\)
        • \(CACHE_POOL_NO_HIT_SET\)
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • sha1: 17f4da9df2ac8354b028ed986616b2b7ae56f05d
      • cluster: ceph
    • exec:
      • client.0:
        • sudo ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd m=1 k=2
        • sudo ceph osd pool delete rbd rbd --yes-i-really-really-mean-it
        • sudo ceph osd pool create rbd 4 4 erasure teuthologyprofile
        • sudo ceph osd pool create cache 4
        • sudo ceph osd tier add rbd cache
        • sudo ceph osd tier cache-mode cache writeback
        • sudo ceph osd tier set-overlay rbd cache
        • sudo ceph osd pool set cache hit_set_type bloom
        • sudo ceph osd pool set cache hit_set_count 8
        • sudo ceph osd pool set cache hit_set_period 60
        • sudo ceph osd pool set cache target_max_objects 250
        • rbd pool init rbd
    • qemu:
      • all:
        • clone: True
        • test: qa/workunits/suites/bonnie.sh
        • disks:
          • image_name: client.0.0
          • action: create
          • image_size: 10240
          • image_url: http://download.ceph.com/qa/ubuntu-12.04.qcow2
          • device_type: filesystem
          • encryption_format: none
          • image_name: client.0.1
          • action: create
          • image_size: 10240
          • device_type: filesystem
          • encryption_format: none
          • image_name: client.0.0-clone
          • action: clone
          • image_size: 10240
          • image_url: http://download.ceph.com/qa/ubuntu-12.04.qcow2
          • device_type: filesystem
          • device_letter: a
          • encryption_format: none
          • parent_name: client.0.0
          • image_name: client.0.1-clone
          • action: clone
          • image_size: 10240
          • device_type: filesystem
          • device_letter: b
          • encryption_format: none
          • parent_name: client.0.1
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2021-11-22 02:03:37
  • started: 2021-11-23 16:55:19
  • updated: 2021-11-23 17:30:44
  • status_class: danger
  • runtime: 0:35:25
  • wait_time: 0:06:11