Pulpito

pdonnell-2019-08-28_04:00:50-kcephfs-wip-pdonnell-testing-20190827.232557-distro-basic-smithi/4258002

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2019-08-28 04:03:37 2019-08-28 14:35:06 2019-08-28 14:57:05 0:21:59 0:00:04 0:21:55 smithi master rhel 7.6 6 fail

Description: kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_7.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/strays.yaml}

Log: http://qa-proxy.ceph.com/teuthology/pdonnell-2019-08-28_04:00:50-kcephfs-wip-pdonnell-testing-20190827.232557-distro-basic-smithi/4258002/teuthology.log

Failure Reason:

Command failed with status 4: 'ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi072.front.sepia.ceph.com,smithi136.front.sepia.ceph.com,smithi050.front.sepia.ceph.com,smithi014.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml'

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/pdonnell-2019-08-28_04:00:50-kcephfs-wip-pdonnell-testing-20190827.232557-distro-basic-smithi/4258002/teuthology.log
  • archive_path: /home/teuthworker/archive/pdonnell-2019-08-28_04:00:50-kcephfs-wip-pdonnell-testing-20190827.232557-distro-basic-smithi/4258002
  • description: kcephfs/recovery/{begin.yaml clusters/1-mds-4-client.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} kclient/{mount.yaml overrides/{distro/rhel/{k-distro.yaml rhel_7.yaml} ms-die-on-skipped.yaml}} objectstore-ec/bluestore-comp.yaml overrides/{frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} tasks/strays.yaml}
  • duration: 0:00:04
  • email: pdonnell@redhat.com
  • failure_reason: Command failed with status 4: 'ansible-playbook -v --extra-vars \'{"ansible_ssh_user": "ubuntu"}\' -i /etc/ansible/hosts --limit smithi114.front.sepia.ceph.com,smithi072.front.sepia.ceph.com,smithi136.front.sepia.ceph.com,smithi050.front.sepia.ceph.com,smithi014.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml'
  • flavor:
  • job_id: 4258002
  • kernel:
    • client:
      • sha1: distro
    • kdb: True
    • sha1: distro
  • last_in_suite: False
  • machine_type: smithi
  • name: pdonnell-2019-08-28_04:00:50-kcephfs-wip-pdonnell-testing-20190827.232557-distro-basic-smithi
  • nuke_on_error: True
  • os_type: rhel
  • os_version: 7.6
  • overrides:
    • ceph-deploy:
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
    • workunit:
      • sha1: 6bb88e0d90bf75b204b8d60ac680e5fa340e6ecb
      • branch: wip-pdonnell-testing-20190827.232557
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
      • fs: xfs
      • conf:
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • osd op complaint time: 180
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds verify scatter: True
          • mds bal fragment size max: 10000
          • mds op complaint time: 180
          • mds debug scatterstat: True
          • mds debug frag: True
        • client:
          • debug ms: 1
          • debug client: 20
          • client mount timeout: 600
        • global:
          • ms die on skipped message: False
        • osd:
          • mon osd full ratio: 0.9
          • debug ms: 1
          • bluestore fsck on mount: True
          • debug osd: 25
          • bluestore compression mode: aggressive
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • mon osd backfillfull_ratio: 0.85
          • mon osd nearfull ratio: 0.8
          • osd op complaint time: 180
          • bluestore block size: 96636764160
          • debug filestore: 20
          • debug rocksdb: 10
          • osd shutdown pgref assert: True
          • osd failsafe full ratio: 0.95
          • debug journal: 20
        • mon:
          • debug ms: 1
          • debug mon: 20
          • debug paxos: 20
          • mon op complaint time: 120
      • sha1: 6bb88e0d90bf75b204b8d60ac680e5fa340e6ecb
    • install:
      • ceph:
        • sha1: 6bb88e0d90bf75b204b8d60ac680e5fa340e6ecb
    • admin_socket:
      • branch: wip-pdonnell-testing-20190827.232557
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_pdonnell@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'mds.b', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event:
  • status: fail
  • success: False
  • branch: wip-pdonnell-testing-20190827.232557
  • seed:
  • sha1: 6bb88e0d90bf75b204b8d60ac680e5fa340e6ecb
  • subset:
  • suite:
  • suite_branch: wip-pdonnell-testing-20190827.232557
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 6bb88e0d90bf75b204b8d60ac680e5fa340e6ecb
  • targets:
    • smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcx1vYo+rkrbdcPmddHQ0CKE9zbXmvZD0bptG6I89rUL4X6fqIuoNMoNOHOHGtqqPDkrzxADllxNR5EF/N/sAV7LWprvSQg8uUdtmvVENYntQcJeIiawikcrKSRtOqVkqK2Dpq4908Fy856uvqtainb1ySwrJnXm3PQvAC4B0blMP4kelImsRThRjoPmdCzg5LrrTvy+jl5MsSi/Rvzx59P0xp7sPs60vPP2bA4a5CkSJ76IGnK2UHUf4Do49BO81lmzD+7JQH68QB9bdw+7A+2ZACML6W65ObRW2hCqhqZ7IL5KIK3HRJIROYq5P1JAyJVTFFRywTL6++dnkdg7dB
    • smithi050.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCRjiUIONUo7dFZAtBOb4u3/hPliu+msVugfaBcJfSGJa3OhsBRkd4vhZ67J0HdrKB6ued5EeX6gFjU9VARCDCDQMlUGeYBU9nrNA/ITc6w3mGfB3yoH0N78sb7OTxdvqMAFjhiFL81DILXo0K3KSY1P3GfxwFtBEkuBfOJRzZvfXesCS6gQeptHGg3EOL59mP9S5F6ePuAB1Wd+q+AGMQjWIKBhRvCw1WIK1MgcDUn+KEDwbflI1bPJvwF6hkzbiedzXAt7UgKHxG2Q1Xljq+a3JU17/eqdpEqQi2v/bZoP+9+IUHa6mWpo4rcnS0UtSzC5zr+1CY7xSU633WDBNL
    • smithi114.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVuIaP2J22NHdMSQWxJpkY5R03ZPe6y/kZbEcz0zh0mS10GBVLZiuB6tDwoUS6jd6rHbtev7glY5wk+N1Fa/rvO4CymCnlNAXj2608IS/iZySifjH50yEIcFx2kGhFUVBHfmNdoSJfpSFRVx5njMu2K1qjtsxUqIUUECtUhfz057Ie4gbu0Du9AErDavgw56RUovfhMUcEMgieiznepC3Ddq1hbn7zGADWp8xU6PS/XM4kBWxc7Fl7oi+0miPMBmImdzTU8sZp90FfocXxq3r5/z0CG51cVsk3f4WO41JDeZRo0ejoGlfG3j/IiVI83rPCIRR+CGDp8H90+t3ncaOF
    • smithi193.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPq9RxcIoX0PBBsYolUtNJW4cPzNJ3uu/xpH8DfavLBNE7j4J/bZoY0gojFY9jsa3zOk2aZBYKmB2tJRlAw738UuDjtU0dEljXGJzBpGZm6e5Jw/otAMueiR6n5k7izUB6+V1nba4e7oZGehNsh4vY85pq2CMzn999UTEiJfXxTZsl531zWGqyy2v5Va6+PSp1hGKbDLyJddnIqBR+ktES7163vKe+0lnBguFjw7xwHlNtsEgD3rsnTvP9L5NcfUHfi6IipG1IFNLggElK2GjdQI/KciWeEW3YUutmJvJhqkJQEgTGphllruzg0IFoWlT/Jgv4Fik/ZKBMz1Wzn9Bj
    • smithi072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDsfWqYUhE+W1qtmahpvIkYCvlU8Lhb4W21Lv6G5s3LfwKXUvCTSOhr8BMy+F1daARwhNVPJ9T0TliJ+XvEIxzPx1Z+7tHxS89FZli4P5SC7xObfBKO8X3q/v13oXAUArQTLcNUngYox3TajJXs8KxjnCWPnCQiK2JCNP/o5OLHEL54Ep5RfjUAcR9dUpuD3T3f6hu3yPMjXbDUQ/QbJ7p3M1UFe/UDaJnrGmjuDMOP9aXvk5ePx124opIBAHiAL8VJlk/QXg2rLovHt3wLb6h2if9YDhmaoMYsSmkzpCsX9SC7YcWUYWAmIiOfsutQNFHPMH6X/F54M5qN63q/E10f
    • smithi014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHWoJHYVjL6WKHgusretNHp8KoNreJ2QMhtuVqUfA0FZqKN08RG9PRC/ML7DoxtSIYOLdu5g6sx1e1y7qwCk/I0Btxh2qWJhW+RKX9brg8mhUSAoSkWoQxKoXXtEV8/YxcCFFgfpJi6JjJtqrTqGewxyJbMoAKICJDakipomINUR8ONgeMvFMz9ovUyR7rVrWmxECO7bIMu2S73/aCTbkil2JNlZaApJpYRuRUM2gCZezMKR2TBKitOEj9CizOB7tcZsxXpTNrTbp1yBBF1UAK+YUhmfxVOyzuJBTK5WXd35K2fkEGYEtsI2FByLsiujf+YwZQ+RO2c0t8gHywcpCN
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 6
      • smithi
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • client:
        • sha1: distro
      • kdb: True
      • sha1: distro
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
      • extra_system_packages:
        • deb:
          • bison
          • flex
          • libelf-dev
          • libssl-dev
        • rpm:
          • bison
          • flex
          • elfutils-libelf-devel
          • openssl-devel
      • extra_packages:
        • deb:
          • python3-cephfs
          • cephfs-shell
        • rpm:
          • python3-cephfs
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_strays
  • teuthology_branch: master
  • verbose: False
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2019-08-28 04:03:37
  • started: 2019-08-28 14:35:06
  • updated: 2019-08-28 14:57:05
  • status_class: danger
  • runtime: 0:21:59
  • wait_time: 0:21:55