Pulpito

teuthology-2018-09-05_05:20:02-kcephfs-mimic-testing-basic-ovh/2982491

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2018-09-05 05:20:59 2018-09-10 15:04:56 2018-09-10 19:09:01 4:04:05 0:15:43 3:48:22 ovh master ubuntu 18.04 6 fail

Description: kcephfs/recovery/{clusters/1-mds-4-client.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml overrides/{debug.yaml frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/client-recovery.yaml whitelist_health.yaml}

Log: http://qa-proxy.ceph.com/teuthology/teuthology-2018-09-05_05:20:02-kcephfs-mimic-testing-basic-ovh/2982491/teuthology.log

Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=fb5f1a7d1c564eb4a929562a7a4b3847

Failure Reason:

{'ovh017.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:03:22.139919', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:03:22.133590', 'delta': '0:00:00.006329', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh092.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:04:05.377064', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:04:05.370082', 'delta': '0:00:00.006982', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh065.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:05:56.858687', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:05:56.852688', 'delta': '0:00:00.005999', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh099.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:07:44.302965', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:07:44.296551', 'delta': '0:00:00.006414', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh023.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:03:30.883260', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:03:30.877764', 'delta': '0:00:00.005496', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh016.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:03:37.801676', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:03:37.796232', 'delta': '0:00:00.005444', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/teuthology-2018-09-05_05:20:02-kcephfs-mimic-testing-basic-ovh/2982491/teuthology.log
  • archive_path: /home/teuthworker/archive/teuthology-2018-09-05_05:20:02-kcephfs-mimic-testing-basic-ovh/2982491
  • description: kcephfs/recovery/{clusters/1-mds-4-client.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml overrides/{debug.yaml frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/client-recovery.yaml whitelist_health.yaml}
  • duration: 0:15:43
  • email: ceph-qa@lists.ceph.com
  • failure_reason: {'ovh017.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:03:22.139919', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:03:22.133590', 'delta': '0:00:00.006329', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh092.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:04:05.377064', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:04:05.370082', 'delta': '0:00:00.006982', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh065.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:05:56.858687', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:05:56.852688', 'delta': '0:00:00.005999', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh099.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:07:44.302965', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:07:44.296551', 'delta': '0:00:00.006414', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh023.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:03:30.883260', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:03:30.877764', 'delta': '0:00:00.005496', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh016.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-09-10 19:03:37.801676', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-09-10 19:03:37.796232', 'delta': '0:00:00.005444', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}
  • flavor:
  • job_id: 2982491
  • kernel:
    • flavor: basic
    • sha1: 56f94a157df74137ea5626a561794780d16e608c
    • kdb: True
  • last_in_suite: False
  • machine_type: ovh
  • name: teuthology-2018-09-05_05:20:02-kcephfs-mimic-testing-basic-ovh
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version: 18.04
  • overrides:
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • but it is still running
        • slow request
        • evicting unresponsive client
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • fs: xfs
      • conf:
        • client.1:
          • debug ms: 1
          • debug client: 20
        • client.0:
          • debug ms: 1
          • debug client: 20
        • client:
          • debug ms: 1
          • debug client: 10
        • mon:
          • debug mon: 20
          • debug paxos: 20
          • debug ms: 1
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds bal fragment size max: 10000
        • osd:
          • debug ms: 1
          • debug journal: 20
          • debug osd: 25
          • osd objectstore: filestore
          • osd sloppy crc: True
          • debug filestore: 20
          • osd shutdown pgref assert: True
      • sha1: 42cfc1b5895a6eee8b7519d555b828ccf60ec2c0
    • ceph-deploy:
      • fs: xfs
      • filestore: True
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • osd sloppy crc: True
          • osd objectstore: filestore
    • workunit:
      • sha1: 42cfc1b5895a6eee8b7519d555b828ccf60ec2c0
      • branch: mimic
    • install:
      • ceph:
        • sha1: 42cfc1b5895a6eee8b7519d555b828ccf60ec2c0
    • admin_socket:
      • branch: mimic
  • owner: scheduled_teuthology@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.a-s', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event: http://sentry.ceph.com/sepia/teuthology/?q=fb5f1a7d1c564eb4a929562a7a4b3847
  • status: fail
  • success: False
  • branch: mimic
  • seed:
  • sha1: 42cfc1b5895a6eee8b7519d555b828ccf60ec2c0
  • subset:
  • suite:
  • suite_branch: mimic
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 42cfc1b5895a6eee8b7519d555b828ccf60ec2c0
  • targets:
    • ovh023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF1/d+vCU3zDx+eXkbYJ/Nf+uQ1wbE++JcEpaMynU1X4K1aytimE62J1g1fuOHrus1fieCcq5FUtCQvc1qFTVFsrjoQNYYa/S7FAjCkYdhxGR+Oxk0fHpZJiI4cmy3n82ncnAAgIUsqXbcmfVMym5QJPsJQ8i6cJp0Kt+5SSwDisGQ1NaS9b2LUdsuOQ0WdEHpnZe+DBdJvIkzyDBHLtzmcExdBSbV1S03WwiJT3qOQmmJBgyaG2B0x8O4UfYwj3I0WmaElhoDyEfaRCBjLrRu+e5xO7wA7ZNnCl87XWHMdrMVNspL/I4fyedgIPinSrXncKtJbTtuuxjv8mbJpIkj
    • ovh016.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDbmb5QnkxsH/00W+tjg/GK+dWVJXXC6hAiYnoeyVXXAxmTGTPeZoFT8moEuIuSEQJxzeeJ+u4igOx+gmPObfjil+YgTs6Rs2NCSxnDoID6fF0ORMipQLQCl02Gtl6zGChI2nNYA1OIlFcvyCc8hLm32gO+dbN6lU9mUmaJq/x/1eFiFLl/EtORBNb8BU5JyP1Qy3UKIARz4Pq6Z0MvkrXQHSTEWj66oA5lZGAnN/QID8zSb8JfJhtc7UGrf0v0Jl1S2AUJ15p3hHpPEYS8Jajd4sw8AhDDEGPuFVEcVnW54NXKhb0FcuaFKDu+ZsyuoY2/jv5tlegXo1epD1jRWPpp
    • ovh017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxseumZT4ix4P6swusXN4ptrN5q1s8onEppF5A0LsyOSFMoMPGlGi/wQP5H04Fr47SNqgMbSkqLocIjRL6Ixc9zyDlqQgov4H2eWnPUMf69NO59Kr46F4Cef/gMeo0NjpY6mnwmrZb5N4c4zJa0fS6rskGr818q/KFoct19RCN2UkWpJ1s3ZGuTMdNcysK+8eBO+ptS04fAwWx4uWRcGk+duyVSVxkrZ2pRVqMKbwKfIzLKhtw7ZoSBDF0QdKNgRcVSMFABy0nak7QY59+cRwvSBZkAj7ruyIXJQayrRe+3I7wbqv/TcYLlfnuA+o+7BtEmZRoAlEo97RmK8/msz9T
    • ovh065.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDE0X5bR9Ki29yvpaW2cJnGX+zqL0x+DjujAwfDqxI5N3wbqIuGh7BKXAutqdWQAZd0uxSlpjkhAHsj31oIS6jrA2BB78EiQLKqa/5U//DVvZwLCB+4f91354zGyF8Kd8IK43M2C1ojwVG2jvvDUQm540o5HxBZwBmPAx2Gj3e9SzalSHTchF+JTXc3CqAYo7rsHdEFeIeGL1rsBOZmUOHRHUlWWW/uoAXTLrF48VA+S2PA4xZyEz8zf+/ynLqlN9WJI9h6FYFsWDWzLWUsbp/bJ7HqlttuDit3vhiYIKXjPtUZF3W438+fQy80/eiYD2PcdL0av5UqQ5Rihjpp6kDB
    • ovh099.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQa4Cl3ADwJJ1q1QC37U4RqLdTjzi0ETUuTW6fashC1qPqe6ijNN73Hu8Ia1lSxDSyL4NlGSAs5kwy9h1/teWuf7L8e0fLyNro7T1qxLyhY4hBQTqjHA2nMyoJYZlCBEnXRIL3S3lq74gOOASw6oNcihq5bF8xHNSC3jKLHiPUEtGaa1RSofRio2DlpnlytNjLdT+GwTfrK43F9BuxHoJjY/C7WUwdp6HTODfcoF+1LWNrpJ0Ob+hHqwqkEq3ihN/VDHQx+ElGEAvUFjpQXWqAM900EIRdzyfKn4D5ZzZAdbmlVUCJsk2yjtm4FLP89iuqV3ntye5yFismgVJ8p+GH
    • ovh092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaAaQsriTbb9OMPcapnsLAp3e3w/08cjY0QW2WX/eQYLn9/3oeNtw5vDqRumfL8Psmnc8Ur24Wxx+qub1tGFtu/mqJcnufwpIoMZp2N+A6NR73f47DFhj6T6Nk0PLr9CCsZTf6oHRzErzSFarTe23QrjO3EkR1FLjV2wNQUT+54n+H/CHwRTLatBFWDgTywzGpf1my2AeJgJCFm8qgEmQ105EmbjTJtJfAmdaJKA89ooEDDQgsO0LQGy0mH46edemaBFEgGvUgxCfpWinlMdPP7sxg52bfskcBJW7PyKYZvYMcZ7+5AR8Mr9wZJ4nkiTTGPDe48VAnWgGNhvdBc74x
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 6
      • ovh
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • flavor: basic
      • sha1: 56f94a157df74137ea5626a561794780d16e608c
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_client_recovery
      • fail_on_skip: False
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2018-09-05 05:20:59
  • started: 2018-09-10 15:04:56
  • updated: 2018-09-10 19:09:01
  • status_class: danger
  • runtime: 4:04:05
  • wait_time: 3:48:22