Pulpito

teuthology-2018-08-15_05:20:01-kcephfs-mimic-testing-basic-ovh/2906943

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2018-08-15 05:21:45 2018-08-19 15:04:59 2018-08-19 20:45:03 5:40:04 0:16:08 5:23:56 ovh master ubuntu 18.04 6 fail

Description: kcephfs/recovery/{clusters/1-mds-4-client.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml overrides/{debug.yaml frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/failover.yaml whitelist_health.yaml}

Log: http://qa-proxy.ceph.com/teuthology/teuthology-2018-08-15_05:20:01-kcephfs-mimic-testing-basic-ovh/2906943/teuthology.log

Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=4e8886195bab407d898808a2c49e1d6c

Failure Reason:

{'ovh031.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:40:03.438125', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:40:03.432383', 'delta': '0:00:00.005742', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh017.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:39:59.242773', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:39:59.235736', 'delta': '0:00:00.007037', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh014.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:40:16.823579', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:40:16.818132', 'delta': '0:00:00.005447', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh015.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:42:18.078755', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:42:18.073032', 'delta': '0:00:00.005723', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh054.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:44:30.216612', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:44:30.211603', 'delta': '0:00:00.005009', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh016.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:39:59.407154', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:39:59.401789', 'delta': '0:00:00.005365', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/teuthology-2018-08-15_05:20:01-kcephfs-mimic-testing-basic-ovh/2906943/teuthology.log
  • archive_path: /home/teuthworker/archive/teuthology-2018-08-15_05:20:01-kcephfs-mimic-testing-basic-ovh/2906943
  • description: kcephfs/recovery/{clusters/1-mds-4-client.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/filestore-xfs.yaml overrides/{debug.yaml frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/failover.yaml whitelist_health.yaml}
  • duration: 0:16:08
  • email: ceph-qa@lists.ceph.com
  • failure_reason: {'ovh031.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:40:03.438125', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:40:03.432383', 'delta': '0:00:00.005742', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh017.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:39:59.242773', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:39:59.235736', 'delta': '0:00:00.007037', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh014.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:40:16.823579', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:40:16.818132', 'delta': '0:00:00.005447', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh015.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:42:18.078755', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:42:18.073032', 'delta': '0:00:00.005723', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh054.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:44:30.216612', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:44:30.211603', 'delta': '0:00:00.005009', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh016.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-19 20:39:59.407154', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-19 20:39:59.401789', 'delta': '0:00:00.005365', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}
  • flavor:
  • job_id: 2906943
  • kernel:
    • flavor: basic
    • sha1: 8b082c6ca01f6f063c725092620e58fbdd66350b
    • kdb: True
  • last_in_suite: False
  • machine_type: ovh
  • name: teuthology-2018-08-15_05:20:01-kcephfs-mimic-testing-basic-ovh
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version: 18.04
  • overrides:
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • not responding, replacing
        • \(MDS_INSUFFICIENT_STANDBY\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • fs: xfs
      • conf:
        • client.1:
          • debug ms: 1
          • debug client: 20
        • client.0:
          • debug ms: 1
          • debug client: 20
        • client:
          • debug ms: 1
          • debug client: 10
        • mon:
          • debug mon: 20
          • debug paxos: 20
          • debug ms: 1
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds bal fragment size max: 10000
        • osd:
          • debug ms: 1
          • debug journal: 20
          • debug osd: 25
          • osd objectstore: filestore
          • osd sloppy crc: True
          • debug filestore: 20
          • osd shutdown pgref assert: True
      • sha1: 8878eb09cd9da6cfa434101a4b4f2b078ea79ef6
    • ceph-deploy:
      • fs: xfs
      • filestore: True
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • osd sloppy crc: True
          • osd objectstore: filestore
    • workunit:
      • sha1: 8878eb09cd9da6cfa434101a4b4f2b078ea79ef6
      • branch: mimic
    • install:
      • ceph:
        • sha1: 8878eb09cd9da6cfa434101a4b4f2b078ea79ef6
    • admin_socket:
      • branch: mimic
  • owner: scheduled_teuthology@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.a-s', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event: http://sentry.ceph.com/sepia/teuthology/?q=4e8886195bab407d898808a2c49e1d6c
  • status: fail
  • success: False
  • branch: mimic
  • seed:
  • sha1: 8878eb09cd9da6cfa434101a4b4f2b078ea79ef6
  • subset:
  • suite:
  • suite_branch: mimic
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 8878eb09cd9da6cfa434101a4b4f2b078ea79ef6
  • targets:
    • ovh031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHVCXa2lUc/JQJWXYhzKgaSvrLBTZGE6aJEtlyt8v2DbmesM1lbVIQMjRr+6vGtri6TbEGUX9grA+z1sgzKslFkfAlWNNBX+RpQ0yBWcP+VI4rfx6xjySGZrTUsMlJ2+1NhhUzQ2QgC4DuvlwsC+EXq8E53luRwcE8qiHnaIa8oFS/dwirL7s01+QisTWCgmMbZc6iIvw7prl6SHXOsEV9BxthocfO6+Q4L9NMyJHrHAEsm5X0F0J6vkO04dffSQggG6Re+Y5uT71kObWHBO9spDlTAjay4FtA5zh97UfuXPY++pjJNEryefXBbpZc4tRqdE/iof0cllfvcr/ZD3Cp
    • ovh016.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLLEiGfwC3VY5IxCZCPQg4tIdF4Afn7pdJHl7n4eNGKYcachwAhB9+3NzRXBMd95IkM0HEXJo6ME5wo1+7mNYm1eAsnmn2zX0ReglzLXPRk7fnawBeFyGSYwL0/tTt2RjJyGo9h1FyNzWjAQ/HRD8VhNM9AMJ74dr2zI6Lm3Ndgy9N2WJsu1+aOuF5XbEirGRM/FK3+kU7b6MyXRRGltjJakfmd+TMrMsSZTwK3H2lAW4suZ5N77jYnm43nY2VOFnONe23q9X8/jgTRiJFF5OMxQJk01kr6Um/xsqOD3TJIIlS8ufG3XMo6z69u8EClWnqKjrwBZ9hAn22v+0F60Qr
    • ovh017.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCygw7eXIHGAjYpOyQbQEBCeJOSSxpDmrz41V8+tbesuqcgTjgvgMPzkwblhJ4yBIf+Uf5sLz09QYC2vAO2wpZODX3SdiWhqtY5JKYb0LbvZ1tMHU6Sxq0Pi0G/LpbJB8PmHtb81Kts2dW+5+5SML1GZQgG0ogMmRLol2Jirwhv5FwoqXIySBPE+m9npHnOQrEKdEgB4MvMKftFFB5OoK54YaujgODslFB1U+FddnleZkDK3tlLCpGId1rt46ZUtGJ7JdG+YvDkNTr7+/dmXTxj7g3b3He8faeI7ZdD+rUEi6+yZ0+fNl3XgGwAXP/0eqzyr9R5sCqr//g3XicWy7dh
    • ovh054.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1hO63fvzBBdRUg6vqgfTstE8Ci7qVKpDtRdw/Ex1q2IyyCOqbqscXJxGj0Qp7Jl4s1VlrgSWHrn9qWD5giY8gU/dYy5k+8iLdB4+tfWqpvAX2CUI4OgDzz4r4lPH0dJzboQuqv3kCW4DCmbGMmaXXewZZ9PzaVQ6JGNkWl8+aUe0NUcD4Ibnbb3NgMutNdDZ3jC0ysuzZxNC8F8xQaVSp2hNupRh9xu/qcF4bJvzpAoWLRU+iNs3k2fsNfgm/gcOBuugZQj7nPmuZeZPoSV5zCHTkUUTZYbtuTdx54GXGL3p7vSReYc3L2m8BbXobu6u/VsHgpVkUHgKlOoCIfjY5
    • ovh014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw/Wl/6PCI9RJQLD367aFkCf/wvXKjvc8k/RZ+H1r4Uwp+J/ATk28gIFtv15CAaDlcVFB3dtUn7O+uziSeBy495QbGarOESB6kqzLqyrYeaITxl8dP1rB/hKyrWN91pO+oRa0IiljR8ZPFTBiEobaAteJdw2Vyiyu+iZX9W7skbZAD4nTlLy91gdEg8MhtbWJhdCAFxq2V+h7LQZpo7a13GuULIJQtQ/N8lx440wAoxu1V6b4n1r5/jhQENDKu82zK6eCG2WXPfuY92Tz85QtrJLJTRjiY3JigAJI0ghyQYF0pk5i+WdMGLGPpN61BRl7fwppKwIaWiN3+SJVTAPfz
    • ovh015.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCf7ieBJpWW7JcCHpdqxGjlTdiJNonXkowEnNY0s1V/SaN1SAr7zynN7wAoLOzdqZh0uINEHPJdHey8NvM181qgs5rG9LpyrVwJWSzdXEXlPPCQyom8Cq4hzWCcFsMu+ALkI60eJndztmMYAM2GMwonFkNJ8ua2zcqj98CxUCZB5sngRLNWHRzVp9z+MrDgnCcm43Jgiy4xJjjwAOIWt+OwD76yvcRSbcTzN1/Sd5ph1zjkXVCQYAAnmwmIu3KOnKZwnrFIzxCP41myMn3b9bStjy/6p9FE6434FST9ko8TZZKu8uAVF+R7saBrlIdsRSquVWBTr28cPIUyGcMPHRX5
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 6
      • ovh
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • flavor: basic
      • sha1: 8b082c6ca01f6f063c725092620e58fbdd66350b
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_failover
      • fail_on_skip: False
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2018-08-15 05:21:45
  • started: 2018-08-19 15:04:59
  • updated: 2018-08-19 20:45:03
  • status_class: danger
  • runtime: 5:40:04
  • wait_time: 5:23:56