Pulpito

teuthology-2018-08-13_05:20:02-kcephfs-mimic-testing-basic-ovh/2900341

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2018-08-13 05:21:48 2018-08-16 04:59:40 2018-08-16 07:25:42 2:26:02 0:14:42 2:11:20 ovh master ubuntu 18.04 6 fail

Description: kcephfs/recovery/{clusters/1-mds-4-client.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{debug.yaml frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/mds-flush.yaml whitelist_health.yaml}

Log: http://qa-proxy.ceph.com/teuthology/teuthology-2018-08-13_05:20:02-kcephfs-mimic-testing-basic-ovh/2900341/teuthology.log

Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=dc07ee52485b491f86f3aa5681721240

Failure Reason:

{'ovh091.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:04:44.911931', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:04:44.906257', 'delta': '0:00:00.005674', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh095.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:09:02.524854', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:09:02.518430', 'delta': '0:00:00.006424', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh059.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:05:55.363388', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:05:55.357782', 'delta': '0:00:00.005606', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh053.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:04:50.166359', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:04:50.160394', 'delta': '0:00:00.005965', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh072.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:05:27.899175', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:05:27.892931', 'delta': '0:00:00.006244', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh068.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:07:17.716719', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:07:17.710755', 'delta': '0:00:00.005964', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/teuthology-2018-08-13_05:20:02-kcephfs-mimic-testing-basic-ovh/2900341/teuthology.log
  • archive_path: /home/teuthworker/archive/teuthology-2018-08-13_05:20:02-kcephfs-mimic-testing-basic-ovh/2900341
  • description: kcephfs/recovery/{clusters/1-mds-4-client.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore-ec-root.yaml overrides/{debug.yaml frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/mds-flush.yaml whitelist_health.yaml}
  • duration: 0:14:42
  • email: ceph-qa@lists.ceph.com
  • failure_reason: {'ovh091.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:04:44.911931', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:04:44.906257', 'delta': '0:00:00.005674', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh095.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:09:02.524854', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:09:02.518430', 'delta': '0:00:00.006424', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh059.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:05:55.363388', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:05:55.357782', 'delta': '0:00:00.005606', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh053.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:04:50.166359', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:04:50.160394', 'delta': '0:00:00.005965', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh072.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:05:27.899175', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:05:27.892931', 'delta': '0:00:00.006244', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh068.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-08-16 07:07:17.716719', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-08-16 07:07:17.710755', 'delta': '0:00:00.005964', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}
  • flavor:
  • job_id: 2900341
  • kernel:
    • flavor: basic
    • sha1: 8cb27127e6dd5dbbaaeb7c1f8c731d3cbec73360
    • kdb: True
  • last_in_suite: False
  • machine_type: ovh
  • name: teuthology-2018-08-13_05:20:02-kcephfs-mimic-testing-basic-ovh
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version: 18.04
  • overrides:
    • ceph-deploy:
      • fs: xfs
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • mon osd full ratio: 0.9
          • mon osd backfillfull_ratio: 0.85
          • bluestore fsck on mount: True
          • mon osd nearfull ratio: 0.8
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • bluestore block size: 96636764160
          • debug rocksdb: 10
          • osd failsafe full ratio: 0.95
      • bluestore: True
    • workunit:
      • sha1: 35dbe36f19323b6b885fad71f47ad0ed4e9d09e9
      • branch: mimic
    • ceph:
      • sha1: 35dbe36f19323b6b885fad71f47ad0ed4e9d09e9
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • fs: xfs
      • conf:
        • client.1:
          • debug ms: 1
          • debug client: 20
        • client.0:
          • debug ms: 1
          • debug client: 20
        • client:
          • debug ms: 1
          • debug client: 10
        • mon:
          • debug mon: 20
          • debug paxos: 20
          • debug ms: 1
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds bal fragment size max: 10000
        • osd:
          • mon osd full ratio: 0.9
          • debug ms: 1
          • debug journal: 20
          • debug osd: 25
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • mon osd backfillfull_ratio: 0.85
          • bluestore block size: 96636764160
          • osd shutdown pgref assert: True
          • debug filestore: 20
          • debug rocksdb: 10
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 0.95
          • bluestore fsck on mount: True
      • cephfs_ec_profile:
        • m=2
        • k=2
        • crush-failure-domain=osd
    • install:
      • ceph:
        • sha1: 35dbe36f19323b6b885fad71f47ad0ed4e9d09e9
    • admin_socket:
      • branch: mimic
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_teuthology@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.a-s', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event: http://sentry.ceph.com/sepia/teuthology/?q=dc07ee52485b491f86f3aa5681721240
  • status: fail
  • success: False
  • branch: mimic
  • seed:
  • sha1: 35dbe36f19323b6b885fad71f47ad0ed4e9d09e9
  • subset:
  • suite:
  • suite_branch: mimic
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 35dbe36f19323b6b885fad71f47ad0ed4e9d09e9
  • targets:
    • ovh091.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnk+OeKbsEKTzskM6jNMNE9tTjlVVJRrVs3IQbTYF6WORLWIFEojKjKhBKsOc31NyorH/B19GNjVvGk34Oq12vqpxmcdxMdJ8Gj/SHHmHOVdOKrIJjmrsecG6QNieOpnRT/6s4MroHvILOozQblvW0cT5xgb2OSH74+JSoNc0tBIUH+1cRPvPAq5qehWL7d32HoTRJJX3CLgoOr0l1U3n/z2Gu49os2HnKBlYSfKzZRoJTS1JzECJ0vgPhf/9hi0VwCkwVY0PcaD4r3ZyNjBXHCYl/L5yuBKxQHmHkHuEOw6f8MPxKlPCjR/8CpXQaiz65MAv3ISPlA3EKyDTufuPN
    • ovh095.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDERW/mMEsftw3T8jpVbWhLF2c7oAX72E/Oq0DpEIoJzLq1AaVdHM28F8poDYlE7S//ZfiCiT97WaJ3MkjJjWJsKUR0VqyyZXiirtkjlilmFkgVsmqshkoM1UsGv3JquHHGi8jyOJTndzFAVIw8920rfI+s8xUs14HXs6yKUM4Olkk5/TN6T7hGpMZTgQmP8rcJKKQ4COLIGBrks0L6YK3B2JxomG3Qd4Ocol3sunnlrAn0+POSTkdoxtfIqN+nurcB7y7hLaSWGFzjCJWRRRYr8CBYV4S8poKCwl+GCiiHs5ZiHoXXnRzH2adxyZ4z8MeCqHwVp7ibjINSo83E7uTZ
    • ovh068.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYkOkoS8Jik5GIhi5GXHIf1b6LBZYzaqIGsHAJxbAK805Ot2MqHyQh4JVRqUTGAG3cJf5IzscTywJF62U/CwYe6Rb6zMMoGec8lkt5ANEFSJ0ji1rplifGn/JSsj9xX+2uzSMDQN1yLZ+j7FfO2zavj3IDC0rE9+lFir/h015lvsLPgM53euamJoCeg1rQqLU/sagThnvqGz7ga5YK+KpwVsnj3Hj+ejnDTnIwMjVf5mUyoVsg9n8lOuT/5+RcG38r8vrBi3zdS3V3HL+8mEHPJorjmK2ZPLc0nO0+GpwsD1ocoAObQFTV2vulhiJn0X6ivxLtH2q90jwG65i1PFLh
    • ovh059.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrI2HqrChyumZONciiD309ODe9Je9C7gD2qPlaADEieL+UT3v1DYNfK7FkuwV0A922u5KEqt5WqIctYhh0jlGHFofDqBysta2FB7/ixwfOLY8sdZGWshNiwu4HfEenkLXsjXfmUt/2dsUC3sOgOIe5X1OgZObM4h8GZUQxVDtXhG2j8byMrpYnp5L1q56/c2Fxgk/5mBeGyHGwGMHta9M+uCokdGthV5EkZ1iB2+IxfhLsPFY/Sz0tp2qXPN5qH1PtYEWZDq8MyDannUMrsUNwETn3iax+d/iM5JthQhpm5mS7yiZW6+HR7fILWXX+H5T6bPiP23JRKYUT/1HPu2NB
    • ovh053.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDs4Po7pMgshjdLacFBiRR0m3/i0CUY7gY3LhhIgLoP/VbiXVdTTJ+feXbO/lNIZOqeCWtQyUpeJ9UfVuUi7OMfzd2zvnczJGkMqUuXGabUTMxhEiKHf3WhVOGszaICS72ZhX7k6wD+RVMGUWOkqMtD11nDyUB1+yo6/TFud8R0QCalMrxk1Bw2dNGuuovw+xx5Pfw7xydu5jPH5oGhJsW9hc1U73cbAeShoI7FYF8e/z2vCln36QL9eISlb4+PLlL9LfZNNha0/HblxNamLj6oKifv8E6ATrqxQdUMVb4c+4weFjkK8Zaw9Y2SJrs0LBDgITjVdReewDNIOYTJ6nwj
    • ovh072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDsUhE9nXfH6PHP8Zk2MF1pZUjrk+F6XlJ+oDHIVgmzxLacQJe+bCQPOiDy7KcRb2nkSlHTyvXGIZvxRAnSIKYWIzUc9GKN1P3LaMkd2CFi5/CV3IpvbeJb+vHXLJoHMp8vhUD1KiyT1IVWeQh7zJPehFjKC3yGgoS2PgN541AL5MJEQxzr3YUbyWLvjpNxeBWlU03KrLSUjMpD33+Lv0o3aYo8/+RdFtwJv2uynqc8T9p8P8Bl6WE+lT6igyS2YYxEDUZH7Ge64c27SkYPd7TqA3i6CiHYvjd4pcoo5PHJP3OzLKaeVb4rA10eVqMzt7sXAzeuJbsu6JerhRZMZ48V
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 6
      • ovh
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • flavor: basic
      • sha1: 8cb27127e6dd5dbbaaeb7c1f8c731d3cbec73360
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_flush
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2018-08-13 05:21:48
  • started: 2018-08-16 04:59:40
  • updated: 2018-08-16 07:25:42
  • status_class: danger
  • runtime: 2:26:02
  • wait_time: 2:11:20