Pulpito

teuthology-2018-07-18_05:20:02-kcephfs-mimic-testing-basic-ovh/2793420

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2018-07-18 05:21:10 2018-07-21 12:06:18 2018-07-21 21:04:28 8:58:10 0:16:53 8:41:17 ovh master ubuntu 18.04 6 fail

Description: kcephfs/recovery/{clusters/1-mds-4-client.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml overrides/{debug.yaml frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/client-recovery.yaml whitelist_health.yaml}

Log: http://qa-proxy.ceph.com/teuthology/teuthology-2018-07-18_05:20:02-kcephfs-mimic-testing-basic-ovh/2793420/teuthology.log

Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=5040a57f9bbc4d5cae6d6592331cfea7

Failure Reason:

{'ovh020.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:42:57.258695', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:42:57.252618', 'delta': '0:00:00.006077', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh058.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:43:43.532251', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:43:43.528224', 'delta': '0:00:00.004027', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh040.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:43:17.016169', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:43:17.011140', 'delta': '0:00:00.005029', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh073.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:47:54.352552', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:47:54.346308', 'delta': '0:00:00.006244', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh044.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:42:45.639119', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:42:45.634730', 'delta': '0:00:00.004389', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh051.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:45:19.817415', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:45:19.809346', 'delta': '0:00:00.008069', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/teuthology-2018-07-18_05:20:02-kcephfs-mimic-testing-basic-ovh/2793420/teuthology.log
  • archive_path: /home/teuthworker/archive/teuthology-2018-07-18_05:20:02-kcephfs-mimic-testing-basic-ovh/2793420
  • description: kcephfs/recovery/{clusters/1-mds-4-client.yaml debug/mds_client.yaml dirfrag/frag_enable.yaml mounts/kmounts.yaml objectstore-ec/bluestore.yaml overrides/{debug.yaml frag_enable.yaml log-config.yaml osd-asserts.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{ubuntu_latest.yaml} tasks/client-recovery.yaml whitelist_health.yaml}
  • duration: 0:16:53
  • email: ceph-qa@ceph.com
  • failure_reason: {'ovh020.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:42:57.258695', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:42:57.252618', 'delta': '0:00:00.006077', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh058.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:43:43.532251', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:43:43.528224', 'delta': '0:00:00.004027', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh040.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:43:17.016169', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:43:17.011140', 'delta': '0:00:00.005029', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh073.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:47:54.352552', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:47:54.346308', 'delta': '0:00:00.006244', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh044.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:42:45.639119', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:42:45.634730', 'delta': '0:00:00.004389', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh051.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-07-21 20:45:19.817415', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-07-21 20:45:19.809346', 'delta': '0:00:00.008069', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}
  • flavor:
  • job_id: 2793420
  • kernel:
    • flavor: basic
    • sha1: cf2a3320fdeb89818377b3f4ed19bbf6af353b35
    • kdb: True
  • last_in_suite: False
  • machine_type: ovh
  • name: teuthology-2018-07-18_05:20:02-kcephfs-mimic-testing-basic-ovh
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version: 18.04
  • overrides:
    • ceph-deploy:
      • fs: xfs
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • mon osd full ratio: 0.9
          • mon osd backfillfull_ratio: 0.85
          • bluestore fsck on mount: True
          • mon osd nearfull ratio: 0.8
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • bluestore block size: 96636764160
          • debug rocksdb: 10
          • osd failsafe full ratio: 0.95
      • bluestore: True
    • workunit:
      • sha1: 6720f47157bf50c5a7b0d6c5ab3a3babecef39f6
      • branch: mimic
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
        • but it is still running
        • slow request
        • evicting unresponsive client
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • fs: xfs
      • conf:
        • client.1:
          • debug ms: 1
          • debug client: 20
        • client.0:
          • debug ms: 1
          • debug client: 20
        • client:
          • debug ms: 1
          • debug client: 10
        • mon:
          • debug mon: 20
          • debug paxos: 20
          • debug ms: 1
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds bal fragment size max: 10000
        • osd:
          • mon osd full ratio: 0.9
          • debug ms: 1
          • debug journal: 20
          • debug osd: 25
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • mon osd backfillfull_ratio: 0.85
          • bluestore block size: 96636764160
          • osd shutdown pgref assert: True
          • debug filestore: 20
          • debug rocksdb: 10
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 0.95
          • bluestore fsck on mount: True
      • sha1: 6720f47157bf50c5a7b0d6c5ab3a3babecef39f6
    • install:
      • ceph:
        • sha1: 6720f47157bf50c5a7b0d6c5ab3a3babecef39f6
    • admin_socket:
      • branch: mimic
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_teuthology@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'mon.c', 'mgr.x', 'mds.a-s', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['client.0']
    • ['client.1']
    • ['client.2']
    • ['client.3']
  • sentry_event: http://sentry.ceph.com/sepia/teuthology/?q=5040a57f9bbc4d5cae6d6592331cfea7
  • status: fail
  • success: False
  • branch: mimic
  • seed:
  • sha1: 6720f47157bf50c5a7b0d6c5ab3a3babecef39f6
  • subset:
  • suite:
  • suite_branch: mimic
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 6720f47157bf50c5a7b0d6c5ab3a3babecef39f6
  • targets:
    • ovh020.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdk6JoXRxDkUU2h9Omon2M3Has2Gx0SdwU/O1CKVSNwZEiAfrfEvmeJr7JAPNr9Bhp/LLwaJ+26Px+YzlVenZHmzNX6gWIGzjAV+tW9SvWSDEWAYYVTLpp2wKQvwRghcchShJ1xik0iSPUcKRZuIqmBs96XJ0FYDR+Z8JtitHSE8Z1ZOMcdBmL45SW7xJ+CcDF6MlaqFH+eH675afArUEFIcD12rCLplHTsPYpfe7sNV6NAlxKdk4bA9IkqtEfAS8qjMiIMv2JnwQ25YHsmiyLHCs/2372KNCquqVXJDzZFYGCvvP8xe2JsCOXU/i131B06xNHT1fe/WyWk0LiQ3rb
    • ovh058.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdDMOUKK1SAyK13M5bXvg2EEYPCQK/nsSN693QN835ySBAf2j4ud88dyiIiBnkOjZMJ1SQGz5OcoMjUaPxs00cneidx4+Dh2SCKujI8kvRzNDocE8kW+qTUzmGfja5WcsPaA8C6PgfjVKcLt2h9JpxCKsb2VDtTqsS803JqSCGOUQj8BsnVmd4BcFPp2bjBKpVJkFGxFhGb6Fwnwwn1yrEfyOYTSecoMxc4r1O3brRX+O15t6vO/sRdvVznGZhrkEAaMDWb7/vOjlLX88tUF9JQsujKMviVn1h9s6GBvY8RbVzQloLIqOt3DrSSiiFlqczePvDK/UfIkrfj3sNz1wh
    • ovh040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUXqR8c0Z2q6KbyYIaZRg4ZNFXVg4i1G1eFIkdUJtObijamZTPK9Oc/1pYXv+KxxeWNyE9MSbJWq8vG58fn9Xf36lS27Z8K72SHyM4EgHSWFsLzujTIhJwn18RnK9SaSjhtXjs1jc83hcj9OKN9bTHqU3U+lX+ndrmJ+nQmbTBAbqeJw6v4W21yviTrU+1EpvzCCqF8lB4Rn/zw+FfpywrP3ml6I89pul5RtKFk1VShmYAPHSvlfDwwsdTnb6B1l2a70wix64aLehrzXNzAHKCOzuAeVB1Yz/NvRcXJX/t3ZgSTceRPrZjGPLzjxdT76N2Qd1YT+f/cdxcmeZSow9z
    • ovh073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCwx6RuphotAqouwMSVTGWEPbwi8/r+gBe213q4XXS5bv0kEdVP2KZIQBHvFfyghN+1UdA92cpI/Pc+3yOEm7M4to9BUAeOiozi2vIw+vjEFyhyOB5SSd212dQc280J2WFs/rVNlMKXuwbbQL2Ibdpp2CfZilCF79S8dcJ05mzrkV5C3o9gO8l97UHT1vnAs9OmvrIzzvs8Ztmf1zziHUW2XfafSXORAbb0ObS7Y3NY55m+GWztPEvKLr+i6E+ugEXEc8Pxtvpmye/GanhMNM6kDtbMXDCmC6xFmllOOx2nb4cSW+/hG5aZIIbb7HTSbSYTcQEiPWWT4hNHplKFuB0P
    • ovh044.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOAO3FfOSth6QHq4YkXQC3saZQFvXuQWJgxGieMNVX8K/TjKqZSYy0QWNTFc0dOTn1aIuF+VOUrfyCDpcKeLtI+bsFbb/vym1bR3QoWj+qnUkVr5jQHNKdB6kBXuOE+3Tljs+tptis0aVYmE/fYGAUJURTgWykwXJEL/Gf1rE+GCxCbpGsDbUpRuQjuDi2RJbwIz0WLQrtYk6O8SlnZbIYASaZHGdQb7uemFvRrERlzp9U7VEAgRh0lgmvflKAIeah8J+0mD4U0oHBkpQJzX07ngmAvHtM2yxyulqxyA27DAa62lmfCBt5tnvBcdXJVPmgoV0XRXoCKSAlIylQSzJ3
    • ovh051.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDTr0eqdsewL6yoBA7nlgTUZi+SnqbxVIkP/oGgX8/rN136sD2r5oiikjMcb6ryIMj+Yszf6BhotXX4VVtpsU1//zH0SldcukFz3W2YYQoWSVDd5vMHop0at7fb5Xsg69nwQ0O0hNoGfnGGFrzF+T8Llxjk1P2c2j3zVh7ReHHzkZFi5YRFUJgo4CqK2t+8814r1U0CO1D1gAiK7qAblcsMhrcBwkuVI3KZscKl9P7xvug8Bk20nPPhBGaSBA71XZ+dNot4UCNRSBc3ilCf2ik29z8VDJ4CiwZltJqjyn9SejGSyhX5exjKkSrNcngwo51f0tCcLbheqDHwTgz40NJT
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 6
      • ovh
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • flavor: basic
      • sha1: cf2a3320fdeb89818377b3f4ed19bbf6af353b35
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
    • ceph:
    • kclient:
    • cephfs_test_runner:
      • modules:
        • tasks.cephfs.test_client_recovery
      • fail_on_skip: False
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2018-07-18 05:21:10
  • started: 2018-07-21 12:06:18
  • updated: 2018-07-21 21:04:28
  • status_class: danger
  • runtime: 8:58:10
  • wait_time: 8:41:17