Pulpito

teuthology-2016-07-17_03:00:02-rados-hammer-distro-basic-vps/319327

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2016-07-17 10:01:51 2016-07-17 10:02:57 2016-07-17 10:22:32 0:19:35 0:11:54 0:07:41 vps master ubuntu 2 fail
Nodes: vpm095 vpm101

Description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr-failures/few.yaml thrashers/default.yaml workloads/snaps-few-objects.yaml}

Log: http://qa-proxy.ceph.com/teuthology/teuthology-2016-07-17_03:00:02-rados-hammer-distro-basic-vps/319327/teuthology.log

Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=d5f54fc215ef4dea94a75e6df12d8902

Performance graphs: http://pcp.front.sepia.ceph.com:44323/grafana/index.html#/dashboard/script/index.js?time_to=2016-07-17T10%3A18%3A33&time_from=2016-07-17T10%3A08%3A25&hosts=vpm095%2Cvpm101

Failure Reason:

{'vpm095.front.sepia.ceph.com': {'cmd': '/usr/bin/git reset --hard origin/master', '_ansible_no_log': False, 'stdout': '', 'changed': False, 'invocation': {'module_name': 'git', 'module_args': {'force': True, 'track_submodules': False, 'reference': None, 'dest': '/var/lib/teuthworker/.cache/src/keys', 'verify_commit': False, 'clone': True, 'update': True, 'ssh_opts': None, 'repo': 'https://github.com/ceph/keys', 'bare': False, 'refspec': None, 'executable': None, 'remote': 'origin', 'recursive': True, 'accept_hostkey': False, 'depth': None, 'version': 'master', 'key_file': None}}, 'failed': True, 'stderr': "fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.\n\nIf no other git process is currently running, this probably means a\ngit process crashed in this repository earlier. Make sure no other git\nprocess is running and remove the file manually to continue.\n", 'rc': 128, '_ansible_delegated_vars': {'ansible_host': 'localhost'}, 'stdout_lines': [], 'msg': 'Failed to checkout master'}}

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/teuthology-2016-07-17_03:00:02-rados-hammer-distro-basic-vps/319327/teuthology.log
  • archive_path: /var/lib/teuthworker/archive/teuthology-2016-07-17_03:00:02-rados-hammer-distro-basic-vps/319327
  • description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr-failures/few.yaml thrashers/default.yaml workloads/snaps-few-objects.yaml}
  • duration: 0:11:54
  • email: ceph-qa@ceph.com
  • failure_reason: {'vpm095.front.sepia.ceph.com': {'cmd': '/usr/bin/git reset --hard origin/master', '_ansible_no_log': False, 'stdout': '', 'changed': False, 'invocation': {'module_name': 'git', 'module_args': {'force': True, 'track_submodules': False, 'reference': None, 'dest': '/var/lib/teuthworker/.cache/src/keys', 'verify_commit': False, 'clone': True, 'update': True, 'ssh_opts': None, 'repo': 'https://github.com/ceph/keys', 'bare': False, 'refspec': None, 'executable': None, 'remote': 'origin', 'recursive': True, 'accept_hostkey': False, 'depth': None, 'version': 'master', 'key_file': None}}, 'failed': True, 'stderr': "fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.\n\nIf no other git process is currently running, this probably means a\ngit process crashed in this repository earlier. Make sure no other git\nprocess is running and remove the file manually to continue.\n", 'rc': 128, '_ansible_delegated_vars': {'ansible_host': 'localhost'}, 'stdout_lines': [], 'msg': 'Failed to checkout master'}}
  • flavor:
  • job_id: 319327
  • kernel:
    • sha1: distro
    • kdb: True
  • last_in_suite: False
  • machine_type: vps
  • name: teuthology-2016-07-17_03:00:02-rados-hammer-distro-basic-vps
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version:
  • overrides:
    • ceph:
      • log-whitelist:
        • slow request
      • fs: xfs
      • conf:
        • global:
          • osd_pool_default_size: 3
          • ms inject socket failures: 5000
          • osd_pool_default_min_size: 2
        • mon:
          • debug mon: 20
          • debug paxos: 20
          • debug ms: 1
        • osd:
          • debug ms: 1
          • debug journal: 20
          • debug osd: 25
          • osd sloppy crc: True
          • osd heartbeat use min delay socket: True
          • debug filestore: 20
      • sha1: 387d5c1ba836833a0cf11ddf9a4fb8690a532878
    • ceph-deploy:
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
          • debug mon: 1
          • debug paxos: 20
          • debug ms: 20
    • workunit:
      • sha1: 387d5c1ba836833a0cf11ddf9a4fb8690a532878
    • install:
      • ceph:
        • sha1: 387d5c1ba836833a0cf11ddf9a4fb8690a532878
    • admin_socket:
      • branch: hammer
  • owner: scheduled_teuthology@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2', 'client.0']
    • ['mon.b', 'osd.3', 'osd.4', 'osd.5', 'client.1']
  • sentry_event: http://sentry.ceph.com/sepia/teuthology/?q=d5f54fc215ef4dea94a75e6df12d8902
  • status: fail
  • success: False
  • branch: hammer
  • seed:
  • sha1: 387d5c1ba836833a0cf11ddf9a4fb8690a532878
  • subset:
  • suite:
  • suite_branch: hammer
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1:
  • targets:
    • ubuntu@vpm095.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRQXJrtM8ks0pYCCsJxuVE0dgARIqqvLLPFTR9UJj7LhekgCCdOeex8+x6qnP5buNk3/aAwTVY4z8Oe4CEBKEdiXry/lNvh34vPRIF6IjcIHIO9bQvjAWEz6DpYR4gR+LqeeWGI8HdKlGKD2z/GDJ1nRnSTDBdLaE6J3h4ZXCzGS5St4J6fxwnRpY4sHZI45SmLmjh04YOruALUWPsBDUtzg0hIaF52V+Tsk9lJ/jvRBMFLEhri+/35EXNkgzZRaqiDWLS7GsoJD9qL8dEJOHj4Q4Tnff/dFcTO50X5GDOGP34T9cd/XSY8soLj9uGQgFFjFqJSa7rpIhjQBWlZjdf
    • ubuntu@vpm101.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqjOoor/Tdl5c7hOA20wPFDAVIECZan21GdLE4JH8O55sqjI2RfLuEauIVr0xMM7KZ0MVXIAyOIrREOgJTGX1T1PGIkf1Xke4Z36133ngnB8afzcadcmHp4TbhE1sMrZ4se5vPHcOKuxsrXgMO5Y45OgN/9d39bJVAxEPMWvh+rAI0UPeTj7xGvutN59PplfQj2u96IlAajVReLiYse65mgi+emAVWJ70+h/eeNhr1MVvL5wsykUxY5WiiztuHGnQ6STfX+c0dckDv0e49qlQPy9Ca0T557MPJtDehp588/MrIL6jgqS7B+IuqE1T3ggjbyFToQlxoljOL9y/oJUbX
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.lock_machines:
      • 2
      • vps
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • sha1: distro
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock.check:
    • install:
    • ceph:
      • log-whitelist:
        • wrongly marked me down
        • objects unfound and apparently lost
      • conf:
        • osd:
          • osd max backfills: 1
          • osd scrub min interval: 60
          • osd scrub max interval: 120
          • osd debug reject backfill probability: 0.3
    • thrashosds:
      • chance_pgnum_grow: 1
      • chance_pgpnum_fix: 1
      • timeout: 1200
    • rados:
      • op_weights:
        • snap_remove: 50
        • snap_create: 50
        • rollback: 50
        • read: 100
        • copy_from: 50
        • write: 100
        • delete: 50
      • clients:
        • client.0
      • objects: 50
      • ops: 4000
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url: http://pcp.front.sepia.ceph.com:44323/grafana/index.html#/dashboard/script/index.js?time_to=2016-07-17T10%3A18%3A33&time_from=2016-07-17T10%3A08%3A25&hosts=vpm095%2Cvpm101
  • priority:
  • user:
  • queue:
  • posted: 2016-07-17 10:01:51
  • started: 2016-07-17 10:02:57
  • updated: 2016-07-17 10:22:32
  • status_class: danger
  • runtime: 0:19:35
  • wait_time: 0:07:41