Pulpito

ideepika-2021-03-12_19:31:07-rados-wip-yuri2-testing-2021-03-09-1006-octopus-distro-basic-gibba/5958925

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2021-03-12 19:32:31 2021-03-12 19:45:48 2021-03-12 20:04:27 0:18:39 0:01:49 0:16:50 gibba master ubuntu 18.04 4 dead

Description: rados/upgrade/mimic-x-singleton/{0-cluster/{openstack start} 1-install/mimic 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap thrashosds-health ubuntu_latest}

Log: http://qa-proxy.ceph.com/teuthology/ideepika-2021-03-12_19:31:07-rados-wip-yuri2-testing-2021-03-09-1006-octopus-distro-basic-gibba/5958925/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5798f593e50142ebb1c7d4ca1784d6bd

Failure Reason:

{'Failure object was': {'gibba040.front.sepia.ceph.com': {'msg': 'Failed to checkout branch master', 'stdout': '', 'stderr': "fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n", 'rc': 128, 'invocation': {'module_args': {'repo': 'https://github.com/ceph/keys', 'version': 'master', 'force': True, 'dest': '/home/teuthworker/.cache/src/keys', 'remote': 'origin', 'clone': True, 'update': True, 'verify_commit': False, 'gpg_whitelist': [], 'accept_hostkey': False, 'bare': False, 'recursive': True, 'track_submodules': False, 'refspec': 'None', 'reference': 'None', 'depth': 'None', 'key_file': 'None', 'ssh_opts': 'None', 'executable': 'None', 'umask': 'None', 'archive': 'None', 'separate_git_dir': 'None'}}, 'stdout_lines': [], 'stderr_lines': ["fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.", '', 'Another git process seems to be running in this repository, e.g.', "an editor opened by 'git commit'. Please make sure all processes", 'are terminated then try again. If it still fails, a git process', 'may have crashed in this repository earlier:', 'remove the file manually to continue.'], '_ansible_no_log': False, 'attempts': 5, 'changed': False, '_ansible_delegated_vars': {}}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"}

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/ideepika-2021-03-12_19:31:07-rados-wip-yuri2-testing-2021-03-09-1006-octopus-distro-basic-gibba/5958925/teuthology.log
  • archive_path: /home/teuthworker/archive/ideepika-2021-03-12_19:31:07-rados-wip-yuri2-testing-2021-03-09-1006-octopus-distro-basic-gibba/5958925
  • description: rados/upgrade/mimic-x-singleton/{0-cluster/{openstack start} 1-install/mimic 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap thrashosds-health ubuntu_latest}
  • duration: 0:01:49
  • email:
  • failure_reason: {'Failure object was': {'gibba040.front.sepia.ceph.com': {'msg': 'Failed to checkout branch master', 'stdout': '', 'stderr': "fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n", 'rc': 128, 'invocation': {'module_args': {'repo': 'https://github.com/ceph/keys', 'version': 'master', 'force': True, 'dest': '/home/teuthworker/.cache/src/keys', 'remote': 'origin', 'clone': True, 'update': True, 'verify_commit': False, 'gpg_whitelist': [], 'accept_hostkey': False, 'bare': False, 'recursive': True, 'track_submodules': False, 'refspec': 'None', 'reference': 'None', 'depth': 'None', 'key_file': 'None', 'ssh_opts': 'None', 'executable': 'None', 'umask': 'None', 'archive': 'None', 'separate_git_dir': 'None'}}, 'stdout_lines': [], 'stderr_lines': ["fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.", '', 'Another git process seems to be running in this repository, e.g.', "an editor opened by 'git commit'. Please make sure all processes", 'are terminated then try again. If it still fails, a git process', 'may have crashed in this repository earlier:', 'remove the file manually to continue.'], '_ansible_no_log': False, 'attempts': 5, 'changed': False, '_ansible_delegated_vars': {}}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_e8438aef42f98e8ef3e8cdc260580ac63cdc5a1f/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"}
  • flavor:
  • job_id: 5958925
  • kernel:
    • sha1: distro
    • kdb: True
  • last_in_suite: False
  • machine_type: gibba
  • name: ideepika-2021-03-12_19:31:07-rados-wip-yuri2-testing-2021-03-09-1006-octopus-distro-basic-gibba
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version: 18.04
  • overrides:
    • ceph-deploy:
      • fs: xfs
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • mon osd full ratio: 0.9
          • mon osd backfillfull_ratio: 0.85
          • bluestore fsck on mount: True
          • mon osd nearfull ratio: 0.8
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • bluestore block size: 96636764160
          • debug rocksdb: 10
          • bdev enable discard: True
          • osd failsafe full ratio: 0.95
          • bdev async discard: True
      • bluestore: True
    • workunit:
      • sha1: dbb79e0547db3abf076b9bc9b6ad97ede0519a0e
      • branch: wip-yuri2-testing-2021-03-09-1006-octopus
    • ceph:
      • mon_bind_addrvec: False
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(MON_DOWN\)
        • \(MGR_DOWN\)
        • slow request
        • but it is still running
        • objects unfound and apparently lost
        • log bound mismatch
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_
      • sha1: dbb79e0547db3abf076b9bc9b6ad97ede0519a0e
      • fs: xfs
      • mon_bind_msgr2: False
      • conf:
        • mds:
          • debug ms: 1
          • debug mds: 20
        • global:
          • ms dump corrupt message level: 0
          • bluestore warn on no per pool omap: False
          • mon pg warn min per osd: 0
          • ms bind msgr2: False
          • bluestore warn on legacy statfs: False
        • osd:
          • mon osd full ratio: 0.9
          • bluestore allocator: bitmap
          • bluestore fsck on mount: True
          • debug osd: 25
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • osd max markdown count: 1000
          • debug ms: 1
          • debug journal: 20
          • osd failsafe full ratio: 0.95
          • mon osd nearfull ratio: 0.8
          • bluestore block size: 96636764160
          • debug filestore: 20
          • debug rocksdb: 10
          • bdev enable discard: True
          • mon osd backfillfull_ratio: 0.85
          • bdev async discard: True
        • mon:
          • debug paxos: 20
          • debug mon: 20
          • debug ms: 1
        • mgr:
          • debug ms: 1
          • debug mgr: 20
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(OSDMAP_FLAGS\)
        • \(OSD_
        • \(PG_
        • \(POOL_
        • \(CACHE_POOL_
        • \(SMALLER_PGP_NUM\)
        • \(OBJECT_
        • \(SLOW_OPS\)
        • \(REQUEST_SLOW\)
        • \(TOO_FEW_PGS\)
        • slow request
    • install:
      • ceph:
        • sha1: dbb79e0547db3abf076b9bc9b6ad97ede0519a0e
    • admin_socket:
      • branch: wip-yuri2-testing-2021-03-09-1006-octopus
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_ideepika@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
    • ['mon.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
    • ['mon.c', 'mgr.x', 'mgr.y', 'osd.8', 'osd.9', 'osd.10', 'osd.11']
    • ['client.0']
  • sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5798f593e50142ebb1c7d4ca1784d6bd
  • status: dead
  • success: False
  • branch: wip-yuri2-testing-2021-03-09-1006-octopus
  • seed:
  • sha1: dbb79e0547db3abf076b9bc9b6ad97ede0519a0e
  • subset:
  • suite:
  • suite_branch: wip-yuri2-testing-2021-03-09-1006-octopus
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: dbb79e0547db3abf076b9bc9b6ad97ede0519a0e
  • targets:
    • gibba015.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCn+Uhvq5knnvBf1mwXn3b+Bp1gCpxEzUtPmMNhQ+x30OpEyn9/ZusM2rAnlwxwfvoboMjsD8XBKytEfUV+1v43dTFRWUUiiWHOw6DiUyzUJiw1niSiwuOEfoQ+pzxU5DqsGOQOcPUJHG2aQN/pHdOS7avwOGxlVXu5s4rHOVRZBUrcr9eV4MZuJFNIn37AnQHe0Do3FTVw7aoVgOejYVp/X35CfQ0DmtdIeGwH1aOUN/wiEOGJk5G0H0cv49UXA6rQ45WUx9Fslsqv+O6kd/PiRmmkpeejlVgX9ZCs1rurF/fsDmBIMfWUs3DGVfINKKRPOidhIEiUGZdLKFnpQU1V
    • gibba023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHnPGj3IVXkHuk0xADi3LqYHO+VunEFAlottxpoFurUVSLJdiwNesLqthl9FTBvqM62rZXHcyh48ngIzOI8rPTafhRfoWXg6/sSEa6zZGxpBouXsuMC9Kzgd/CSfBAJPjXC7cHxeQ49xULdho2efMm3IUB89cUU6gr8aCn5ar2gC7k+wDoA5UxAgN5TGXB2p0EVFiZzqBJF0ab19iI+qmFbR4RI2xwEu+wVAEs0mkWaTDTVuCWDZiDlr2l80SDu1rdVD5W24fXst9T4IuzxNXl0jKyeIzbLcwM68nAZoYnkTPHsJ0d6h92V6rCqKDjir1aDVtsiJSIbkVC60ascTMZ
    • gibba040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw8dCr4pxvXpNQUdVC/mXqlcmGyvWrPPwPxHsbeXp2URGv+SmBoCO58W4us7H/QPDm2ytLyRaH8tqZxF4Lckx96EGRWbXg1tVSJ2wQqzaVKpaGbRrheph3yb3pj0Vvnhs6H4pkDn+mPbZYtgZzS+XfGR7s91UzTPlKrr1QgGGG+HywchWwggA3JWQD1MPR2rKKd0SD3onvLUBDRdhDdCIL4D+zngLgrsa6ek4iY+fFOOW0/RzCufMK6TUptUO7zKq/2UZYHlWmb4ggMwYK0KltHIr3nDmQmNc2uZtAWs0UejaIAF/jyu7ntJWMaJPqpu4ul8x58rkUaD6ObuDdqT2P
    • gibba045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCeyJhrbo503YX7cKz9pGxhlnQiOB+tgAzcXHkR5yW3vuN9Flhv28/GJ+H0/iuqCUFv+Ri9kxn2X1bKfn8/Ml1ujlfCTtYvY/bPwdcFmfrKgUZ4eiQ0MeBjr6E5ZhWp6SZDjHOUA9IX38MHlx5v46ONvYPzD6DnB09/VImdBbp+n979mw7XQr6taiAR7Xn4EVprUzYJThVaeOhVx4/kRtwKB70SkHyn/yaU4mxIbME1dXVFO5bnS0zD1Uuw8CIFyzKkStR1V3e8oIRs7KDYKLs8o2DllYaqFO8LPkCrkhXBtAo+VQ0k8m5xTQYNzgsplKAT0dVKCD+GUwnojz1yPbtz
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • sha1: distro
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
      • extra_packages:
        • librados2
      • exclude_packages:
        • librados3
        • ceph-mgr-dashboard
        • ceph-mgr-diskprediction-local
        • ceph-mgr-rook
        • ceph-mgr-cephadm
        • cephadm
      • branch: mimic
    • print: **** done install mimic
    • ceph:
    • print: **** done ceph
    • install.upgrade:
      • mon.a:
      • mon.b:
    • print: **** done install.upgrade osd.0
    • ceph.restart:
      • daemons:
        • mon.a
        • mon.b
      • wait-for-healthy: False
      • mon-health-to-clog: False
    • ceph.restart:
      • daemons:
        • osd.0
        • osd.1
        • osd.2
        • osd.3
        • osd.4
        • osd.5
        • osd.6
        • osd.7
      • wait-for-healthy: False
    • print: **** done ceph.restart 1st 2/3s
    • parallel:
      • split_tasks
    • install.upgrade:
      • mon.c:
    • ceph.restart:
      • daemons:
        • mon.c
        • mgr.x
        • mgr.y
      • wait-for-healthy: False
      • wait-for-up: True
    • ceph.restart:
      • daemons:
        • osd.8
        • osd.9
        • osd.10
        • osd.11
      • wait-for-healthy: False
      • wait-for-up: True
    • ceph.restart:
      • daemons:
        • mds.a
      • wait-for-healthy: False
      • wait-for-up: True
    • exec:
      • mon.a:
        • ceph mon enable-msgr2
    • install.upgrade:
      • client.0:
    • exec:
      • osd.0:
        • ceph osd require-osd-release octopus
        • ceph osd set-require-min-compat-client octopus
        • for f in `ceph osd pool ls` ; do ceph osd pool set $f pg_autoscale_mode off ; done
    • ceph.healthy:
    • workunit:
      • clients:
        • client.0:
          • rbd/test_librbd_python.sh
    • print: **** done rbd/test_librbd_python.sh 9-workload
    • rados:
      • op_weights:
        • snap_remove: 50
        • write: 100
        • rollback: 50
        • read: 100
        • snap_create: 50
        • delete: 50
      • clients:
        • client.0
      • write_append_excl: False
      • objects: 500
      • ops: 4000
  • teuthology_branch: master
  • verbose: False
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2021-03-12 19:32:31
  • started: 2021-03-12 19:45:48
  • updated: 2021-03-12 20:04:27
  • status_class: danger
  • runtime: 0:18:39
  • wait_time: 0:16:50