Pulpito

teuthology-2021-04-02_08:00:03-smoke-octopus-distro-basic-gibba/6016906

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2021-04-02 08:00:41 2021-04-02 08:09:12 2021-04-02 08:30:42 0:21:30 0:05:52 0:15:38 gibba master rhel 8.2 3 dead

Description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_latest} tasks/cfuse_workunit_suites_blogbench}

Log: http://qa-proxy.ceph.com/teuthology/teuthology-2021-04-02_08:00:03-smoke-octopus-distro-basic-gibba/6016906/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=41f5933d0ec34a07b5b24300d0d9e9d0

Failure Reason:

{'Failure object was': {'gibba023.front.sepia.ceph.com': {'msg': 'Failed to checkout master', 'stdout': '', 'stderr': "fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n", 'rc': 128, 'cmd': '/usr/bin/git reset --hard origin/master', 'invocation': {'module_args': {'repo': 'https://github.com/ceph/keys', 'version': 'master', 'force': True, 'dest': '/home/teuthworker/.cache/src/keys', 'remote': 'origin', 'clone': True, 'update': True, 'verify_commit': False, 'gpg_whitelist': [], 'accept_hostkey': False, 'bare': False, 'recursive': True, 'track_submodules': False, 'refspec': 'None', 'reference': 'None', 'depth': 'None', 'key_file': 'None', 'ssh_opts': 'None', 'executable': 'None', 'umask': 'None', 'archive': 'None', 'separate_git_dir': 'None'}}, 'stdout_lines': [], 'stderr_lines': ["fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.", '', 'Another git process seems to be running in this repository, e.g.', "an editor opened by 'git commit'. Please make sure all processes", 'are terminated then try again. If it still fails, a git process', 'may have crashed in this repository earlier:', 'remove the file manually to continue.'], '_ansible_no_log': False, 'attempts': 5, 'changed': False, '_ansible_delegated_vars': {}}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"}

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/teuthology-2021-04-02_08:00:03-smoke-octopus-distro-basic-gibba/6016906/teuthology.log
  • archive_path: /home/teuthworker/archive/teuthology-2021-04-02_08:00:03-smoke-octopus-distro-basic-gibba/6016906
  • description: smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_latest} tasks/cfuse_workunit_suites_blogbench}
  • duration: 0:05:52
  • email: ceph-qa@ceph.io
  • failure_reason: {'Failure object was': {'gibba023.front.sepia.ceph.com': {'msg': 'Failed to checkout master', 'stdout': '', 'stderr': "fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n", 'rc': 128, 'cmd': '/usr/bin/git reset --hard origin/master', 'invocation': {'module_args': {'repo': 'https://github.com/ceph/keys', 'version': 'master', 'force': True, 'dest': '/home/teuthworker/.cache/src/keys', 'remote': 'origin', 'clone': True, 'update': True, 'verify_commit': False, 'gpg_whitelist': [], 'accept_hostkey': False, 'bare': False, 'recursive': True, 'track_submodules': False, 'refspec': 'None', 'reference': 'None', 'depth': 'None', 'key_file': 'None', 'ssh_opts': 'None', 'executable': 'None', 'umask': 'None', 'archive': 'None', 'separate_git_dir': 'None'}}, 'stdout_lines': [], 'stderr_lines': ["fatal: Unable to create '/home/teuthworker/.cache/src/keys/.git/index.lock': File exists.", '', 'Another git process seems to be running in this repository, e.g.', "an editor opened by 'git commit'. Please make sure all processes", 'are terminated then try again. If it still fails, a git process', 'may have crashed in this repository earlier:', 'remove the file manually to continue.'], '_ansible_no_log': False, 'attempts': 5, 'changed': False, '_ansible_delegated_vars': {}}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_85d61eae4759f46ce21e9a37cd816a7a1a66c9d5/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', '_ansible_no_log')"}
  • flavor:
  • job_id: 6016906
  • kernel:
    • sha1: distro
    • kdb: True
  • last_in_suite: False
  • machine_type: gibba
  • name: teuthology-2021-04-02_08:00:03-smoke-octopus-distro-basic-gibba
  • nuke_on_error: True
  • os_type: rhel
  • os_version: 8.2
  • overrides:
    • ceph-deploy:
      • fs: xfs
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • mon osd full ratio: 0.9
          • mon osd backfillfull_ratio: 0.85
          • bluestore fsck on mount: True
          • mon osd nearfull ratio: 0.8
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • bluestore block size: 96636764160
          • debug rocksdb: 10
          • bdev enable discard: True
          • osd failsafe full ratio: 0.95
          • bdev async discard: True
      • bluestore: True
    • selinux:
      • whitelist:
        • scontext=system_u:system_r:logrotate_t:s0
    • workunit:
      • sha1: 5feb67b5bda898b8c739180363606432d4f39831
      • branch: octopus
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • sha1: 5feb67b5bda898b8c739180363606432d4f39831
      • fs: xfs
      • conf:
        • mgr:
          • debug ms: 1
          • debug mgr: 20
        • mon:
          • debug paxos: 20
          • debug mon: 20
          • debug ms: 1
        • osd:
          • mon osd full ratio: 0.9
          • bluestore allocator: bitmap
          • bluestore fsck on mount: True
          • debug osd: 20
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • debug ms: 1
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 0.95
          • bluestore block size: 96636764160
          • osd shutdown pgref assert: True
          • debug rocksdb: 10
          • bdev enable discard: True
          • mon osd backfillfull_ratio: 0.85
          • bdev async discard: True
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
    • install:
      • ceph:
        • sha1: 5feb67b5bda898b8c739180363606432d4f39831
    • admin_socket:
      • branch: octopus
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_teuthology@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mds.a', 'mgr.x', 'osd.0', 'osd.1']
    • ['mon.b', 'mds.b', 'mon.c', 'mgr.y', 'osd.2', 'osd.3']
    • ['client.0']
  • sentry_event: https://sentry.ceph.com/organizations/ceph/?query=41f5933d0ec34a07b5b24300d0d9e9d0
  • status: dead
  • success: False
  • branch: octopus
  • seed:
  • sha1: 5feb67b5bda898b8c739180363606432d4f39831
  • subset:
  • suite:
  • suite_branch: octopus
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 5feb67b5bda898b8c739180363606432d4f39831
  • targets:
    • gibba019.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdJOBZf/aGNlKxtgsEx17bGBXz5zEdQsV13YgxGMYQ6qh5JYAUhNzvA1H8ozty9/Crb6jKW4y9A2eNbTDXJn1aS8Crb3bG5br6MB70WHSUJb3Cm2P1VHYcb2aU7NHEHC4vnCJsWrAzYFI3CqwDdktcFT5SihfaeJ8ATqavrUrlnR9rsfPzvBFHZmP2sOcnFk6Rp01wdK9VqRBRP4zlAIOi7xUW6bdHW46KMTexacteye5pgSmkB6lWXG3DhrPiUHhqXLUaQzrIB5qu0Zmsr5t6dFibETg8Z5Ye8SLyhn1N0WAEzAscbXKQiDxt1kSX8ugGUVQxjOrkpcG1TaTrkpfi1K01Q8bpSrOTKAFFLtwQs/eaalfmCmNwPCnC5he9kQvJQd0OQ8tI6x4o7fx0gTNXukSrooa+RGGtskjUBaMlUc0RPO3WK32uN2rAQNiaIJu9VfTZgXXzHyBN+hQzw5FI2XN2RVD0Eve1m5+re4uMLD+iZsKBnSsbLIGroaZBCL8=
    • gibba023.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbTabIsd9MTSI4HiOiQXgS298zzuHR253dSukJ7z1YnUDXyS6PozAQZALeG+YxUD73MZA7sU6WQNpA5qAQ40tAvSJBE2810jMFcoZpurBfNcEgxGEs3CC5jqivHvr4Qy/NAofHUmeNS7u5mWNSZmhZt160DOt+spaSuXTOt1qslhzDn8FWJtrgeuakGnC+RFOERpniYgY/iapR3OT/y6JSfiqLA7qKPJb2Kk6Rb/oU2LznGdnYqTTsidpMMT0mV4sngpYPWFr2vU4s5QPDcRcn4ObCVHjGeEkmWmuh+6Yuib1sTnn/LoJzr/4CzAlkeWzbzHI0V8uyBcnGrgvWPAG5QxMV5fGw9DIVHgeiqpfjT8ACPMcYZ7Iw1WKZOMIj7+tv+l5zzR9618RpF9ZJIFfc12EI2r+XtDwiarK1ae8CKRDSVS9EDBcT7pVt5oYZL7QCT34mOA8iygWwoyoXtfxhGW8oiDR8cUgbFMeolJW2zGhX9W7ur25XW2V2/uOU1p8=
    • gibba036.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7bdVkDpjQ18yT5xliJAmRA8QijV4mJyFu/sXzH/2FgHCWk+WQTadSjE77J8R4LUTBNjvkMMCxoK3Qcv1BJa8sBquSe2KhOvpANOs5ehywtzxBCAXZoswUcir+z11+NEMBhpW1JPE2mIUVqp/3q3TV7bZ1hrieofk7p3C0yHAISlFdwMwxN4/ja9lYDT1YWdJxVGawfE/mTyTy2+UyidDlAgXMuR9tP9L+5xBF8KNiFABSe0+lz4wsuY4w1Oh2NhIvrLGmn10GW5HxAVPbpDFomUeqvOmpfQ7ShGK7NLrQcXBBfdqhlilgEmGAmHbhitl1N4qgC3R8Aaaf23fJZkY4FF7ICWj6OPO8jxDnhafAnqgS/L57Y1XjKNlT4hlFSATp1AaN3OL0/hKRkQzy5+hxJkmzSsbdwKXuu0J2LG/WyDI4fiuHaDZMezdKo13DxqMOFm20zDZvHcpLGCV96kpkeoGWamIw27UGfbpSU2ALPvALmfkpZevOzSpbEgFK80c=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • sha1: distro
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
    • ceph:
      • fs: xfs
    • ceph-fuse:
    • workunit:
      • clients:
        • all:
          • suites/blogbench.sh
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2021-04-02 08:00:41
  • started: 2021-04-02 08:09:12
  • updated: 2021-04-02 08:30:42
  • status_class: danger
  • runtime: 0:21:30
  • wait_time: 0:15:38