Pulpito

yuriw-2021-08-26_18:40:53-rados-wip-yuri7-testing-2021-08-26-0841-distro-basic-smithi/6360385

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2021-08-26 18:44:06 2021-08-26 19:11:48 2021-08-26 19:31:48 0:20:00 0:03:21 0:16:39 smithi master ubuntu 20.04 4 dead

Description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}

Log: http://qa-proxy.ceph.com/teuthology/yuriw-2021-08-26_18:40:53-rados-wip-yuri7-testing-2021-08-26-0841-distro-basic-smithi/6360385/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f6dd14ed5feb40ffaa4f3f83746b6f22

Failure Reason:

{'Failure object was': {'smithi168.front.sepia.ceph.com': {'msg': 'Failed to manage policy for boolean nagios_run_sudo: [Errno 0] Error', 'invocation': {'module_args': {'name': 'nagios_run_sudo', 'state': True, 'persistent': True, 'ignore_selinux_state': False}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'invocation')"}

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/yuriw-2021-08-26_18:40:53-rados-wip-yuri7-testing-2021-08-26-0841-distro-basic-smithi/6360385/teuthology.log
  • archive_path: /home/teuthworker/archive/yuriw-2021-08-26_18:40:53-rados-wip-yuri7-testing-2021-08-26-0841-distro-basic-smithi/6360385
  • description: rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{default} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2}
  • duration: 0:03:21
  • email: ceph-qa@ceph.io
  • failure_reason: {'Failure object was': {'smithi168.front.sepia.ceph.com': {'msg': 'Failed to manage policy for boolean nagios_run_sudo: [Errno 0] Error', 'invocation': {'module_args': {'name': 'nagios_run_sudo', 'state': True, 'persistent': True, 'ignore_selinux_state': False}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'invocation')"}
  • flavor:
  • job_id: 6360385
  • kernel:
    • sha1: distro
    • kdb: True
  • last_in_suite: False
  • machine_type: smithi
  • name: yuriw-2021-08-26_18:40:53-rados-wip-yuri7-testing-2021-08-26-0841-distro-basic-smithi
  • nuke_on_error: True
  • os_type: ubuntu
  • os_version: 20.04
  • overrides:
    • ceph-deploy:
      • fs: xfs
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • mon osd full ratio: 0.9
          • mon osd backfillfull_ratio: 0.85
          • bluestore fsck on mount: True
          • mon osd nearfull ratio: 0.8
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • bluestore block size: 96636764160
          • debug rocksdb: 10
          • bdev enable discard: True
          • osd failsafe full ratio: 0.95
          • bdev async discard: True
      • bluestore: True
    • workunit:
      • sha1: 973584e02ba886811cc8104ef7bb1723cc895a27
      • branch: wip-yuri7-testing-2021-08-26-0841
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • sha1: 973584e02ba886811cc8104ef7bb1723cc895a27
      • fs: xfs
      • conf:
        • mgr:
          • debug ms: 1
          • debug mgr: 20
        • global:
          • ms inject socket failures: 5000
          • mon client directed command retry: 5
          • mon election default strategy: 1
        • mon:
          • debug paxos: 20
          • mon scrub interval: 300
          • debug mon: 20
          • debug ms: 1
        • osd:
          • osd debug verify missing on start: True
          • mon osd backfillfull_ratio: 0.85
          • osd shutdown pgref assert: True
          • bdev async discard: True
          • osd max backfills: 3
          • mon osd nearfull ratio: 0.8
          • osd objectstore: bluestore
          • osd op queue: debug_random
          • osd heartbeat use min delay socket: True
          • osd scrub min interval: 60
          • bdev enable discard: True
          • osd failsafe full ratio: 0.95
          • mon osd full ratio: 0.9
          • osd op queue cut off: debug_random
          • debug ms: 1
          • bluestore fsck on mount: True
          • osd scrub max interval: 120
          • debug osd: 20
          • debug bluestore: 20
          • debug bluefs: 20
          • osd debug reject backfill probability: 0.3
          • debug rocksdb: 10
          • bluestore allocator: stupid
          • osd debug verify cached snaps: True
          • osd max markdown count: 1000
          • bluestore block size: 96636764160
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(OSD_SLOW_PING_TIME
        • but it is still running
        • objects unfound and apparently lost
        • slow request
        • overall HEALTH_
        • \(OSDMAP_FLAGS\)
        • \(OSD_
        • \(PG_
        • \(POOL_
        • \(CACHE_POOL_
        • \(SMALLER_PGP_NUM\)
        • \(OBJECT_
        • \(SLOW_OPS\)
        • \(REQUEST_SLOW\)
        • \(TOO_FEW_PGS\)
        • slow request
    • install:
      • ceph:
        • sha1: 973584e02ba886811cc8104ef7bb1723cc895a27
    • admin_socket:
      • branch: wip-yuri7-testing-2021-08-26-0841
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_yuriw@teuthology
  • pid:
  • roles:
    • ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12']
    • ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13']
    • ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14']
    • ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']
  • sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f6dd14ed5feb40ffaa4f3f83746b6f22
  • status: dead
  • success: False
  • branch: wip-yuri7-testing-2021-08-26-0841
  • seed:
  • sha1: 973584e02ba886811cc8104ef7bb1723cc895a27
  • subset:
  • suite:
  • suite_branch: wip-yuri7-testing-2021-08-26-0841
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: 973584e02ba886811cc8104ef7bb1723cc895a27
  • targets:
    • smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNFbkpCp/mn2kTZ0q/qIBzccqGg0HkEo0lPuTEymIzRj7AK6nZVpeN+U8cTRmrAoAmweZjAJ6c/daIo1RVKRpUZcAinrCwpDNLgtVokZFUshbpNmkgF9OeIcwt28iKkBsDAZSwWbpikw7MbB4pAQBQTutT52DepdqhPXT/vQFfPw4Yrqbg00fyV3Rg1iyBOX8T0xOpEZCsBfe2hEXtPxfkcLa8BTXw/X3PpUsRTw2Qj/dIreb2zn3atiEuKisFIF4rRI5KBJD6POCv4ks7NmjatTMv2dDygq8nib7H/91WCmkktkz8WGSJbswYeuLV6wf+1y6G8DLtyH4hwAx0IFGfu428wU1AXm/PuMWD2FqG1c4+qHTJSu+eA0DCRKxAxdiPynkfX1X7IXj/2PQPLxD/30x8S5Z6WjCCi4y728oVxqgfZ539AZeZKEuWjec3GoBPA9hz6plyymj8NInRcJ43QpT1y9AxYuwxQnGN478k6JywUt3ceWAX+f93SrDLzFE=
    • smithi071.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDwk8+MLhPRtFjA1WmU5MCmt3uy4WSblK+QorUhhiLlBHj8BY9qyWb1iIBtfyLXXCr1AQBwxXN3GcgZ58lN/aWh7E8DJ01whKfs5SyjV/qhCWAeDMsElcE5J51rz+nySSWuhh1czrGm3XSiQxgW00o0BgtCbqjNPqqjczQHYuBpkzoXZaYcnkMXxOGIk+SURUgk9auG/ZBOFAt1MmmKIdCVaIOkzV0QikBRXUXnZR/t2wnxA51CnsmsCYDi1ohBrD7nY7ZZ2fMPQDx4kkLF8AqLyh14iwUsmpRo4mETOUTbZfcLyiNr0ufl+Ra3YVYipZ8IHNXuoRc9LrhjDl+MycIE1HZ9A+hTGmn4nDzqCskxEZ1jQAE80xbnsfeBnqHGmXd4fgMVpNz8FVfqxJvFfybKFgNcLQwS1KGmnWI3pthK+ylINwHptVEwKijFpA9Rm2qW0K4+9FVVtAEhwi0aZcrZv184wJ1oUFAOejGCs6iplySz56NhSz3xVAYdTpbaW5U=
    • smithi112.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+vNGhzdP2H9VXVOv+NhKaamqj4TntesN0FzqOn/uJVP43CHJZo4iQYggy6fWS79XOqL/dFhwQmrm8Q50vnJOO0qdWR6bKHtLrQvW/yg9/HUy6gmfZ/n0i2buD7T1lpExoesl0FhOT9RBOTH0y5+1kLtDEujHeh6asYpkZ6eBepr2swTMv7T+4jCBY1oTFAPxuajijsz6S8N6eRBLHnBGSdSTa2XgcN0q2RXcsAX1xaVjdatrD3N/0TARO+GdRQ7qG0naWM4Dgf8NKmVITXtFSCBFp+aOE+2AeV4GbxxdTYkFJe+ia76ljtET+78mhL1AdB671evWJWtXqIoOfF7bLNN1l6hTt5nv3bDX25Yrk0XiRddIuFWgd/3PNODAHpa4aDRrPNT2o8zQ+Ds3dSTN4jFF4dn44FCCyKzWTkY9+eNPZL1UrZgc25UexRtlaOR6/4IkrjOdNjBjESxCMzNBQ/mwoauPfGbYKaM3IIB1TWPdo7bqJ6Lh7YxxDyZ837a0=
    • smithi168.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDImSCZAMCJfV1sDjEZRFeNTUb30cCGZKVUknjYg/GJ2bqLeXKHBniOJXWzby8zlBsj3PghN7NmCzx7vObsZphHQLE4ahEkMzmBdVtom8g44GzUZKah1Cx0gIQFE/8UiSeQFA92148ZaaE91S9NsV2V1ESI9drNZyuy1m5F9C4ZlOpxVpbVxACbZFQ8SzllhBYaPFWxyQkCu01xvrFHgLnVvdzB4IWAWDS4+kgkT378U7QGw9fqgKu+ww2RGF5t28ktNUQIaXwc0x5YJDX9dbYChE5PAgO22XCFC0mVS+6MVua/5H6d4ovF536ZydPIWOpcit0KFD0zxJnazR0KvJQDFEkTCs9x+URobbRotzprj5NopPQvvpXBUn3JW3BEZSkJze3dMGEYWOV7mT8faTLq9dCVdQO7ib/gtTH4Xak+f2tXCciBVynbPoDkpYwBjQu1oRVzUdbwALNj08/VHRA/eiVc5Jp/aC+YVkk3lDIm+oL5890vJ+RXQGClNWdiLRM=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • sha1: distro
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
    • ceph:
      • conf:
        • osd:
          • debug monc: 20
    • thrashosds:
      • aggressive_pg_num_changes: False
      • chance_pgpnum_fix: 1
      • timeout: 1200
      • min_in: 8
      • chance_pgnum_grow: 1
      • chance_pgnum_shrink: 1
    • rados:
      • op_weights:
        • snap_remove: 50
        • write: 0
        • rollback: 50
        • setattr: 25
        • read: 100
        • copy_from: 50
        • snap_create: 50
        • rmattr: 25
        • append: 100
        • delete: 50
      • ops: 400
      • ec_pool: True
      • clients:
        • client.0
      • objects: 50
      • erasure_code_profile:
        • c: 2
        • name: shecprofile
        • plugin: shec
        • crush-failure-domain: osd
        • k: 4
        • m: 3
      • write_append_excl: False
  • teuthology_branch: master
  • verbose: True
  • pcp_grafana_url:
  • priority:
  • user:
  • queue:
  • posted: 2021-08-26 18:44:06
  • started: 2021-08-26 19:11:48
  • updated: 2021-08-26 19:31:48
  • status_class: danger
  • runtime: 0:20:00
  • wait_time: 0:16:39